Methods, apparatus, and computer programs

Abstract

To provide a technique that reduces the time and effort required to prepare data for learning.SOLUTION: A method has a mobile route acquisition process for acquiring a movement route of a mobile entity, a superimposed image generation process for generating a superimposed image by superimposing an image or three-dimensional data representing the mobile entity and an image or three-dimensional data representing the movement environment including the movement route, a label information acquisition process for acquiring label information regarding the position and orientation of the mobile entity, and a dataset generation process for generating a learning dataset including the superimposed image and label information.SELECTED DRAWING: Figure 11

Description

Technical Field

[0006] , ,

[0005] , ,

[0001] The present disclosure relates to a method, an apparatus, a computer program, and a learning dataset.

Background Art

[0002] A technique for detecting the position of a vehicle in motion using a captured image of a camera located outside the vehicle is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When detecting a moving object such as a vehicle from a captured image of a camera using artificial intelligence, it is necessary to prepare a large amount of data for training the artificial intelligence. Therefore, a technique for reducing the labor required to prepare training data is desired.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, a method is provided. The method includes a moving path acquisition step of acquiring a moving path of a moving object, a superimposed image generation step of generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path, a label information acquisition step of acquiring label information regarding the position and orientation of the moving object, and a dataset generation step of generating a learning dataset including the superimposed image and the label information. This method generates a training dataset that includes superimposed images, thus reducing the effort required to prepare the training dataset. (2) The above-described method may further include a step of performing predetermined processing on the image or three-dimensional data of the moving environment prior to the superimposed image generation step. This method allows for the generation of a training dataset containing superimposed images of the desired state. (3) The above-described method may further include a step of performing predetermined processing on the image or three-dimensional data of the moving object prior to the step of generating the superimposed image. This method allows for the generation of a training dataset containing superimposed images of the desired state. (4) The above-described method may further include a step of performing predetermined processing on the superimposed image. This method allows for the generation of a training dataset containing superimposed images of the desired state. (5) In the above-described method, the processing may include at least one of adding or changing the light irradiated onto the moving body and adding or changing the shadow of the moving body. This method allows for altering the appearance of moving objects represented in superimposed images. (6) In the above-described method, the processing may include changing at least one of brightness, saturation, and contrast. This method allows for altering the appearance of moving objects and environments represented in superimposed images. (7) In the above-described method, the superimposed image generation step may be performed multiple times, and in at least two of the multiple superimposed images generated by performing the superimposed image generation step multiple times, the position of the moving body in a direction parallel to the movement path may be different. This method allows for the generation of multiple superimposed images in which the positions of the moving objects in directions parallel to the movement path are different from each other. (8) In the above-described method, the superimposed image generation step may be performed multiple times, and in at least two of the multiple superimposed images generated by performing the superimposed image generation step multiple times, the position of the moving body in a direction perpendicular to the movement path may be different. This method allows for the generation of multiple superimposed images in which the positions of the moving objects in directions perpendicular to the movement path are different from each other. (9) In the above-described method, the superimposed image generation step may be performed multiple times, and in at least two of the multiple superimposed images generated by performing the superimposed image generation step multiple times, the orientation of the moving body with respect to the movement path may be different. This method allows for the generation of multiple superimposed images in which the orientation of the moving objects relative to the movement path differs from one another. (10) In the above-described method, the superimposed image generation step may be performed multiple times, and the color of the moving body may be different in at least two of the multiple superimposed images generated by performing the superimposed image generation step multiple times. This method allows for the generation of multiple superimposed images in which the moving objects have different colors. (11) In the above-described method, the label information acquisition step may be performed by acquiring the label information from an image or information associated with the three-dimensional data of the moving object. This method eliminates the need to separately prepare information regarding the position and orientation of the moving object. (12) The above-described method may further include a learning step in which machine learning is performed using the set of training datasets, including the training dataset. This method allows for machine learning to be performed using training datasets that include superimposed images. (13) The above-described method may further include a step of acquiring information regarding the position and orientation of a moving object moving through the moving environment, using the trained model generated by the training step and an image of the moving object moving through the moving environment. This method allows for the acquisition of information regarding the position and orientation of a moving object while it is moving through a moving environment, using a pre-trained model. (14) In the above-described method, the superimposed image generation step is performed multiple times and further comprises a learning step in which machine learning is performed using a group of learning datasets including multiple learning datasets, and a specific position is determined among multiple positions on the movement path in which the accuracy of the output of the trained model generated by the learning step is relatively low, and the number of learning datasets corresponding to the specific position may be increased compared to other positions. This method can improve the accuracy of the output of the trained model at a specific location. (15) In the above-described method, the superimposed image generation step is performed multiple times and further comprises a learning step in which machine learning is performed using a group of learning datasets including multiple learning datasets, and a specific location is determined among multiple locations on the movement path where the accuracy of the output of the trained model generated by the learning step is relatively low, and a learning dataset corresponding to the environment of the specific location is generated. This method can improve the accuracy of the output of the trained model at a specific location. (16) According to a second embodiment of the present disclosure, an apparatus is provided. The apparatus comprises: a movement path acquisition unit for acquiring the movement path of a moving object; a superimposed image generation unit for generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing the movement environment including the movement path; a label information acquisition unit for acquiring label information relating to the position and orientation of the moving object; and a dataset generation unit for generating a training dataset including the superimposed image and the label information. This type of device generates training datasets that include superimposed images, thus reducing the effort required to prepare the training dataset. (17) According to a third embodiment of the present disclosure, a computer program is provided. This computer program enables the computer to perform the following functions: acquire an image or three-dimensional data representing a moving object, or an image or three-dimensional data representing the moving environment of the moving object; display a screen on which processing conditions for processing to be applied to the acquired image or three-dimensional data can be selected; and perform the processing on the acquired image or three-dimensional data according to the selected processing conditions. This type of computer program can generate superimposed images according to processing conditions selected by the user. (18) In a computer program of the above form, the processing may include changing at least one of brightness, saturation, and contrast. This type of computer program can generate superimposed images of a desired state. (19) In the computer program of the above form, the processing may include at least one of adding or changing the light irradiated onto the moving body and adding or changing the shadow of the moving body. This type of computer program can generate superimposed images of a desired state. (20) According to a fourth embodiment of the present disclosure, a computer program is provided. This computer program enables the computer to perform the following functions: acquire an image or three-dimensional data representing a moving object and an image or three-dimensional data representing the moving environment of the moving object; display a screen on which the computer can select superposition conditions for superimposing the image or three-dimensional data representing the moving object and the image or three-dimensional data representing the moving environment; and generate a superimposed image by superimposing the image or three-dimensional data representing the moving object and the image or three-dimensional data representing the moving environment according to the selected superposition conditions. This type of computer program can generate superimposed images according to superposition conditions selected by the user. (21) In the computer program of the above-described form, the superimposition condition may include at least one of the position and orientation of the moving object with respect to the moving environment. According to the computer program of this form, a superimposed image in a desired state can be generated. (22) According to the fifth form of the present disclosure, a learning dataset used for machine learning is provided. This learning dataset includes an image or three-dimensional data representing a moving object, a superimposed image generated by superimposing an image or three-dimensional data representing the moving environment of the moving object, and information regarding the position and orientation of the moving object. According to the learning dataset of this form, the learning dataset can be used to generate a learned model for acquiring information regarding the position and orientation of a moving object. The present disclosure can also be realized in various forms other than methods, apparatuses, computer programs, and learning datasets. For example, it can be realized in the form of a system, a recording medium on which a computer program is recorded, and the like.

Brief Description of Drawings

[0007] [Figure 1] An explanatory diagram showing the configuration of the system according to the first embodiment. [Figure 2] An explanatory diagram showing the configuration of the vehicle according to the first embodiment. [Figure 3] An explanatory diagram showing the configuration of the server device according to the first embodiment. [Figure 4] A flowchart showing the content of the driverless operation process according to the first embodiment. [Figure 5] A flowchart showing an example of a method for generating shape data and a method for acquiring the position of a vehicle using the shape data. [Figure 6] A schematic diagram showing examples of various images when the method shown in FIG. 5 is executed. [Figure 7] An explanatory diagram for explaining the details of the coordinate point calculation step. [Figure 8] An explanatory diagram for explaining a method for calculating a base coordinate point. [Figure 9]The first explanatory diagram for detailing the position change process. [Figure 10] A second explanatory diagram illustrating the details of the position change process. [Figure 11] A flowchart illustrating the process for generating a pre-trained model. [Figure 12] A schematic diagram illustrating how a factory model and a vehicle model are superimposed. [Figure 13] A schematic diagram illustrating the training datasets. [Figure 14] A schematic diagram illustrating the selection screen. [Figure 15] An explanatory diagram showing the configuration of the vehicle according to the second embodiment. [Figure 16] An explanatory diagram showing the configuration of the server device according to the second embodiment. [Figure 17] A flowchart illustrating the contents of the unmanned operation process in the second embodiment. [Figure 18] A first explanatory diagram schematically showing a trained model of another embodiment. [Figure 19] A second explanatory diagram schematically shows a trained model of another embodiment. [Modes for carrying out the invention]

[0008] A. First Embodiment: Figure 1 is an explanatory diagram showing the configuration of the system 50 in the first embodiment. Figure 2 is an explanatory diagram showing the configuration of the vehicle 100 in the first embodiment. Figure 3 is an explanatory diagram showing the configuration of the server device 200 in the first embodiment. As shown in Figure 1, in this embodiment, the system 50 comprises a mobile vehicle 100, a server device 200, and at least one external camera 300.

[0009] In this disclosure, “mobile object” means an object that can move. A mobile object may be, for example, a vehicle or an electric vertical take-off and landing aircraft (a so-called flying car). A vehicle may be a wheeled vehicle or a tracked vehicle. In other words, the form of the vehicle is not particularly limited, and a vehicle may be, for example, a passenger car, truck, bus, motorcycle, car, tank, construction vehicle, etc. A vehicle may be an electric vehicle (BEV: Battery Electric Vehicle), a gasoline vehicle, a hybrid vehicle (HEV: Hybrid Electric Vehicle), a fuel cell vehicle (FCEV: Fuel Cell Electric Vehicle), etc. If the mobile object is not a vehicle, the expressions “vehicle” or “car” in this disclosure may be replaced with “mobile object” as appropriate, and the expression “driving” may be replaced with “moving” as appropriate.

[0010] Vehicle 100 is configured to operate autonomously. Here, "autonomous operation" means operation without driver operation by an occupant on board Vehicle 100. "Driver operation" means operation relating to at least one of the following: "going," "turning," or "stopping" of Vehicle 100. Autonomous operation is achieved by automatic or manual remote control using a device located outside Vehicle 100, or by autonomous control of Vehicle 100. Vehicle 100 operating autonomously may have an occupant on board who does not perform driver operation. An occupant who does not perform driver operation includes, for example, a person simply sitting in a seat on Vehicle 100, or a person on board Vehicle 100 performing an action other than driver operation. An action other than driver operation includes, for example, assembling parts for Vehicle 100, inspecting Vehicle 100, or operating switches provided on Vehicle 100. Note that operation by an occupant is sometimes called "manned operation."

[0011] In this disclosure, "remote control" includes "fully remote control," in which all operations of the vehicle 100 are completely determined from outside the vehicle 100, and "partial remote control," in which some operations of the vehicle 100 are determined from outside the vehicle 100. Furthermore, "autonomous control" includes "fully autonomous control," in which the vehicle 100 autonomously controls its own operations without receiving any information from a device located outside the vehicle 100, and "partial autonomous control," in which the vehicle 100 autonomously controls its own operations using information received from a device located outside the vehicle 100.

[0012] In this embodiment, system 50 is used in a factory FC where vehicle 100 is manufactured. The reference coordinate system of the factory FC is the global coordinate system GC. That is, any position within the factory FC can be represented by X, Y, Z coordinates in the global coordinate system GC. The factory FC comprises a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a track TR on which vehicle 100 can travel. Multiple external cameras 300 are installed along the track TR in the factory FC. The external cameras 300 are equipped with communication devices (not shown) and can communicate with a server device 200 by wired or wireless communication. The position and orientation of each external camera 300 in the factory FC are pre-adjusted. Vehicle 100 moves from the first location PL1 to the second location PL2 along the track TR by unmanned operation.

[0013] As shown in Figure 2, the vehicle 100 includes a vehicle control device 110 for controlling various parts of the vehicle 100, an actuator group 120 driven under the control of the vehicle control device 110, and a communication device 130 for communicating with a server device 200 via wireless communication. The actuator group 120 includes at least one actuator. In this embodiment, the actuator group 120 includes an actuator for a drive system to accelerate the vehicle 100, an actuator for a steering system to change the direction of travel of the vehicle 100, and an actuator for a braking system to decelerate the vehicle 100. The drive system includes a battery, a drive motor driven by the battery's power, and wheels rotated by the drive motor. The drive motor is included in the actuator of the drive system.

[0014] The vehicle control device 110 is comprised of a computer comprising a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected via the internal bus 114 to enable bidirectional communication. The input / output interface 113 is connected to an actuator group 120 and a communication device 130.

[0015] The processor 111 functions as a driving control unit 115 by executing a computer program PG1 pre-stored in memory 112. The driving control unit 115 drives the vehicle 100 by controlling the actuator group 120. If there is a passenger in the vehicle 100, the driving control unit 115 can drive the vehicle 100 by controlling the actuator group 120 in accordance with the passenger's operation. In this embodiment, the driving control unit 115 can drive the vehicle 100 by controlling the actuator group 120 in accordance with the driving control signal received from the server device 200, regardless of whether there is a passenger in the vehicle 100 or not. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of the acceleration of the vehicle 100, or in addition to the acceleration of the vehicle 100.

[0016] As shown in Figure 3, the server device 200 is composed of a computer comprising a processor 201, memory 202, input / output interface 203, and internal bus 204. The processor 201, memory 202, and input / output interface 203 are connected via the internal bus 204 to enable bidirectional communication. A communication device 205 for communicating with the vehicle 100 via wireless communication is connected to the input / output interface 203. In this embodiment, the communication device 205 can communicate with the external camera 300 via wired or wireless communication. In addition to the communication device 205, an input device 206 and a display device 207 are connected to the input / output interface 203. The input device 206 is, for example, a mouse or keyboard. The display device 207 is, for example, a liquid crystal display.

[0017] The processor 201 functions as a remote control unit 210, a driving path acquisition unit 220, a driving environment information acquisition unit 230, a vehicle information acquisition unit 240, a superimposed image generation unit 250, a processing unit 260, a label information acquisition unit 270, a dataset generation unit 280, and a learning unit 290 by executing a computer program PG2 pre-stored in memory 202. The driving path acquisition unit 220 is sometimes referred to as the travel path acquisition unit, the driving environment information acquisition unit 230 is sometimes referred to as the travel environment information acquisition unit, and the vehicle information acquisition unit 240 is sometimes referred to as the mobile object information acquisition unit.

[0018] The remote control unit 210 drives the vehicle 100 by remotely controlling it. In this embodiment, the remote control unit 210 remotely controls the vehicle 100 so that it travels along a reference path RR that is pre-stored in the memory 202. The remote control unit 100 uses a pre-trained model DM pre-stored in the memory 202 and captured images acquired from the external camera 300 to acquire vehicle position information including the position and orientation of the vehicle 100, and remotely controls the vehicle 100 using the vehicle position information and the reference path RR. A pre-trained model DM is prepared in advance for each external camera 300.

[0019] The travel path acquisition unit 220 acquires the travel path of the vehicle 100. In this embodiment, the travel path acquisition unit 220 acquires a reference path RR that is pre-stored in the memory 202 as the travel path of the vehicle 100.

[0020] The driving environment information acquisition unit 230 acquires driving environment information that represents the appearance of the driving environment of the vehicle 100. The driving environment information is an image representing the appearance of the driving environment of the vehicle 100, or three-dimensional data representing the appearance of the driving environment of the vehicle 100. In this embodiment, the driving environment information acquisition unit 230 acquires a factory model MF that is pre-stored in the memory 202 as driving environment information. The factory model MF is three-dimensional data that represents the appearance of the factory FC, which is the driving environment of the vehicle 100. The appearance of the factory FC includes the appearance of the factory FC building, the appearance of various equipment installed inside and outside the factory FC building, and the appearance of the road surface of the track TR inside the factory FC.

[0021] The vehicle information acquisition unit 240 acquires vehicle information representing the appearance of the vehicle 100. The vehicle information is an image representing the appearance of the vehicle 100, or three-dimensional data representing the appearance of the vehicle 100. In this embodiment, the vehicle information acquisition unit 240 acquires a vehicle model MV, which is pre-stored in the memory 202, as vehicle information. The vehicle model MV is three-dimensional data representing the appearance of the vehicle 100.

[0022] The vehicle model MV and the factory model MF are generated, for example, using 3D CAD software or 3D CG software. The vehicle model MV and the factory model MF may be generated on the server device 200 or on a computer different from the server device 200. Preferably, the appearance of the vehicle model MV is a reproduction of the appearance of the vehicle 100 as accurately as possible, and preferably, the appearance of the factory model MF is a reproduction of the appearance of the factory FC as accurately as possible.

[0023] The superimposed image generation unit 250 generates a superimposed image by superimposing vehicle information and driving environment information. In this embodiment, the superimposed image generation unit 250 generates a superimposed image by superimposing the vehicle model MV and the factory model MF, and outputs the generated superimposed image.

[0024] The processing unit 260 processes at least one of the vehicle information, driving environment information, and superimposed image. The processing may include at least one of adding or changing the light illuminating the vehicle 100 and adding or changing the shadows of the vehicle 100. The processing may include changing at least one of the brightness, saturation, and contrast. In this embodiment, the processing unit 260 can change the color and texture of at least a part of the vehicle 100 represented in the vehicle model MV by processing the vehicle model MV. The processing unit 260 can change the color and texture of at least a part of the factory FC represented in the factory model MF by processing the factory model MF. The processing unit 260 can change at least one of the brightness, saturation, and contrast of at least a part of the superimposed image by processing the superimposed image.

[0025] The label information acquisition unit 270 acquires label information relating to at least one of the position and orientation of the vehicle 100. The label information is used as the correct label in supervised machine learning.

[0026] The dataset generation unit 280 generates a training dataset that includes superimposed images and label information associated with those superimposed images.

[0027] The learning unit 290 generates a trained model DM by performing machine learning using a set of training datasets that include at least one training dataset.

[0028] Figure 4 is a flowchart illustrating the contents of the unmanned operation process in this embodiment. In this embodiment, the unmanned operation process is performed by the remote control unit 210 of the server device 200 and the driving control unit 115 of the vehicle 100. In step S110, the remote control unit 210 acquires vehicle position information of the vehicle 100 using the captured image output from the external camera 300 and the trained model DM. The vehicle position information is the position information that forms the basis for generating the driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC.

[0029] In detail, in step S110, the remote control unit 210 obtains shape data relating to the contour of the vehicle 100 from the captured image using, for example, a trained model DM utilizing artificial intelligence, calculates the coordinates of the positioning points of the vehicle 100 in the coordinate system of the captured image, i.e., the local coordinate system of the factory FC, and obtains the position of the vehicle 100 by converting the calculated coordinates to coordinates in the global coordinate system GC. In this embodiment, the trained model DM is generated in advance by a trained model generation method described later and stored in the memory 202. The remote control unit 210 obtains the orientation of the vehicle 100 by, for example, using the optical flow method to estimate the direction of the vehicle 100's movement vector calculated from the positional changes of the vehicle 100's feature points between frames of the captured image.

[0030] In step S120, the remote control unit 210 determines the next target location to which the vehicle 100 should go. In this embodiment, the target location is represented by X, Y, Z coordinates in the global coordinate system GC. The memory 202 pre-stores a reference route RR, which is the route that the vehicle 100 should travel. The route is represented by a node indicating the starting point, nodes indicating waypoints, a node indicating the destination, and links connecting each node. The remote control unit 210 uses the vehicle position information and the reference route RR to determine the next target location to which the vehicle 100 should go. The remote control unit 210 determines the target location on the reference route RR beyond the current location of the vehicle 100.

[0031] In step S130, the remote control unit 210 generates a driving control signal to drive the vehicle 100 toward the determined target position. The remote control unit 210 calculates the vehicle's speed from the change in the vehicle's position and compares the calculated speed with a predetermined target speed for the vehicle 100. Overall, the remote control unit 210 determines the acceleration so that the vehicle 100 accelerates if the speed is lower than the target speed, and determines the acceleration so that the vehicle 100 decelerates if the speed is higher than the target speed. If the vehicle 100 is located on the reference path RR, the remote control unit 210 determines the acceleration and steering angle so that the vehicle 100 does not deviate from the reference path RR, and if the vehicle 100 is not located on the reference path RR, in other words, if the vehicle 100 has deviated from the reference path RR, the remote control unit 210 determines the acceleration and steering angle so that the vehicle 100 returns to the reference path RR.

[0032] In step S140, the remote control unit 210 transmits the generated driving control signal to the vehicle 100. The remote control unit 210 repeats the process of acquiring the position of the vehicle 100, determining the target position, generating the driving control signal, and transmitting the driving control signal at predetermined intervals.

[0033] In step S150, the driving control unit 115 receives a driving control signal transmitted from the server device 200. In step S160, the driving control unit 115 controls the actuator group 120 using the driving control signal received from the server device 200, thereby driving the vehicle 100 at the acceleration and steering angle indicated in the driving control signal. The driving control unit 115 repeats the reception of the driving control signal and the control of the actuator group 120 at predetermined intervals. According to the system 50 in this embodiment, the vehicle 100 can be driven by remote control, and therefore the vehicle 100 can be moved without using transport equipment such as cranes or conveyors.

[0034] The method for acquiring vehicle position information in step S110 of Figure 4 will be explained in more detail. Figure 5 is a flowchart showing an example of a method for generating shape data and a method for acquiring the position of vehicle 100 using the shape data. Figure 6 is a schematic diagram showing examples of various images Im1 to Im6 when the method shown in Figure 5 is executed. Figure 6 shows the step numbers corresponding to each step in Figure 5. In this embodiment, the positioning point 10e of vehicle 100 is the rear end on the left side of vehicle 100. Note that the positioning point 10e of vehicle 100 may be other than the rear end on the left side of vehicle 100.

[0035] In the image acquisition process (step S410) shown in Figure 5, the remote control unit 210 acquires the original image Im1 output from the external camera 300. In the distortion correction process (step S420), the remote control unit 210 corrects the distortion of the original image Im1 to generate a corrected image Im2 as a processed image. In the rotation processing process (step S430), the remote control unit 210 rotates the corrected image Im2. As a result, the remote control unit 210 generates a rotated image Im3 as a processed image. In the cropping process (step S440), the remote control unit 210 removes unnecessary areas A2 from the rotated image Im3, excluding the necessary area A1 which consists of the vehicle 100 and the area surrounding the vehicle 100. As a result, the remote control unit 210 generates a processed image Im4 as a processed image. Each step from step S420 to step S440 is a pre-processing step to improve the detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4. Therefore, at least one of the steps from step S420 to step S440 may be omitted.

[0036] In the detection step (step S450), the remote control unit 210 inputs the processed image Im4 to the trained model DM. As a result, the remote control unit 210 detects the vehicle 100 from the processed image Im4 and obtains the first mask image Im5 as shape data Da. The first mask image Im5 is an image obtained by adding a mask region Ms to the processed image Im4 by masking the region representing the vehicle 100 from each region that constitutes the processed image Im4.

[0037] In the perspective transformation process (step S460), the remote control unit 210 generates a second mask image Im6 by performing a perspective transformation on the first mask image Im5. The remote control unit 210 transforms the first mask image Im5 into a bird's-eye view image viewed from above a vehicle 100 that is approximately perpendicular to the road surface Rs, for example, using predetermined perspective transformation parameters. The perspective transformation parameters are, for example, parameters related to the position information and internal parameters of the external camera 300 obtained by calibration. As a result, the remote control unit 210 generates a second mask image Im6, represented in image coordinates, from the first mask image Im5, represented in camera coordinates. The camera coordinate system is a coordinate system with the focal point of the external camera 300 as the origin and having coordinate axes indicated by the Xc axis and the Yc axis perpendicular to the Xc axis. The image coordinate system is a coordinate system with a point on the image plane as the origin and having coordinate axes indicated by the Xi axis and the Yi axis perpendicular to the Xi axis.

[0038] In the coordinate point calculation process (step S470), the remote control unit 210 calculates image coordinate points indicating the position of the vehicle 100 in the image coordinate system. Figure 7 is an explanatory diagram illustrating the details of the coordinate point calculation process. To calculate the image coordinate points, the remote control unit 210 first calculates the base coordinate points P0 from the first bounding rectangle R1 set in the mask region Ms in the first mask image Im5, which is the image before perspective transformation. Figure 8 is an explanatory diagram illustrating the method for calculating the base coordinate points P0. To calculate the base coordinate points P0, the remote control unit 210 sets the base bounding rectangle R0 for the mask region Ms in the first mask image Im5. Next, the remote control unit 210 rotates the first mask image Im5 by the required amount, with the centroid C of the mask region Ms as the rotation center, so that the direction of the movement vector V of the vehicle 100 corresponding to the mask region Ms in the first mask image Im5 points in a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device that displays the first mask image Im5. Next, the remote control unit 210 sets a first circumscribing rectangle R1 with respect to the mask region Ms of the rotated first mask image Im5 such that its longer side is parallel to the movement vector V. Next, the remote control unit 210 rotates the first mask image Im5, to which the first circumscribing rectangle R1 has been added, in the opposite direction by the above amount, using the centroid C of the mask region Ms as the rotation center. As a result, the remote control unit 210 sets the coordinate point of one of the four vertices of the first circumscribing rectangle R1 that has the coordinate closest to the positioning point 10e of the vehicle 100 as the base coordinate point P0.

[0039] Next, as shown in Figure 7, the remote control unit 210 performs a perspective transformation on the first mask image Im5 after it has been rotated in the reverse direction, in other words, on the first mask image Im5 after the base coordinate point P0 has been calculated. As a result, the remote control unit 210 sets the coordinate point corresponding to the base coordinate point P0 in the first bounding rectangle R1, which has been deformed by the perspective transformation, as the first coordinate point P1.

[0040] Next, the remote control unit 210 sets a second bounding rectangle R2 for the mask region Ms in the second mask image Im6, which is obtained by perspective transforming the first mask image Im5. Then, the remote control unit 210 sets the vertex of the second bounding rectangle R2 that is at the same position as the first coordinate point P1 as the second coordinate point P2. In other words, since the first coordinate point P1 and the second coordinate point P2 are coordinate points that indicate the same position, they are correlated with each other.

[0041] Next, the remote control unit 210 performs a correction by replacing the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2, depending on the relative magnitudes of the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction (Xi1 > Xi2), the remote control unit 210 replaces the coordinate value Xi1 of the first coordinate point P1 in the Xi direction with the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. If the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction (Yi1 > Yi2), the remote control unit 210 replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. In the example shown in Figure 7, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Also, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has the coordinates (Xi2, Yi1). In this way, the remote control unit 210 calculates the image coordinate point P3 that indicates the position of the vehicle 100 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2.

[0042] In the position conversion process (step S480), the remote control unit 210 converts the image coordinate point P3 into a vehicle coordinate point to calculate the vehicle coordinate point that indicates the position of the vehicle 100's positioning point 10e in the global coordinate system GC. The remote control unit 210 converts the image coordinate point P3 into a vehicle coordinate point using the relational equations (1) to (3) described later, which include the vehicle coordinate point as the objective variable and the image coordinate point P3, imaging parameters, and vehicle parameters as explanatory variables. The imaging parameters are parameters relating to the distance of the external camera 300 from a predetermined reference point. In this embodiment, the imaging parameters are the height H of the external camera 300 from the road surface Rs (Figure 9 described later). The vehicle parameters are parameters relating to the distance of the vehicle 100's positioning point 10e from the reference point. In this embodiment, the vehicle parameters are the height h of the vehicle 100's positioning point 10e from the road surface Rs (Figure 9 described later).

[0043] Figure 9 is the first explanatory diagram for detailing the position transformation process. Figure 9 shows the state of the vehicle 100 as viewed from the left side. Figure 10 is the second explanatory diagram for detailing the position transformation process. Figure 10 shows the state of the vehicle 100 as viewed from the roof side. The global coordinate system GC shown in Figures 9 and 10 is a coordinate system that has a fixed coordinate point Pf, which indicates an arbitrary reference position on the road surface Rs, as its origin, and has coordinate axes indicated by the Xg axis and the Yg axis which is orthogonal to the Xg axis. The imaging coordinate point Pc is a coordinate point that indicates the position in the global coordinate system GC of the external camera 300 that output the original image Im1 used to calculate the image coordinate point P3. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in memory 202, for example.

[0044] As shown in Figure 9, let Do be the observed distance on the XgYg plane between the position of the external camera 300 and the position of the vehicle 100 (image coordinate point P3), let ΔD be the observation error, let H be the height [m] of the external camera 300 from the road surface Rs as an imaging parameter, and let h be the height [m] of the positioning point 10e of the vehicle 100 from the road surface Rs as a vehicle parameter. In this case, the observation error ΔD is expressed by the following equation (1). ΔD = h / H × Do ···(1) In other words, the larger the observation distance Do, the larger the observation error ΔD becomes.

[0045] If the actual distance between the position of the external camera 300 and the position of the vehicle 100's positioning point 10e is defined as the first distance D, then the first distance D is expressed by the following equation (2). D = Do × (1 - h / H) ... (2) In other words, the first distance D is determined by the observation distance Do, the height H of the external camera 300 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 100 as a vehicle parameter.

[0046] As shown in Figure 10, if the estimated distance Dp is the distance between the reference position and the estimated position of the vehicle 100, and the second distance Dt is the actual distance between the reference position and the vehicle 100, then the second distance Dt is expressed by the following equation (3). Dt = Dp × (1 - h / H) ... (3)

[0047] Here, the estimated distance Dp can be calculated using the third distance Dc, which is the actual distance obtained from the fixed coordinate point Pf and the imaging coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the remote control unit 210 calculates the vehicle coordinate point Pv using the second distance Dt, which is obtained by correcting the estimated distance Dp using the above equation (3), and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point that indicates the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC, and therefore corresponds to the position of the vehicle 100 in real space.

[0048] Figure 11 is a flowchart illustrating the contents of the trained model generation method, including the training dataset generation method, in this embodiment. Figure 12 is a schematic diagram illustrating how the factory model MF and the vehicle model MV are superimposed. Figure 13 is a schematic diagram illustrating the training dataset group DG. In this embodiment, the trained model generation method is executed by the processor 201 of the server device 200. First, in step S510, the travel path acquisition unit 220 acquires the travel path of the vehicle 100 in real space. In this embodiment, the travel path acquisition unit 220 acquires the reference path RR stored in memory 202 as the travel path of the vehicle 100 in real space. Step S510 is sometimes referred to as the travel path acquisition process or the movement path acquisition process.

[0049] In step S520, the driving environment information acquisition unit 230 acquires driving environment information representing the appearance of the driving environment of the vehicle 100, and the vehicle information acquisition unit 240 acquires vehicle information representing the appearance of the vehicle 100. In this embodiment, the driving environment information acquisition unit 230 acquires the factory model MF, which is pre-stored in the memory 202, as driving environment information, and the vehicle information acquisition unit 240 acquires the vehicle model MV, which is pre-stored in the memory 202, as vehicle information.

[0050] In step S530, the processing unit 260 performs necessary processing on at least one of the vehicle model MV and the factory model MF as a pre-processing step. The processing unit 260 can change the color and texture of at least a part of the vehicle 100 represented in the vehicle model MV by processing the vehicle model MV, for example. For example, the processing unit 260 may change the color of the door mirrors or roof of the vehicle model. For example, the processing unit 260 may change the texture of the vehicle model MV from metallic to matte by texture mapping. By changing the texture of the vehicle model MV, it is possible to change the highlights and shadows that occur on the surface of the vehicle model MV due to the light irradiated onto the vehicle model MV from the virtual light source VL in the rendering described later. The processing unit 260 can change the color and texture of at least a part of the factory FC represented in the factory model MF by processing the factory model MF, for example. Note that the process in step S530 may be omitted.

[0051] In step S540, the superimposed image generation unit 250 generates a superimposed image GS by superimposing the vehicle model MV and the factory model MF. Specifically, as shown in Figure 12, the superimposed image generation unit 250 superimposes the vehicle model MV and the factory model MF by placing the factory model MF and the vehicle model MV in the virtual space VS. In the following description, the three-dimensional data obtained by superimposing the vehicle model MV and the factory model MF is called the superimposed model MS. The superimposed image generation unit 250 generates the superimposed image GS from the superimposed model MS by rendering. Rendering is the process of generating a two-dimensional image from three-dimensional data. The rendering method used to generate the superimposed image GS is preferably physically based rendering. Physically based rendering can reproduce the behavior of light in real space as accurately as possible. In addition to the vehicle model MV and the factory model MF, a virtual viewpoint VC and a virtual light source VL are placed in the virtual space VS. The virtual viewpoint VC is positioned such that the relative position and orientation between the factory model MF and the virtual viewpoint VC in the virtual space VS are the same as the relative position and orientation between the factory FC and the external camera 300 in the real space. The virtual light source VL is preferably positioned at the location of the light source in the real space. Multiple virtual light sources VL may be positioned in the virtual space VS. For example, if multiple lighting fixtures are positioned in the factory FC in the real space, a virtual light source VL may be positioned at the location of each lighting fixture. The superimposed image GS represents the superimposed model MS as seen from the virtual viewpoint VC. The superimposed image GS represents the shading caused by the light illuminating the superimposed model MS from the virtual light source VL. Step S540 is sometimes referred to as the superimposed image generation process.

[0052] In step S550, the processing unit 260 performs post-processing on the superimposed image GS as necessary. For example, by processing the superimposed image GS, the processing unit 260 can change at least one of the brightness, saturation, and contrast of at least a portion of the superimposed image GS. Note that the step S550 may be omitted.

[0053] In step S560, the label information acquisition unit 270 acquires label information LB relating to the position and orientation of the vehicle 100. In this embodiment, the label information acquisition unit 270 acquires information about a mask region Ms for acquiring the position and orientation of the vehicle 100 as label information LB. The label information acquisition unit 270 can acquire the mask region Ms, for example, by acquiring information about the pixels on which the vehicle model MV is drawn among a plurality of pixels constituting the superimposed image GS. Step S560 is sometimes referred to as the label information acquisition process.

[0054] In step S570, the dataset generation unit 280 generates a training dataset DS that includes the superimposed image GS and label information LB. As shown in Figure 13, in this embodiment, the training dataset DS contains one superimposed image GS and one copy of label information LB corresponding to the superimposed image GS. Step S570 is sometimes referred to as the training dataset generation process.

[0055] In step S580, the dataset generation unit 280 determines whether or not to generate a different pattern of training dataset DS. For example, if the generation of a predetermined number of training datasets DS has not been completed, the dataset generation unit 280 determines to generate a different pattern of training dataset DS, and if the generation of a predetermined number of training datasets DS has been completed, it determines not to generate a different pattern of training dataset DS.

[0056] If it is determined in step S580 to generate a different pattern of training dataset, the process from step S530 to step S580 is executed to generate a different pattern of training dataset DS. The process from step S530 to step S580 is repeated until it is determined in step S580 not to generate a different pattern of training dataset DS. Different patterns of training dataset DS contain different superimposed images GS. In this embodiment, the superimposed image generation unit 250 makes the position of the vehicle 100 in a direction parallel to the travel path different in at least two of the multiple superimposed images GS. In addition, in this embodiment, the superimposed image generation unit 250 makes the position of the vehicle 100 in a direction perpendicular to the travel path different in at least two of the multiple superimposed images GS. The direction perpendicular to the travel path here refers to the direction parallel to the road surface of the road TR, not the direction perpendicular to the road surface of the road TR. In addition, in this embodiment, the superimposed image generation unit 250 makes the orientation of the vehicle 100 relative to the travel path different in at least two of the multiple superimposed images GS. The superimposed image generation unit 250 makes the position and orientation of the vehicle 100 in the superimposed image GS different by changing the position and orientation of the vehicle model MV relative to the factory model MF. For example, the superimposed image generation unit 250 generates a total of 10 × 10 × 10 = 1000 patterns of superimposed images by making 10 patterns by making the position of the vehicle 100 different in the direction parallel to the travel path, 10 patterns by making the position of the vehicle 100 different in the direction perpendicular to the travel path, and 10 patterns by making the orientation of the vehicle 100 different at each position. The processing unit 260 may make at least some of the colors different in at least two of the multiple superimposed images GS. In this case, the superimposed image generation unit 250 generates 1000 superimposed images, for example, 1000 patterns with different positions and orientations of the vehicle 100, and then 10 patterns with different colors, resulting in 1000 × 10 = 10000 patterns of superimposed images. The label information acquisition unit 270 acquires label information corresponding to each superimposed image GS.

[0057] In step S590, the learning unit 290 generates a trained model DM by performing machine learning using a group of training datasets DG, which includes multiple training datasets DS. In this embodiment, the learning unit 290 stores the generated trained model DM in memory 202. The trained model DM can be, for example, a convolutional neural network (CNN) that implements either semantic segmentation or instance segmentation. During CNN training, for example, the parameters of the CNN are updated by backpropagation to reduce the error between the output result of the trained model DM and the correct label. Step S590 is sometimes referred to as the training process.

[0058] Figure 14 is a schematic diagram illustrating the selection screen SW. The selection screen SW includes an overlay condition selection area RS for selecting overlay conditions and a processing condition selection area RK for selecting processing conditions. Overlay conditions are the conditions for overlaying vehicle information and driving environment information. In step S540 of Figure 11, the overlay image generation unit 250 overlays the vehicle information and driving environment information according to the overlay conditions selected in the overlay condition selection area RS. Overlay conditions include, for example, conditions relating to the position of the vehicle model MV relative to the factory model MF, and conditions relating to the orientation of the vehicle model MV relative to the factory model MF. Processing conditions are the conditions for processing at least one of the vehicle information, driving environment information, and the overlay image. In steps S530 and S550 of Figure 11, the processing unit 260 processes at least one of the vehicle information, driving environment information, and the overlay image according to the processing conditions selected in the processing condition selection area RK. Processing conditions include, for example, conditions relating to the color of the vehicle 100. The processing conditions may include conditions related to the weather at the factory FC. The processing condition selection area RK may be configured to allow selection between sunny and rainy weather, for example. If rainy weather is selected, the processing unit 260 may darken the virtual light source VL compared to when sunny weather is selected. Alternatively, the processing condition selection area RK may be configured to allow selection of the longitude and latitude, time, and date of the factory FC. Depending on the conditions selected in the processing condition selection area RK, the processing unit 260 may acquire weather information regarding the position of the sun from an external source and change the position of the virtual light source VL based on the weather information. In this case, a decrease in the detection accuracy of the vehicle 100 by the trained model MD due to changes in weather can be suppressed. Furthermore, the processing condition selection area RK may be configured to allow selection of conditions such as the size of the raindrops, the spacing between raindrops, the falling speed of the raindrops, and whether it is water or ice. In this case, a decrease in the detection accuracy of the vehicle 100 by the trained model MD when occlusion occurs on the vehicle 100 due to raindrops can be suppressed. By selecting the falling speed of the raindrops, it becomes possible to select the length of the afterimage of the raindrops represented in the superimposed image.

[0059] As described above, the system 50 in this embodiment generates a superimposed image GS by superimposing the vehicle model MV and the factory model MF, and the dataset generation unit 280 generates a training dataset DS that includes the superimposed image GS. Therefore, the effort of acquiring training images by, for example, imaging the vehicle 100 located in the factory FC in real space is eliminated. Consequently, the effort required to generate the training dataset DS can be reduced. In addition, phenomena that occur infrequently in real space can be represented by the superimposed image GS.

[0060] B. Second Embodiment: Figure 15 is an explanatory diagram showing the configuration of the vehicle 100 in the second embodiment. Figure 16 is an explanatory diagram showing the configuration of the server device 200 in the second embodiment. Figure 17 is a flowchart showing the contents of the unmanned operation process in the second embodiment. The system 50 in this embodiment differs from the first embodiment in that the server device 200 does not have a remote control unit 210, and the vehicle 100 is driven by autonomous control rather than remote control. The other configurations are the same as in the first embodiment unless otherwise specified.

[0061] As shown in Figure 15, the memory 112 of the vehicle control device 110 has a reference path RR and a learned model DM pre-stored in it. The communication device 130 can communicate with the external camera 300 via wireless communication. The processor 111 of the vehicle control device 110 functions as a driving control unit 115 by executing a computer program PG1 pre-stored in the memory 112. In this embodiment, the driving control unit 115 does not control the actuator group 120 using a driving control signal acquired from outside the vehicle 100, but rather controls the actuator group 120 using a driving control signal that it generates itself. As shown in Figure 16, the server device 200 functions as a driving path acquisition unit 220, a driving environment information acquisition unit 230, a vehicle information acquisition unit 240, a superimposed image generation unit 250, a processing unit 260, a label information acquisition unit 270, a dataset generation unit 280, and a learning unit 290 by executing a computer program PG2 pre-stored in the memory 202. The trained model DM generated by the learning unit 290 may, for example, be transmitted to the vehicle 100 via wireless communication prior to the vehicle 100's operation and stored in the memory 112 of the vehicle control device 110.

[0062] As shown in Figure 17, in this embodiment, the unmanned operation process is performed by the driving control unit 115. In step S210, the driving control unit 115 acquires vehicle position information using the captured image output from the external camera 300 and the learned model DM. In step S220, the driving control unit 115 uses the vehicle position information and the reference path RR to determine the next target position to which the vehicle 100 should go. In step S230, the driving control unit 115 generates a driving control signal to drive the vehicle 100 toward the determined target position. In step S240, the driving control unit 115 controls the actuator group 120 using the generated driving control signal to drive the vehicle 100 with the acceleration and steering angle expressed in the driving control signal. The driving control unit 115 repeats the acquisition of vehicle position information, determination of the target position, generation of the driving control signal, and control of the actuator group 120 at predetermined intervals.

[0063] According to the system 50 in this embodiment described above, the vehicle 100 can be driven by autonomous control of the vehicle 100 without remote control of the vehicle from an external source.

[0064] C. Other embodiments: (C1) In the first embodiment described above, the server device 200 performs the processing from acquiring vehicle position information to generating a driving control signal. In contrast, the vehicle 100 may perform at least a part of the processing from acquiring vehicle position information to generating a driving control signal. For example, the following forms (1) to (3) may be used.

[0065] (1) The server device 200 may acquire vehicle position information, determine the next target location that vehicle 100 should head to, and generate a route from the vehicle 100's current location, as shown in the acquired vehicle position information, to the target location. The server device 200 may generate a route to the target location between the current location and the destination, or it may generate a route to the destination. The server device 200 may transmit the generated route to vehicle 100. Vehicle 100 may generate a driving control signal so that vehicle 100 travels along the route received from the server device 200, and may use the generated driving control signal to control the actuator group 120.

[0066] (2) The server device 200 may acquire vehicle position information and transmit the acquired vehicle position information to the vehicle 100. The vehicle 100 may determine the next target location to which the vehicle 100 should go, generate a route from the vehicle 100's current location shown in the received vehicle position information to the target location, generate a driving control signal so that the vehicle 100 travels along the generated route, and control the actuator group 120 using the generated driving control signal.

[0067] (3) In the embodiments of (1) and (2) above, the vehicle 100 is equipped with internal sensors, and the detection results output from the internal sensors may be used in at least one of the generation of a route and the generation of a driving control signal. The internal sensors are sensors mounted on the vehicle 100. The internal sensors may include, for example, sensors that detect the motion state of the vehicle 100, sensors that detect the operating state of each part of the vehicle 100, and sensors that detect the environment around the vehicle 100. Specifically, the internal sensors may include, for example, cameras, LiDAR, millimeter-wave radar, ultrasonic sensors, GPS sensors, acceleration sensors, gyro sensors, etc. For example, in the embodiment of (1) above, the server device 200 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the route when generating a route. In the embodiment of (1) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the driving control signal when generating a driving control signal. In the embodiment of (2) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the route when generating a route. In the embodiment described in (2) above, the vehicle 100 may acquire the detection results of the internal sensors and reflect the detection results of the internal sensors in the driving control signal when generating the driving control signal.

[0068] (C2) In each of the above embodiments, the vehicle 100 is equipped with an internal sensor, and the detection result output from the internal sensor may be used in at least one of the generation of the route and the generation of the driving control signal. For example, the vehicle 100 may acquire the detection result from the internal sensor and reflect the detection result from the internal sensor in the route when generating the route. The vehicle 100 may acquire the detection result from the internal sensor and reflect the detection result from the internal sensor in the driving control signal when generating the driving control signal.

[0069] (C3) In the second embodiment described above, the vehicle 100 acquires vehicle position information using the detection results of the external camera 300. In contrast, the vehicle 100 may be equipped with an internal sensor, and the vehicle 100 may acquire vehicle position information using the detection results of the internal sensor, determine the next target location to which the vehicle 100 should go, generate a route from the vehicle 100's current location shown in the acquired vehicle position information to the target location, generate a driving control signal for driving along the generated route, and control the actuator group 120 using the generated driving control signal. In this case, the vehicle 100 can drive without using the detection results of the external camera 300 at all. The vehicle 100 may also acquire the target arrival time and congestion information from outside the vehicle 100 and reflect the target arrival time and congestion information in at least one of the route and the driving control signal. Furthermore, all the functional configurations of the system 50 may be provided in the vehicle 100. That is, the processing realized by the system 50 in this disclosure may be realized by the vehicle 100 alone.

[0070] (C4) In the first embodiment described above, the server device 200 automatically generates a driving control signal to be transmitted to the vehicle 100. Alternatively, the server device 200 may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, an external operator may operate a control device that includes a display for displaying captured images output from an external camera 300, a steering wheel for remotely controlling the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server device 200 via wired or wireless communication, and the server device 200 may generate a driving control signal in accordance with the operation applied to the control device.

[0071] (C5) In each of the above embodiments, the vehicle 100 only needs to have a configuration that allows it to move by unmanned operation, and may take the form of a platform having the configuration described below. Specifically, in order for the vehicle 100 to perform the three functions of "driving," "turning," and "stopping" by unmanned operation, it is sufficient to have at least a vehicle control device 110 and an actuator group 120. When the vehicle 100 acquires information from the outside for unmanned operation, the vehicle 100 may further have a communication device 130. That is, the vehicle 100 that can move by unmanned operation does not need to have at least some of the interior parts such as the driver's seat and dashboard attached, it does not need to have at least some of the exterior parts such as the bumper and fender attached, and it does not need to have a body shell attached. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory FC, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 has been shipped from the factory FC without the remaining parts such as the body shell attached to the vehicle 100. Each component may be attached to the vehicle 100 from any direction, such as the top, bottom, front, rear, right, or left side, and may be attached from the same direction or from different directions. The positioning of the platform can also be determined in the same way as for the vehicle 100 in the first embodiment.

[0072] (C6) Vehicle 100 may be manufactured by combining multiple modules. A module means a unit composed of multiple parts grouped together according to the part or function of the vehicle 100. For example, the platform of vehicle 100 may be manufactured by combining a front module that constitutes the front part of the platform, a central module that constitutes the central part of the platform, and a rear module that constitutes the rear part of the platform. The number of modules that constitute the platform is not limited to three, and may be two or fewer, or four or more. In addition to, or instead of, the parts that constitute the platform, parts that constitute parts of the vehicle 100 that are different from the platform may be modularized. Furthermore, various modules may include any exterior parts such as bumpers and grilles, or any interior parts such as seats and consoles. Moreover, not limited to vehicle 100, any type of mobile body may be manufactured by combining multiple modules. Such modules may be manufactured, for example, by joining multiple parts by welding or fasteners, or by integrally molding at least a part of the parts that constitute the module as a single part by casting. A molding technique for integrally molding a single component, especially a relatively large component, is also called gigacast or megacast. For example, the front module, central module, and rear module mentioned above may be manufactured using gigacast.

[0073] (C7) Transporting vehicle 100 using the unmanned operation of vehicle 100 is also called "autonomous transport." The configuration for realizing autonomous transport is also called a "vehicle remote control autonomous driving transport system." Furthermore, a production method that uses autonomous transport to produce vehicle 100 is also called "autonomous production." In autonomous production, for example, at a factory FC that manufactures vehicle 100, at least a portion of the transport of vehicle 100 is realized by autonomous transport.

[0074] (C8) In each of the above embodiments, some or all of the functions and processes implemented in software may be implemented in hardware. Also, some or all of the functions and processes implemented in hardware may be implemented in software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0075] (C9) Figure 18 is a first explanatory diagram schematically showing a trained model DM in another embodiment. Figure 19 is a second explanatory diagram schematically showing a trained model DM in another embodiment. In each of the above embodiments, the trained model DM is configured to output a mask image when an captured image is input. In contrast, the trained model DM may be configured to output coordinate points relating to the contour of the vehicle 100 when an captured image is input. For example, as shown in Figure 18, it may be configured to output four coordinate points. As shown in Figure 19, it may be configured to output eight coordinate points. In these cases, the label information includes the position coordinates of the coordinate points. The remote control unit 210 and the driving control unit 115 can, for example, acquire vehicle position information using the coordinate points output from the trained model DM.

[0076] (C10) In each of the above embodiments, the processor 201 may determine a specific location among a plurality of locations on the vehicle 100's travel path where the detection accuracy of the vehicle 100 by the trained model MD is relatively low, and may increase the number of training datasets DS included in the training dataset group DG corresponding to the specific location compared to other locations. For example, in dark places, places with many obstacles, or places where the position and orientation of the vehicle 100 are likely to change due to the influence of disturbances, the processor 201 may increase the number of training datasets DS included in the training dataset group DG compared to when generating a trained model DM for acquiring vehicle position information using images captured by an external camera 300 located in a dark place, a place with many obstacles, or a place where the position and orientation of the vehicle 100 are likely to change due to the influence of disturbances, compared to when generating a trained model DM for acquiring vehicle position information using images captured by an external camera 300 located in a bright place, a place with few obstacles, or a place where the position and orientation of the vehicle 100 are unlikely to change due to disturbances. For example, the number of training datasets DS included in the training dataset group DG may be increased to include more patterns of brightness, vehicle position, and vehicle orientation. In this case, the detection accuracy of vehicle 100 can be improved in dark places, places with many obstacles, and places where the position and orientation of vehicle 100 are likely to change due to disturbances.

[0077] (C11) In each of the above embodiments, the processor 201 may determine a specific location among a plurality of locations on the vehicle 100's travel path where the detection accuracy of the vehicle 100 by the trained model MD is relatively low, and generate a training dataset DS corresponding to the environment of the specific location. For example, in dark places, places with many obstacles, or places where the position and orientation of the vehicle 100 are likely to change due to disturbances, the processor 201 may, when generating a trained model DM for acquiring vehicle position information using images captured by an external camera 300 located in a dark place, lower the brightness in the superimposed image GS compared to when generating a trained model DM for acquiring vehicle position information using images captured by an external camera 300 located in a bright place. For places where the position and orientation of the vehicle 100 are likely to change, the influence of disturbances on the position and orientation of the vehicle 100 may be taken into consideration. In this case, the detection accuracy of the vehicle 100 can be improved in dark places, places with many obstacles, or places where the position and orientation of the vehicle 100 are likely to change due to disturbances.

[0078] (C12) In the first embodiment described above, the remote control unit 210 may be located on a computer different from the server device 200. In this case, the computer equipped with the remote control unit 210 is equipped with a communication device for communicating with the vehicle 100 via wireless communication. The computer equipped with the remote control unit 210 acquires a trained model DM generated on the server device 200 and uses the trained model DM to acquire the position and orientation of the vehicle 100.

[0079] (C13) In each of the above embodiments, the driving environment information acquisition unit 230 acquires the factory model MF as driving environment information, the vehicle information acquisition unit 240 acquires the vehicle model MV as vehicle information, and the superimposed image generation unit 250 generates a superimposed image by superimposing the vehicle model MV and the factory model MF. Alternatively, the driving environment information acquisition unit 230 may acquire an image representing the appearance of the factory FC including the driving route as driving environment information. The image of the factory FC may be an image captured by capturing the appearance of the factory FC including the driving route in real space, or an image generated by rendering using the factory model MF. The vehicle information acquisition unit 240 may acquire an image representing the appearance of the vehicle 100 as vehicle information. The image of the vehicle 100 may be an image captured by capturing the appearance of the vehicle 100 in real space, or an image generated by rendering using the vehicle model MV. The superimposed image generation unit 250 may generate a superimposed image by superimposing the image of the vehicle 100 and the image of the factory FC, in other words, by combining the image of the vehicle 100 and the image of the factory FC. The processing unit 260 may perform predetermined processing on the image of the vehicle 100 and the image of the factory FC before generating a superimposed image by overlapping the images of the vehicle 100 and the factory FC. Alternatively, predetermined processing may be performed on the superimposed image after generating a superimposed image by overlapping the images of the vehicle 100 and the factory FC.

[0080] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0081] 50...System, 100...Vehicle, 110...Vehicle control device, 111...Processor, 112...Memory, 113...Input / Output Interface, 114...Internal Bus, 115...Driving Control Unit, 120...Actuator Group, 130...Communication Device, 200...Server Device, 201...Processor, 202...Memory, 203...Input / Output Interface, 204...Internal Bus, 205...Communication Device, 206...Input Device, 207...Display Device, 210...Remote Control Unit, 220...Driving Path Acquisition Unit, 230...Driving Environment Information Acquisition Unit, 240...Vehicle Information Acquisition Unit, 250...Superimposed Image Generation Unit, 260...Processing Unit, 270...Label Information Acquisition Unit, 280...Dataset Generation Unit, 290...Learning Unit, 300...External Camera

Claims

1. It is a method, A process for acquiring the movement path of a moving object, A superimposed image generation step generates a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing the moving environment including the moving path. A label information acquisition step for acquiring label information relating to the position and orientation of the moving body, A dataset generation step that generates a training dataset including the superimposed image and the label information, A method having

2. The method according to claim 1, A method further comprising a step of performing predetermined processing on an image or three-dimensional data of the moving environment prior to the superimposed image generation step.

3. The method according to claim 1, A method further comprising the step of performing predetermined processing on the image or three-dimensional data of the moving object prior to the step of generating the superimposed image.

4. The method according to claim 1, A method further comprising the step of applying predetermined processing to the superimposed image.

5. The method according to claim 4, The processing method includes at least one of adding or changing the light irradiated onto the moving body and adding or changing the shadow of the moving body.

6. A method according to any one of claims 2 to 4, The processing method includes changing at least one of brightness, saturation, and contrast.

7. The method according to claim 1, The superimposed image generation process is performed multiple times. A method comprising performing the superimposed image generation step multiple times to generate a plurality of superimposed images, wherein the position of the moving body in a direction parallel to the movement path is different in at least two of the plurality of superimposed images.

8. The method according to claim 1, The superimposed image generation process is performed multiple times. A method comprising performing the superimposed image generation step multiple times to generate a plurality of superimposed images, wherein the position of the moving body in a direction perpendicular to the movement path is different in at least two of the plurality of superimposed images.

9. The method according to claim 1, The superimposed image generation process is performed multiple times. A method comprising: making the orientation of the moving body relative to the movement path different in at least two of the multiple superimposed images generated by performing the superimposed image generation step multiple times.

10. The method according to claim 1, The superimposed image generation process is performed multiple times. A method for making the color of the moving object different in at least two of the multiple superimposed images generated by performing the superimposed image generation step multiple times.

11. The method according to claim 1, The method for acquiring label information in the label information acquisition step involves acquiring the label information from information associated with an image or three-dimensional data of the moving object.

12. The method according to claim 1, A method further comprising a learning step of performing machine learning using a set of training datasets including the aforementioned training dataset.

13. A method according to claim 12, A method further comprising the step of acquiring information regarding the position and orientation of a moving object while it is moving through the moving environment, using the trained model generated by the training step and an image of the moving object as it moves through the moving environment.

14. The method according to claim 1, The superimposed image generation process is performed multiple times. The system further comprises a learning process that performs machine learning using a set of training datasets that includes multiple training datasets, Among the multiple locations along the aforementioned movement path, a specific location is determined in which the accuracy of the output of the trained model generated by the training process is relatively low. A method for increasing the number of training datasets corresponding to the specific location compared to other locations.

15. The method according to claim 1, The superimposed image generation process is performed multiple times. The system further comprises a learning process that performs machine learning using a set of training datasets that includes multiple training datasets, Among the multiple locations along the aforementioned movement path, a specific location is determined in which the accuracy of the output of the trained model generated by the training process is relatively low. A method for generating the training dataset corresponding to the environment of the specified location.

16. It is a device, A movement path acquisition unit that acquires the movement path of a moving object, A superimposed image generation unit generates a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing the moving environment including the moving path. A label information acquisition unit that acquires label information relating to the position and orientation of the moving object, A dataset generation unit that generates a training dataset including the superimposed image and the label information, A device equipped with the following features.

17. It is a computer program, A function to acquire an image or three-dimensional data representing a moving object, or an image or three-dimensional data representing the environment in which the moving object moves, A function to display a screen on which processing conditions related to processing to be applied to the acquired image or three-dimensional data can be selected, A function to perform the processing on the acquired image or three-dimensional data according to the selected processing conditions, A computer program that enables a computer to realize something.

18. A computer program according to claim 17, The aforementioned processing includes a computer program that modifies at least one of brightness, saturation, and contrast.

19. A computer program according to claim 17, The processing is a computer program that includes at least one of adding or changing the light irradiated onto the moving body and adding or changing the shadow of the moving body.

Images