Method, device, computer program and dataset for learning

Abstract

To provide a technique that reduces the time and effort required to prepare data for learning.SOLUTION: A method has a mobile route acquisition process for acquiring a movement route of a mobile entity, a superimposed image generation process for generating a superimposed image by superimposing an image or three-dimensional data representing the mobile entity and an image or three-dimensional data representing the movement environment including the movement route, a label information acquisition process for acquiring label information regarding the position and orientation of the mobile entity, and a dataset generation process for generating a learning dataset including the superimposed image and label information.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure relates to a method, an apparatus, a computer program and a training dataset. [Background technology]

[0002] 2. Description of the Related Art There is known a technique for detecting the position of a traveling vehicle using an image captured by a camera positioned outside the vehicle (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] When using artificial intelligence to detect moving objects such as vehicles from images captured by a camera, it is necessary to prepare a large amount of data for the artificial intelligence to learn from. Therefore, technology that reduces the effort required to prepare data for learning is desired. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, there is provided a method, comprising: a moving path acquisition step of acquiring a moving path of a moving object, a superimposed image generation step of generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path, a label information acquisition step of acquiring label information related to a position and an orientation of the moving object, and a dataset generation step of generating a learning dataset including the superimposed image and the label information. According to the method of this aspect, a training data set including superimposed images is generated, so that the effort required for preparing the training data set can be reduced. (2) The method of the above aspect may further include a step of performing a predetermined processing on the image or three-dimensional data of the moving environment prior to the superimposed image generating step. According to the method of this aspect, a learning dataset including superimposed images in a desired state can be generated. (3) The method of the above aspect may further include a step of performing a predetermined processing on the image or three-dimensional data of the moving object prior to the step of generating the superimposed image. According to the method of this aspect, a learning dataset including superimposed images in a desired state can be generated. (4) The method of the above aspect may further include a step of performing a predetermined process on the superimposed image. According to the method of this aspect, a learning dataset including superimposed images in a desired state can be generated. (5) In the method of the above aspect, the processing may include at least one of adding or changing light irradiated to the moving body and adding or changing a shadow of the moving body. According to the method of this aspect, it is possible to make the moving object depicted in the superimposed image look different. (6) In the method of the above aspect, the processing may include changing at least one of brightness, saturation, and contrast. According to the method of this aspect, it is possible to make the appearance of the moving object and the moving environment depicted in the superimposed image different. (7) In the method of the above aspect, the superimposed image generation process may be performed multiple times, and the position of the moving body in a direction parallel to the movement path may be made different in at least two of the multiple superimposed images generated by performing the superimposed image generation process multiple times. According to the method of this aspect, it is possible to generate a plurality of superimposed images in which the position of the moving object in the direction parallel to the moving path is different from one another. (8) In the method of the above aspect, the superimposed image generation process may be performed multiple times, and the position of the moving body in a direction perpendicular to the movement path may be made different in at least two of the multiple superimposed images generated by performing the superimposed image generation process multiple times. According to the method of this aspect, it is possible to generate a plurality of superimposed images in which the position of the moving object in the direction perpendicular to the moving path is different from one another. (9) In the method of the above aspect, the superimposed image generation process may be performed multiple times, and the orientation of the moving body relative to the travel path may be made different in at least two of the multiple superimposed images generated by performing the superimposed image generation process multiple times. According to the method of this aspect, it is possible to generate a plurality of superimposed images in which the orientation of the moving object with respect to the moving path is different from one another. (10) In the method of the above aspect, the superimposed image generation process may be performed multiple times, and the color of the moving body may be made different in at least two of the multiple superimposed images generated by performing the superimposed image generation process multiple times. According to the method of this aspect, a plurality of superimposed images of the moving object, each of which has a different color, can be generated. (11) In the method according to the above aspect, in the label information acquiring step, the label information may be acquired from information associated with an image or three-dimensional data of the moving object. According to the method of this aspect, it is possible to eliminate the need to separately prepare information regarding the position and orientation of the moving object. (12) The method of the above aspect may further include a learning step of performing machine learning using a group of training datasets including the training dataset. According to the method of this aspect, machine learning can be performed using a training data set that includes superimposed images. (13) The method of the above aspect may further include a step of acquiring information regarding the position and orientation of a moving body moving in the mobile environment using the trained model generated by the learning step and a captured image of the moving body moving in the mobile environment. According to the method of this aspect, it is possible to obtain information regarding the position and orientation of a moving object moving in a mobile environment using a trained model. (14) In the method of the above aspect, the superimposed image generation process may further include a learning process that is executed multiple times and performs machine learning using a group of training datasets including multiple training datasets, and a specific position is determined among multiple positions on the movement path at which the accuracy of the output of the trained model generated by the learning process is relatively low, and the number of the training datasets corresponding to the specific position may be made larger than the number of other positions. According to this form of the method, it is possible to improve the accuracy of the output of the trained model at a specific position. (15) In the method of the above aspect, the superimposed image generation process may be executed multiple times, and may further include a learning process for performing machine learning using a group of training datasets including multiple training datasets, and may determine a specific position among multiple positions on the movement path at which the accuracy of the output of the trained model generated by the learning process is relatively low, and generate the training dataset according to the environment of the specific position. According to this form of the method, it is possible to improve the accuracy of the output of the trained model at a specific position. (16) According to a second aspect of the present disclosure, there is provided an apparatus including: a moving path acquisition unit that acquires a moving path of a moving object; a superimposed image generation unit that generates a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path; a label information acquisition unit that acquires label information regarding a position and an orientation of the moving object; and a dataset generation unit that generates a learning dataset including the superimposed image and the label information. According to the device of this aspect, a training data set including a superimposed image is generated, so that the effort required for preparing the training data set can be reduced. (17) According to a third aspect of the present disclosure, there is provided a computer program that causes a computer to realize a function of acquiring an image or three-dimensional data representing a moving object or an image or three-dimensional data representing a moving environment of the moving object, a function of displaying a screen on which processing conditions for processing the acquired image or three-dimensional data can be selected, and a function of processing the acquired image or three-dimensional data according to the selected processing conditions. According to the computer program of this aspect, it is possible to generate a superimposed image according to the processing conditions selected by the user. (18) In the computer program of the above aspect, the processing may include changing at least one of brightness, saturation, and contrast. According to the computer program of this form, a superimposed image in a desired state can be generated. (19) In the computer program of the above aspect, the processing may include at least one of adding or changing light irradiated to the moving object and adding or changing a shadow of the moving object. According to the computer program of this form, a superimposed image in a desired state can be generated. (20) According to a fourth aspect of the present disclosure, there is provided a computer program that causes a computer to realize a function of acquiring an image or three-dimensional data representing a moving object and an image or three-dimensional data representing an environment in which the moving object moves, a function of displaying a screen on which a superimposition condition for superimposing the image or three-dimensional data representing the moving object and the image or three-dimensional data representing the environment in which the moving object moves can be selected, and a function of generating a superimposed image by superimposing the image or three-dimensional data representing the moving object and the image or three-dimensional data representing the environment in accordance with the selected superimposition condition. According to the computer program of this aspect, it is possible to generate a superimposed image according to the superimposition conditions selected by the user. (21) In the computer program of the above aspect, the superimposition condition may include at least one of a position and an orientation of the moving object with respect to the moving environment. According to the computer program of this form, a superimposed image in a desired state can be generated. (22) According to a fifth aspect of the present disclosure, there is provided a training dataset for use in machine learning, the training dataset including a superimposed image generated by superimposing an image or three-dimensional data representing a moving object and an image or three-dimensional data representing a moving environment of the moving object, and information regarding a position and an orientation of the moving object. According to this form of training dataset, the training dataset can be used to generate a trained model for obtaining information regarding the position and orientation of a moving object. The present disclosure may be realized in various forms other than a method, an apparatus, a computer program, and a training dataset, for example, a system, a recording medium on which a computer program is recorded, or the like. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram showing the configuration of a system according to a first embodiment. [Diagram 2] FIG. 1 is an explanatory diagram showing a configuration of a vehicle according to a first embodiment. [Diagram 3] FIG. 2 is an explanatory diagram showing the configuration of a server device according to the first embodiment. [Figure 4] 4 is a flowchart showing the contents of an unmanned operation process according to the first embodiment. [Diagram 5] 4 is a flowchart showing an example of a method for generating shape data and a method for acquiring a vehicle position using the shape data. [Figure 6] 6A to 6C are schematic diagrams showing examples of various images obtained when the method shown in FIG. 5 is executed. [Figure 7] FIG. 11 is an explanatory diagram for explaining details of a coordinate point calculation step. [Figure 8] FIG. 4 is an explanatory diagram for explaining a method of calculating a base coordinate point. [Figure 9]FIG. 11 is a first explanatory diagram for explaining details of a position conversion step. [Figure 10] FIG. 2 is a second explanatory diagram for explaining details of the position conversion step. [Figure 11] 1 is a flowchart showing the contents of a trained model generation method. [Figure 12] FIG. 4 is an explanatory diagram illustrating a schematic diagram of a factory model and a vehicle model being superimposed on each other; [Figure 13] FIG. 1 is an explanatory diagram illustrating a learning dataset group. [Figure 14] FIG. 4 is an explanatory diagram illustrating a selection screen. [Figure 15] FIG. 11 is an explanatory diagram showing the configuration of a vehicle according to a second embodiment. [Figure 16] FIG. 11 is an explanatory diagram showing the configuration of a server device according to a second embodiment. [Figure 17] 10 is a flowchart showing the contents of an unmanned operation process according to a second embodiment. [Figure 18] FIG. 11 is a first explanatory diagram illustrating a trained model according to another embodiment. [Figure 19] FIG. 2 is a second explanatory diagram illustrating a trained model of another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: Fig. 1 is an explanatory diagram showing the configuration of a system 50 in the first embodiment. Fig. 2 is an explanatory diagram showing the configuration of a vehicle 100 in the first embodiment. Fig. 3 is an explanatory diagram showing the configuration of a server device 200 in the first embodiment. As shown in Fig. 1, in this embodiment, the system 50 includes a vehicle 100 which is a moving body, a server device 200, and at least one external camera 300.

[0009] In the present disclosure, the term "mobile body" refers to an object that can move. The mobile body is, for example, a vehicle or an electric vertical take-off and landing aircraft (a so-called flying car). The vehicle may be a vehicle that runs on wheels or a vehicle that runs on caterpillar tracks. That is, the form of the vehicle is not particularly limited, and the vehicle may be, for example, a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, a construction vehicle, or the like. The vehicle may be an electric vehicle (BEV: Battery Electric Vehicle), a gasoline vehicle, a hybrid vehicle (HEV: Hybrid Electric Vehicle), a fuel cell vehicle (FCEV: Fuel Cell Electric Vehicle), or the like. In addition, when the mobile body is other than a vehicle, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "running" can be appropriately replaced with "movement".

[0010] The vehicle 100 is configured to be capable of traveling by unmanned driving. Here, "unmanned driving" means driving that is not performed by a driver on board the vehicle 100. "Driving operation" means at least one operation of the vehicle 100, including "running," "turning," and "stopping." Unmanned driving is realized by automatic or manual remote control using a device located outside the vehicle 100, or by autonomous control of the vehicle 100. The vehicle 100 traveling by unmanned driving may have a driver who does not perform driving operations on board. The driver who does not perform driving operations includes, for example, a person who simply sits in a seat of the vehicle 100, and a person who boards the vehicle 100 and performs an action other than driving operations. The action other than driving operations includes, for example, assembling parts for the vehicle 100, inspecting the vehicle 100, and operating switches provided on the vehicle 100. Note that driving by a driver's driving operation is sometimes called "manned driving."

[0011] In this disclosure, "remote control" includes "full remote control" in which all of the operations of the vehicle 100 are completely determined from outside the vehicle 100, and "partial remote control" in which some of the operations of the vehicle 100 are determined from outside the vehicle 100. In addition, "autonomous control" includes "full autonomous control" in which the vehicle 100 autonomously controls its own operations without receiving any information from a device located outside the vehicle 100, and "partial autonomous control" in which the vehicle 100 autonomously controls its own operations using information received from a device located outside the vehicle 100.

[0012] In this embodiment, the system 50 is used in a factory FC that manufactures the vehicle 100. The reference coordinate system of the factory FC is a global coordinate system GC. That is, any position in the factory FC can be expressed by X, Y, and Z coordinates in the global coordinate system GC. The factory FC includes a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by a road TR along which the vehicle 100 can travel. In the factory FC, a plurality of external cameras 300 are installed along the road TR. The external cameras 300 include a communication device (not shown) and can communicate with the server device 200 by wired or wireless communication. The positions and orientations of the external cameras 300 in the factory FC are adjusted in advance. The vehicle 100 moves from the first location PL1 to the second location PL2 along the road TR by unmanned driving.

[0013] As shown in Fig. 2, the vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 that operates under the control of the vehicle control device 110, and a communication device 130 for communicating with the server device 200 by wireless communication. The actuator group 120 includes at least one actuator. In this embodiment, the actuator group 120 includes an actuator of a drive device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The drive device includes a battery, a running motor driven by the power of the battery, and wheels that are rotated by the running motor. The running motor is included in the actuator of the drive device.

[0014] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other via the internal bus 114 so as to be able to communicate in both directions. The input / output interface 113 is connected to an actuator group 120 and a communication device 130.

[0015] The processor 111 functions as the driving control unit 115 by executing a computer program PG1 stored in advance in the memory 112. The driving control unit 115 controls the actuator group 120 to drive the vehicle 100. When a passenger is on board the vehicle 100, the driving control unit 115 controls the actuator group 120 in response to the operation of the passenger to drive the vehicle 100. In this embodiment, the driving control unit 115 controls the actuator group 120 in response to a driving control signal received from the server device 200 to drive the vehicle 100, regardless of whether a passenger is on board the vehicle 100 or not. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0016] As shown in FIG. 3, the server device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 so as to be able to communicate bidirectionally. A communication device 205 for communicating with the vehicle 100 by wireless communication is connected to the input / output interface 203. In this embodiment, the communication device 205 can communicate with the external camera 300 by wired communication or wireless communication. In addition to the communication device 205, an input device 206 and a display device 207 are connected to the input / output interface 203. The input device 206 is, for example, a mouse or a keyboard. The display device 207 is, for example, a liquid crystal display.

[0017] The processor 201 executes a computer program PG2 pre-stored in the memory 202, thereby functioning as a remote control unit 210, a driving route acquisition unit 220, a driving environment information acquisition unit 230, a vehicle information acquisition unit 240, a superimposed image generation unit 250, a processing unit 260, a label information acquisition unit 270, a data set generation unit 280, and a learning unit 290. The driving route acquisition unit 220 may be referred to as a travel route acquisition unit, the driving environment information acquisition unit 230 may be referred to as a travel environment information acquisition unit, and the vehicle information acquisition unit 240 may be referred to as a mobile object information acquisition unit.

[0018] The remote control unit 210 remotely controls the vehicle 100 to run the vehicle 100. In this embodiment, the remote control unit 210 remotely controls the vehicle 100 so that the vehicle 100 runs along a reference route RR pre-stored in the memory 202. The remote control unit 100 acquires vehicle position information including the position and orientation of the vehicle 100 using the trained model DM pre-stored in the memory 202 and a captured image acquired from the external camera 300, and remotely controls the vehicle 100 using the vehicle position information and the reference route RR. The trained model DM is prepared in advance for each external camera 300.

[0019] The travel route acquisition unit 220 acquires a travel route of the vehicle 100. In this embodiment, the travel route acquisition unit 220 acquires a reference route RR that is stored in advance in the memory 202 as the travel route of the vehicle 100.

[0020] The driving environment information acquisition unit 230 acquires driving environment information representing the appearance of the driving environment of the vehicle 100. The driving environment information is an image representing the appearance of the driving environment of the vehicle 100, or three-dimensional data representing the appearance of the driving environment of the vehicle 100. In this embodiment, the driving environment information acquisition unit 230 acquires a factory model MF stored in advance in the memory 202 as the driving environment information. The factory model MF is three-dimensional data representing the appearance of the factory FC, which is the driving environment of the vehicle 100. The appearance of the factory FC includes the appearance of the building of the factory FC, the appearance of various facilities installed inside and outside the building of the factory FC, the appearance of the road surface of the road TR in the factory FC, and the like.

[0021] The vehicle information acquisition unit 240 acquires vehicle information representing the appearance of the vehicle 100. The vehicle information is an image representing the appearance of the vehicle 100, or three-dimensional data representing the appearance of the vehicle 100. In this embodiment, the vehicle information acquisition unit 240 acquires a vehicle model MV stored in advance in the memory 202 as the vehicle information. The vehicle model MV is three-dimensional data representing the appearance of the vehicle 100.

[0022] The vehicle model MV and the factory model MF are generated using, for example, three-dimensional CAD software or three-dimensional CG software. The vehicle model MV and the factory model MF may be generated on the server device 200, or may be generated on a computer different from the server device 200. It is preferable that the appearance of the vehicle model MV is a reproduction of the appearance of the vehicle 100 as accurately as possible, and it is preferable that the appearance of the factory model MF is a reproduction of the appearance of the factory FC as accurately as possible.

[0023] The superimposed image generating unit 250 generates a superimposed image by superimposing the vehicle information and the driving environment information. In this embodiment, the superimposed image generating unit 250 generates a superimposed image by superimposing the vehicle model MV and the factory model MF, and outputs the generated superimposed image.

[0024] The processing unit 260 processes at least one of the vehicle information, the driving environment information, and the superimposed image. The processing may include at least one of adding or changing light irradiated to the vehicle 100 and adding or changing shading of the vehicle 100. The processing may include changing at least one of the brightness, saturation, and contrast. In this embodiment, the processing unit 260 can change the color and texture of at least a part of the vehicle 100 represented in the vehicle model MV by processing the vehicle model MV. The processing unit 260 can change the color and texture of at least a part of the factory FC represented in the factory model MF by processing the factory model MF. The processing unit 260 can change at least one of the brightness, saturation, and contrast of at least a part of the superimposed image by processing the superimposed image.

[0025] The label information acquisition unit 270 acquires label information related to at least one of the position and the orientation of the vehicle 100. The label information is used as a correct answer label in supervised machine learning.

[0026] The data set generating unit 280 generates a learning data set including superimposed images and label information associated with the superimposed images.

[0027] The learning unit 290 generates a learned model DM by performing machine learning using a group of learning datasets including at least one learning dataset.

[0028] 4 is a flowchart showing the contents of the unmanned driving process in this embodiment. In this embodiment, the unmanned driving process is executed by the remote control unit 210 of the server device 200 and the driving control unit 115 of the vehicle 100. In step S110, the remote control unit 210 acquires vehicle position information of the vehicle 100 using the captured image output from the external camera 300 and the trained model DM. The vehicle position information is position information that is the basis for generating a driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC.

[0029] In detail, in step S110, the remote control unit 210 acquires shape data related to the outline of the vehicle 100 from the captured image using, for example, a trained model DM utilizing artificial intelligence, calculates the coordinates of the positioning point of the vehicle 100 in the coordinate system of the captured image, i.e., the local coordinate system of the factory FC, and converts the calculated coordinates into coordinates in the global coordinate system GC, thereby acquiring the position of the vehicle 100. In this embodiment, the trained model DM is generated in advance by a trained model generation method described later and stored in the memory 202. The remote control unit 210 acquires the orientation of the vehicle 100 by estimating based on the orientation of the movement vector of the vehicle 100 calculated from the positional change of the feature points of the vehicle 100 between frames of the captured image using, for example, an optical flow method.

[0030] In step S120, the remote control unit 210 determines a target position to which the vehicle 100 should next head. In this embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system GC. The memory 202 stores in advance a reference route RR, which is a route to be traveled by the vehicle 100. The route is represented by nodes indicating the departure point, nodes indicating passing points, nodes indicating the destination, and links connecting the nodes. The remote control unit 210 uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. The remote control unit 210 determines a target position on the reference route RR that is ahead of the current location of the vehicle 100.

[0031] In step S130, the remote control unit 210 generates a travel control signal for driving the vehicle 100 toward the determined target position. The remote control unit 210 calculates the travel speed of the vehicle 100 from the transition of the position of the vehicle 100, and compares the calculated travel speed with a predetermined target speed of the vehicle 100. When the travel speed is lower than the target speed as a whole, the remote control unit 210 determines the acceleration so that the vehicle 100 accelerates, and when the travel speed is higher than the target speed, the remote control unit 210 determines the acceleration so that the vehicle 100 decelerates. When the vehicle 100 is located on the reference route RR, the remote control unit 210 determines the acceleration and steering angle so that the vehicle 100 does not deviate from the reference route RR, and when the vehicle 100 is not located on the reference route RR, in other words, when the vehicle 100 deviates from the reference route RR, the remote control unit 210 determines the acceleration and steering angle so that the vehicle 100 returns to the reference route RR.

[0032] In step S140, the remote control unit 210 transmits the generated driving control signal to the vehicle 100. The remote control unit 210 repeats, at a predetermined cycle, obtaining the position of the vehicle 100, determining the target position, generating the driving control signal, and transmitting the driving control signal.

[0033] In step S150, the driving control unit 115 receives a driving control signal transmitted from the server device 200. In step S160, the driving control unit 115 controls the actuator group 120 using the driving control signal received from the server device 200, thereby causing the vehicle 100 to drive at the acceleration and steering angle represented in the driving control signal. The driving control unit 115 repeats receiving the driving control signal and controlling the actuator group 120 at a predetermined cycle. According to the system 50 in this embodiment, the vehicle 100 can be driven by remote control, and therefore the vehicle 100 can be moved without using transportation equipment such as a crane or a conveyor.

[0034] The method of acquiring vehicle position information in step S110 of Fig. 4 will be described in more detail. Fig. 5 is a flowchart showing an example of a method of generating shape data and a method of acquiring the position of the vehicle 100 using the shape data. Fig. 6 is a schematic diagram showing examples of various images Im1 to Im6 when the method shown in Fig. 5 is executed. Fig. 6 shows step numbers corresponding to each step of Fig. 5. In this embodiment, the positioning point 10e of the vehicle 100 is the rear end on the left side of the vehicle 100. Note that the positioning point 10e of the vehicle 100 may be other than the rear end on the left side of the vehicle 100.

[0035] In an image acquisition step (step S410) shown in FIG. 5, the remote control unit 210 acquires an original image Im1 output from the external camera 300. In a distortion correction step (step S420), the remote control unit 210 corrects the distortion of the original image Im1 to generate a corrected image Im2 as a processed image. In a rotation processing step (step S430), the remote control unit 210 rotates the corrected image Im2. As a result, the remote control unit 210 generates a rotated image Im3 as a processed image. In a crop processing step (step S440), the remote control unit 210 deletes an unnecessary area A2 other than a necessary area A1 constituted by the vehicle 100 and the area surrounding the vehicle 100 from the rotated image Im3 among the areas constituting the rotated image Im3. As a result, the remote control unit 210 generates a processed image Im4 as a processed image. The steps from step S420 to step S440 are pre-processing steps for improving the detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4. Therefore, at least one of the steps from step S420 to step S440 may be omitted.

[0036] In the detection step (step S450), the remote control unit 210 inputs the processed image Im4 to the trained model DM. As a result, the remote control unit 210 detects the vehicle 100 from the processed image Im4 and acquires a first mask image Im5 as shape data Da. The first mask image Im5 is an image in which a mask area Ms is added to the processed image Im4 by masking the area representing the vehicle 100 among the areas constituting the processed image Im4.

[0037] In the perspective transformation step (step S460), the remote control unit 210 performs perspective transformation on the first mask image Im5 to generate a second mask image Im6. The remote control unit 210 performs perspective transformation on the first mask image Im5 into a bird's-eye image seen from a viewpoint above the vehicle 100 that is approximately perpendicular to the road surface Rs, for example, using a predetermined perspective transformation parameter. The perspective transformation parameter is, for example, a parameter related to the position information and internal parameters of the external camera 300 obtained by calibration. In this way, the remote control unit 210 generates a second mask image Im6 expressed in the image coordinate system from the first mask image Im5 expressed in the camera coordinate system. The camera coordinate system is a coordinate system having the focal point of the external camera 300 as the origin and having coordinate axes represented by the Xc axis and the Yc axis perpendicular to the Xc axis. The image coordinate system is a coordinate system having the origin at a point on the image plane and having coordinate axes represented by the Xi axis and the Yi axis perpendicular to the Xi axis.

[0038] In the coordinate point calculation step (step S470), the remote control unit 210 calculates an image coordinate point indicating the position of the vehicle 100 in the image coordinate system. FIG. 7 is an explanatory diagram for explaining the details of the coordinate point calculation step. In order to calculate the image coordinate point, the remote control unit 210 first calculates a base coordinate point P0 from a first circumscribing rectangle R1 set in the mask area Ms in the first mask image Im5, which is an image before perspective transformation. FIG. 8 is an explanatory diagram for explaining a calculation method of the base coordinate point P0. In order to calculate the base coordinate point P0, the remote control unit 210 sets a base circumscribing rectangle R0 for the mask area Ms in the first mask image Im5. Next, the remote control unit 210 rotates the first mask image Im5 by a required rotation amount with the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 100 corresponding to the mask area Ms in the first mask image Im5 faces a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device that displays the first mask image Im5. Next, the remote control unit 210 sets a first circumscribing rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the movement vector V. Next, the remote control unit 210 rotates the first mask image Im5 to which the first circumscribing rectangle R1 has been added in the reverse direction by the above rotation amount, with the center of gravity C of the mask area Ms as the rotation center. As a result, the remote control unit 210 sets the coordinate point of one of the four vertices of the first circumscribing rectangle R1, which has coordinates that are closest to the positioning point 10e of the vehicle 100, as the base coordinate point P0.

[0039] Next, the remote control unit 210 performs perspective transformation on the first mask image Im5 after reverse rotation, in other words, the first mask image Im5 after calculating the base coordinate point P0, as shown in Fig. 7. As a result, the remote control unit 210 sets the coordinate point corresponding to the base coordinate point P0 as the first coordinate point P1 in the first circumscribed rectangle R1 transformed by the perspective transformation.

[0040] Next, the remote control unit 210 sets a second circumscribing rectangle R2 for the mask area Ms in the second mask image Im6 obtained by performing perspective transformation on the first mask image Im5. Then, the remote control unit 210 sets the vertex indicating the same position as the first coordinate point P1 among the vertices of the second circumscribing rectangle R2 as the second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation with each other.

[0041] Next, the remote control unit 210 corrects the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 depending on which is larger between the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the remote control unit 210 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction (Yi1>Yi2), the remote control unit 210 replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. In the example shown in FIG. 7, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Also, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has coordinates (Xi2, Yi1). In this way, the remote control unit 210 calculates the image coordinate point P3 indicating the position of the vehicle 100 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2.

[0042] In the position conversion step (step S480), the remote control unit 210 converts the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point indicating the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC. The remote control unit 210 converts the image coordinate point P3 into a vehicle coordinate point using relational expressions (1) to (3) described below, which use the vehicle coordinate point as a response variable and include the image coordinate point P3, imaging parameters, and vehicle parameters as explanatory variables. The imaging parameters are parameters related to the distance of the external camera 300 from a predetermined reference point. In this embodiment, the imaging parameter is the height H of the external camera 300 from the road surface Rs (FIG. 9 described below). The vehicle parameter is a parameter related to the distance of the positioning point 10e of the vehicle 100 from the reference point. In this embodiment, the vehicle parameter is the height h of the positioning point 10e of the vehicle 100 from the road surface Rs (FIG. 9 described below).

[0043] FIG. 9 is a first explanatory diagram for explaining the details of the position conversion process. FIG. 9 illustrates the state when the vehicle 100 is viewed from the left side. FIG. 10 is a second explanatory diagram for explaining the details of the position conversion process. FIG. 10 illustrates the state when the vehicle 100 is viewed from the roof side. The global coordinate system GC shown in FIG. 9 and FIG. 10 is a coordinate system having a fixed coordinate point Pf indicating an arbitrary reference position of the road surface Rs as the origin and coordinate axes indicated by the Xg axis and the Yg axis perpendicular to the Xg axis. The imaging coordinate point Pc is a coordinate point indicating the position in the global coordinate system GC of the external camera 300 that output the original image Im1 used in the calculation of the image coordinate point P3. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in advance in the memory 202, for example.

[0044] 9, the observation distance on the XgYg plane between the position of the external camera 300 and the position of the vehicle 100 (image coordinate point P3) is Do, the observation error is ΔD, the height [m] of the external camera 300 from the road surface Rs as an imaging parameter is H, and the height [m] of the positioning point 10e of the vehicle 100 from the road surface Rs as a vehicle parameter is h. In this case, the observation error ΔD is expressed by the following formula (1). ΔD = h / H × Do (1) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0045] When the actual distance between the position of the external camera 300 and the position of the positioning point 10e of the vehicle 100 is defined as a first distance D, the first distance D is expressed by the following formula (2). D = Do × (1 - h / H) (2) That is, the first distance D is determined by the observation distance Do, the height H of the external camera 300 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 100 as a vehicle parameter.

[0046] As shown in FIG. 10, if the distance between the reference position and the estimated position of the vehicle 100 is the estimated distance Dp, and the actual distance between the reference position and the vehicle 100 is the second distance Dt, the second distance Dt is expressed by the following equation (3). Dt = Dp × (1 - h / H) (3)

[0047] Here, the estimated distance Dp can be calculated using the third distance Dc, which is the actual distance calculated from the fixed coordinate point Pf and the image coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the remote control unit 210 calculates the vehicle coordinate point Pv using the second distance Dt calculated by correcting the estimated distance Dp using the above formula (3) and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC, and therefore corresponds to the position of the vehicle 100 in the real space.

[0048] FIG. 11 is a flowchart showing the contents of a trained model generation method including a training data set generation method in this embodiment. FIG. 12 is an explanatory diagram that illustrates a state in which a factory model MF and a vehicle model MV are superimposed. FIG. 13 is an explanatory diagram that illustrates a training data set group DG. In this embodiment, the trained model generation method is executed by the processor 201 of the server device 200. First, in step S510, the travel route acquisition unit 220 acquires a travel route of the vehicle 100 in the real space. In this embodiment, the travel route acquisition unit 220 acquires a reference route RR stored in the memory 202 as the travel route of the vehicle 100 in the real space. Note that step S510 may be referred to as a travel route acquisition step or a movement route acquisition step.

[0049] In step S520, the driving environment information acquisition unit 230 acquires driving environment information representing the appearance of the driving environment of the vehicle 100, and the vehicle information acquisition unit 240 acquires vehicle information representing the appearance of the vehicle 100. In this embodiment, the driving environment information acquisition unit 230 acquires a factory model MF pre-stored in the memory 202 as the driving environment information, and the vehicle information acquisition unit 240 acquires a vehicle model MV pre-stored in the memory 202 as the vehicle information.

[0050] In step S530, the processing unit 260 processes at least one of the vehicle model MV and the factory model MF as necessary as a pre-processing. For example, the processing unit 260 can change the color and texture of at least a part of the vehicle 100 represented in the vehicle model MV by processing the vehicle model MV. For example, the processing unit 260 may change the color of the door mirrors and the roof of the vehicle model. For example, the processing unit 260 may perform processing to change the texture of the vehicle model MV from metallic to matte by texture mapping. By changing the texture of the vehicle model MV, it is possible to change highlights and shadows generated on the surface of the vehicle model MV by light irradiated from the virtual light source VL to the vehicle model MV in rendering, which will be described later. For example, the processing unit 260 can change the color and texture of at least a part of the factory FC represented in the factory model MF by processing the factory model MF. Note that the process of step S530 may be omitted.

[0051] In step S540, the superimposed image generating unit 250 generates a superimposed image GS by superimposing the vehicle model MV and the factory model MF. Specifically, as shown in FIG. 12, the superimposed image generating unit 250 superimposes the vehicle model MV and the factory model MF by arranging the factory model MF and the vehicle model MV in the virtual space VS. In the following description, the three-dimensional data obtained by superimposing the vehicle model MV and the factory model MF is called the superimposed model MS. The superimposed image generating unit 250 generates the superimposed image GS from the superimposed model MS by rendering. Rendering means generating a two-dimensional image from three-dimensional data. The rendering method used to generate the superimposed image GS is preferably physics-based rendering. According to physics-based rendering, the behavior of light in the real space can be reproduced as accurately as possible. In addition to the vehicle model MV and the factory model MF, a virtual viewpoint VC and a virtual light source VL are arranged in the virtual space VS. The virtual viewpoint VC is disposed so that the relative position and orientation between the factory model MF and the virtual viewpoint VC in the virtual space VS are the same as the relative position and orientation between the factory FC and the external camera 300 in the real space. The virtual light source VL is preferably disposed at the position of the light source in the real space. A plurality of virtual light sources VL may be disposed in the virtual space VS. For example, when a plurality of lighting equipments are disposed in the factory FC in the real space, the virtual light source VL may be disposed at the position of each lighting equipment. The superimposed image GS shows the superimposed model MS as seen from the virtual viewpoint VC. The superimposed image GS shows the shadow caused by the light irradiated from the virtual light source VL to the superimposed model MS. Note that step S540 may be referred to as a superimposed image generating process.

[0052] In step S550, the processing unit 260 processes the superimposed image GS as necessary as post-processing. For example, the processing unit 260 can change at least one of the brightness, saturation, and contrast of at least a part of the superimposed image GS by processing the superimposed image GS. Note that the process of step S550 may be omitted.

[0053] In step S560, the label information acquisition unit 270 acquires label information LB relating to the position and orientation of the vehicle 100. In this embodiment, the label information acquisition unit 270 acquires, as the label information LB, information on a mask region Ms for acquiring the position and orientation of the vehicle 100. The label information acquisition unit 270 can acquire the mask region Ms by acquiring, for example, information on pixels in which the vehicle model MV is drawn, among a plurality of pixels constituting the superimposed image GS. Note that step S560 may be referred to as a label information acquisition step.

[0054] In step S570, the data set generating unit 280 generates a learning data set DS including a superimposed image GS and label information LB. As shown in Fig. 13, in this embodiment, the learning data set DS includes one superimposed image GS and one piece of label information LB corresponding to the superimposed image GS. Note that step S570 may be referred to as a learning data set generating step.

[0055] In step S580, the dataset generation unit 280 determines whether or not to generate a training dataset DS of another pattern. For example, if the generation of a predetermined number of training datasets DS has not been completed, the dataset generation unit 280 determines to generate a training dataset DS of another pattern, and if the generation of a predetermined number of training datasets DS has been completed, determines not to generate a training dataset DS of another pattern.

[0056] When it is determined in step S580 that a learning data set of another pattern is to be generated, the process from step S530 to step S580 is executed to generate a learning data set DS of another pattern. The process from step S530 to step S580 is repeated until it is determined in step S580 that a learning data set DS of another pattern is not to be generated. The superimposed images GS included in the learning data sets DS of different patterns are different from each other. In this embodiment, the superimposed image generating unit 250 makes the position of the vehicle 100 in a direction parallel to the travel route different in at least two of the multiple superimposed images GS. In addition, in this embodiment, the superimposed image generating unit 250 makes the position of the vehicle 100 in a direction perpendicular to the travel route different in at least two of the multiple superimposed images GS. The direction perpendicular to the travel route here is not a direction perpendicular to the road surface of the road TR, but a direction parallel to the road surface of the road TR. In addition, in this embodiment, the superimposed image generating unit 250 changes the orientation of the vehicle 100 with respect to the travel route in at least two of the multiple superimposed images GS. The superimposed image generating unit 250 changes the position and orientation of the vehicle 100 in the superimposed images GS by changing the position and orientation of the vehicle model MV with respect to the factory model MF. For example, the superimposed image generating unit 250 generates 10 patterns by changing the position of the vehicle 100 in a direction parallel to the travel route, 10 patterns by changing the position of the vehicle 100 in a direction perpendicular to the travel route, and 10 patterns by changing the orientation of the vehicle 100 at each position, for a total of 10×10×10=1000 patterns of superimposed images. The processing unit 260 may change at least a part of the color in at least two of the multiple superimposed images GS. In this case, the superimposed image generating unit 250 generates 1000×10=10000 patterns of superimposed images, for example, 1000 patterns with different positions and orientations of the vehicle 100, and 10 patterns with different colors. The label information acquiring unit 270 acquires label information corresponding to each superimposed image GS.

[0057] In step S590, the learning unit 290 generates a trained model DM by executing machine learning using a training dataset group DG including a plurality of training datasets DS. In this embodiment, the learning unit 290 stores the generated trained model DM in the memory 202. For example, a convolutional neural network (hereinafter, CNN) that realizes either semantic segmentation or instance segmentation can be used as the trained model DM. During training of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result of the trained model DM and the correct label. Note that step S590 may be referred to as a training process.

[0058] FIG. 14 is an explanatory diagram that shows a schematic diagram of the selection screen SW. The selection screen SW includes a superimposition condition selection area RS for selecting a superimposition condition, and a processing condition selection area RK for selecting a processing condition. The superimposition condition is a condition when the vehicle information and the driving environment information are superimposed. In step S540 of FIG. 11, the superimposition image generating unit 250 superimposes the vehicle information and the driving environment information according to the superimposition condition selected in the superimposition condition selection area RS. The superimposition condition includes, for example, a condition regarding the position of the vehicle model MV with respect to the factory model MF, and a condition regarding the orientation of the vehicle model MV with respect to the factory model MF. The processing condition is a condition when processing is performed on at least one of the vehicle information, the driving environment information, and the superimposition image. In steps S530 and S550 of FIG. 11, the processing unit 260 processes at least one of the vehicle information, the driving environment information, and the superimposition image according to the processing condition selected in the processing condition selection area RK. The processing condition includes, for example, a condition regarding the color of the vehicle 100. The processing conditions may include conditions related to the weather of the factory FC. The processing condition selection area RK may be configured to be able to select, for example, sunny weather and rainy weather. When rainy weather is selected, the processing unit 260 may darken the virtual light source VL compared to when sunny weather is selected. Alternatively, the processing condition selection area RK may be configured to be able to select the longitude and latitude, time, and date of the factory FC. The processing unit 260 may acquire weather information related to the position of the sun from outside according to the conditions selected in the processing condition selection area RK, and change the position of the virtual light source VL based on the weather information. In this case, it is possible to suppress a decrease in the detection accuracy of the vehicle 100 by the trained model MD due to a change in weather. In addition, the processing condition selection area RK may be configured to be able to select conditions such as the size of the raindrops, the interval between the raindrops, the falling speed of the raindrops, and whether the raindrops are water or ice. In this case, it is possible to suppress a decrease in the detection accuracy of the vehicle 100 by the trained model MD when the vehicle 100 is occluded by the raindrops. By selecting the falling speed of the raindrops, it is possible to select the length of the afterimage of the raindrops represented in the superimposed image.

[0059] According to the system 50 in the present embodiment described above, the superimposed image generating unit 250 generates the superimposed image GS by superimposing the vehicle model MV and the factory model MF, and the data set generating unit 280 generates the learning data set DS including the superimposed image GS. Therefore, for example, it is possible to eliminate the effort of acquiring learning images by capturing an image of the vehicle 100 placed in the factory FC in the real space. Therefore, it is possible to reduce the effort of generating the learning data set DS. In addition, it is possible to express a phenomenon that occurs rarely in the real space by the superimposed image GS.

[0060] B. Second embodiment: FIG. 15 is an explanatory diagram showing the configuration of the vehicle 100 in the second embodiment. FIG. 16 is an explanatory diagram showing the configuration of the server device 200 in the second embodiment. FIG. 17 is a flowchart showing the contents of the unmanned driving process in the second embodiment. The system 50 in this embodiment differs from the first embodiment in that the server device 200 does not include a remote control unit 210, and the vehicle 100 runs by autonomous control rather than remote control. The other configurations are the same as those in the first embodiment unless otherwise described.

[0061] As shown in FIG. 15, the memory 112 of the vehicle control device 110 stores a reference route RR and a learned model DM in advance. The communication device 130 can communicate with the external camera 300 by wireless communication. The processor 111 of the vehicle control device 110 functions as a driving control unit 115 by executing a computer program PG1 stored in advance in the memory 112. In this embodiment, the driving control unit 115 controls the actuator group 120 using a driving control signal generated by itself, rather than using a driving control signal acquired from outside the vehicle 100. As shown in FIG. 16, the server device 200 functions as a driving route acquisition unit 220, a driving environment information acquisition unit 230, a vehicle information acquisition unit 240, a superimposed image generation unit 250, a processing unit 260, a label information acquisition unit 270, a data set generation unit 280, and a learning unit 290 by executing a computer program PG2 stored in advance in the memory 202. The learned model DM generated by the learning unit 290 may be transmitted to the vehicle 100 via wireless communication before the vehicle 100 starts traveling, and stored in the memory 112 of the vehicle control device 110, for example.

[0062] As shown in FIG. 17, in this embodiment, the unmanned driving process is executed by the driving control unit 115. In step S210, the driving control unit 115 acquires vehicle position information using the captured image output from the external camera 300 and the learned model DM. In step S220, the driving control unit 115 determines a target position to which the vehicle 100 should next head, using the vehicle position information and the reference route RR. In step S230, the driving control unit 115 generates a driving control signal for driving the vehicle 100 toward the determined target position. In step S240, the driving control unit 115 controls the actuator group 120 using the generated driving control signal, thereby causing the vehicle 100 to travel at the acceleration and steering angle represented by the driving control signal. The driving control unit 115 repeats the acquisition of position information of the vehicle 100, the determination of the target position, the generation of the driving control signal, and the control of the actuator group 120 at a predetermined cycle.

[0063] According to the system 50 of the present embodiment described above, the vehicle 100 can be driven by the autonomous control of the vehicle 100 without remote control from the outside.

[0064] C. Other embodiments: (C1) In the first embodiment, the processes from obtaining vehicle position information to generating a driving control signal are executed by the server device 200. In contrast, at least a part of the processes from obtaining vehicle position information to generating a driving control signal may be executed by the vehicle 100. For example, the following forms (1) to (3) may be used.

[0065] (1) The server device 200 may acquire vehicle position information, determine a target position to which the vehicle 100 should next head, and generate a route from the current location of the vehicle 100 indicated in the acquired vehicle position information to the target position. The server device 200 may generate a route to a target position between the current location and the destination, or may generate a route to the destination. The server device 200 may transmit the generated route to the vehicle 100. The vehicle 100 may generate a driving control signal so that the vehicle 100 drives on the route received from the server device 200, and control the actuator group 120 using the generated driving control signal.

[0066] (2) The server device 200 may acquire vehicle position information and transmit the acquired vehicle position information to the vehicle 100. The vehicle 100 may determine a target position to which the vehicle 100 should next head, generate a route from the current location of the vehicle 100 represented in the received vehicle position information to the target position, generate a driving control signal so that the vehicle 100 travels on the generated route, and control the actuator group 120 using the generated driving control signal.

[0067] (3) In the above-mentioned embodiments (1) and (2), an internal sensor may be mounted on the vehicle 100, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. The internal sensor is a sensor mounted on the vehicle 100. The internal sensor may include, for example, a sensor for detecting a motion state of the vehicle 100, a sensor for detecting an operating state of each part of the vehicle 100, and a sensor for detecting an environment around the vehicle 100. Specifically, the internal sensor may include, for example, a camera, a LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyro sensor, and the like. For example, in the above-mentioned embodiment (1), the server device 200 may acquire a detection result of the internal sensor, and may reflect the detection result of the internal sensor on the route when generating the route. In the above-mentioned embodiment (1), the vehicle 100 may acquire a detection result of the internal sensor, and may reflect the detection result of the internal sensor on the driving control signal when generating the driving control signal. In the above-mentioned embodiment (2), the vehicle 100 may acquire a detection result of the internal sensor, and may reflect the detection result of the internal sensor on the route when generating the route. In the above embodiment (2), the vehicle 100 may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor in the driving control signal when generating the driving control signal.

[0068] (C2) In each of the above embodiments, the vehicle 100 may be equipped with an internal sensor, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. For example, the vehicle 100 may acquire the detection result of the internal sensor, and when generating a route, may reflect the detection result of the internal sensor in the route. The vehicle 100 may acquire the detection result of the internal sensor, and when generating a driving control signal, may reflect the detection result of the internal sensor in the driving control signal.

[0069] (C3) In the second embodiment, the vehicle 100 acquires vehicle position information using the detection result of the external camera 300. In contrast, the vehicle 100 may be equipped with an internal sensor, acquire vehicle position information using the detection result of the internal sensor, determine a target position to which the vehicle 100 should next head, generate a route from the current location of the vehicle 100 represented in the acquired vehicle position information to the target position, generate a travel control signal for traveling the generated route, and control the actuator group 120 using the generated travel control signal. In this case, the vehicle 100 can travel without using any of the detection results of the external camera 300. The vehicle 100 may acquire a target arrival time or traffic congestion information from outside the vehicle 100, and reflect the target arrival time or traffic congestion information in at least one of the route and the travel control signal. In addition, all of the functional configurations of the system 50 may be provided in the vehicle 100. That is, the processing realized by the system 50 in the present disclosure may be realized by the vehicle 100 alone.

[0070] (C4) In the above first embodiment, the server device 200 automatically generates a driving control signal to be transmitted to the vehicle 100. In contrast, the server device 200 may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a control device including a display for displaying an image output from the external camera 300, a steering wheel for remotely controlling the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server device 200 via wired communication or wireless communication, and the server device 200 may generate a driving control signal in accordance with the operation applied to the control device.

[0071] (C5) In each of the above embodiments, the vehicle 100 may have a configuration capable of moving by unmanned driving, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100 may have at least the vehicle control device 110 and the actuator group 120 in order to perform the three functions of "running", "turning", and "stopping" by unmanned driving. When the vehicle 100 acquires information from the outside for unmanned driving, the vehicle 100 may further have the communication device 130. That is, the vehicle 100 capable of moving by unmanned driving may not have at least a part of the interior parts such as the driver's seat and the dashboard, may not have at least a part of the exterior parts such as the bumper and the fender, and may not have a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100 before the vehicle 100 is shipped from the factory FC, or the remaining parts such as the body shell may be attached to the vehicle 100 after the vehicle 100 is shipped from the factory FC in a state in which the remaining parts such as the body shell are not attached to the vehicle 100. Each part may be attached from any direction, such as the upper side, lower side, front side, rear side, right side, or left side of the vehicle 100, and may be attached from the same direction or from different directions. Note that the position of the platform shape may be determined in the same manner as the vehicle 100 in the first embodiment.

[0072] (C6) The vehicle 100 may be manufactured by combining a plurality of modules. A module means a unit composed of a plurality of parts grouped according to the location or function of the vehicle 100. For example, the platform of the vehicle 100 may be manufactured by combining a front module that constitutes the front part of the platform, a central module that constitutes the central part of the platform, and a rear module that constitutes the rear part of the platform. The number of modules that constitute the platform is not limited to three, and may be two or less, or four or more. In addition to or instead of the parts that constitute the platform, parts that constitute parts of the vehicle 100 that are different from the platform may be modularized. The various modules may include any exterior parts such as a bumper or a grille, or any interior parts such as a seat or a console. In addition, not limited to the vehicle 100, any type of moving body may be manufactured by combining a plurality of modules. Such a module may be manufactured, for example, by joining a plurality of parts by welding or a fastener, or may be manufactured by integrally molding at least a part of the parts that constitute the module as one part by casting. The molding method for integrally molding a single component, particularly a relatively large component, is also called gigacast or megacast. For example, the front module, the center module, and the rear module may be manufactured using gigacast.

[0073] (C7) Transporting the vehicle 100 by using the driving of the vehicle 100 through unmanned driving is also called "self-propelled transport." Also, a configuration for realizing self-propelled transport is also called a "vehicle remote-controlled autonomous driving transport system." Also, a production method for producing the vehicle 100 using self-propelled transport is also called "self-propelled production." In self-propelled production, for example, at least a part of the transportation of the vehicle 100 is realized by self-propelled transport in the factory FC where the vehicle 100 is manufactured.

[0074] (C8) In each of the above embodiments, some or all of the functions and processes implemented by software may be implemented by hardware. Also, some or all of the functions and processes implemented by hardware may be implemented by software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0075] (C9) FIG. 18 is a first explanatory diagram that illustrates a trained model DM in another embodiment. FIG. 19 is a second explanatory diagram that illustrates a trained model DM in another embodiment. In each of the above embodiments, the trained model DM is configured to output a mask image when a captured image is input. In contrast, the trained model DM may be configured to output coordinate points related to the contour of the vehicle 100 when a captured image is input. For example, as shown in FIG. 18, it may be configured to output four coordinate points. As shown in FIG. 19, it may be configured to output eight coordinate points. In these cases, the label information includes the position coordinates of the coordinate points. The remote control unit 210 and the traveling control unit 115 can acquire vehicle position information using, for example, the coordinate points output from the trained model DM.

[0076] (C10) In each of the above embodiments, the processor 201 may determine a specific position where the detection accuracy of the vehicle 100 by the trained model MD is relatively low among multiple positions on the travel route of the vehicle 100, and may increase the number of training datasets DS included in the training dataset group DG corresponding to the specific position compared to other positions. For example, in a dark place, a place with many obstacles, or a place where the position or orientation of the vehicle 100 is likely to change due to the influence of disturbance, the processor 201 may increase the number of training datasets DS included in the training dataset group DG when generating a trained model DM for acquiring vehicle position information using an image captured by the external camera 300 placed in a dark place, a place with many obstacles, or a place where the position or orientation of the vehicle 100 is likely to change due to the influence of disturbance, since the detection accuracy of the vehicle 100 by the trained model MD is likely to be low in such places. In such places, the processor 201 may increase the number of training datasets DS included in the training dataset group DG compared to when generating a trained model DM for acquiring vehicle position information using an image captured by the external camera 300 placed in a bright place, a place with few obstacles, or a place where the position or orientation of the vehicle 100 is unlikely to change due to the influence of disturbance. For example, the number of training data sets DS included in the training data set group DG may be increased so as to increase the number of brightness patterns, position patterns of the vehicle 100, and orientation patterns of the vehicle 100. In this case, it is possible to improve the detection accuracy of the vehicle 100 in dark places, places with many obstacles, and places where the position or orientation of the vehicle 100 is likely to change due to the influence of disturbances.

[0077] (C11) In each of the above embodiments, the processor 201 may determine a specific position where the detection accuracy of the vehicle 100 by the trained model MD is relatively low among a plurality of positions on the travel route of the vehicle 100, and generate a learning data set DS according to the environment of the specific position. For example, in a dark place, a place with many obstacles, or a place where the position and orientation of the vehicle 100 are likely to change due to the influence of disturbances, the processor 201 may lower the brightness in the superimposed image GS when generating the trained model DM for acquiring vehicle position information using an image captured by the external camera 300 placed in a dark place, compared to when generating the trained model DM for acquiring vehicle position information using an image captured by the external camera 300 placed in a bright place. For a place where the position and orientation of the vehicle 100 are likely to change, the influence of disturbances may be added to the position and orientation of the vehicle 100. In this case, the detection accuracy of the vehicle 100 can be improved in a dark place, a place with many obstacles, or a place where the position and orientation of the vehicle 100 are likely to change due to the influence of disturbances.

[0078] (C12) In the above first embodiment, the remote control unit 210 may be provided on a computer different from the server device 200. In this case, the computer equipped with the remote control unit 210 is provided with a communication device for communicating with the vehicle 100 via wireless communication, and the computer equipped with the remote control unit 210 acquires the trained model DM generated on the server device 200, and acquires the position and orientation of the vehicle 100 using the trained model DM.

[0079] (C13) In each of the above embodiments, the driving environment information acquisition unit 230 acquires a factory model MF as the driving environment information, the vehicle information acquisition unit 240 acquires a vehicle model MV as the vehicle information, and the superimposed image generation unit 250 generates a superimposed image by superimposing the vehicle model MV and the factory model MF. In contrast, the driving environment information acquisition unit 230 may acquire an image representing the appearance of the factory FC including the driving route as the driving environment information. The image of the factory FC may be a captured image generated by capturing an image of the exterior of the factory FC including the driving route in real space, or may be an image generated by rendering using the factory model MF. The vehicle information acquisition unit 240 may acquire an image representing the exterior of the vehicle 100 as the vehicle information. The image of the vehicle 100 may be a captured image generated by capturing an image of the exterior of the vehicle 100 in real space, or may be an image generated by rendering using the vehicle model MV. The superimposed image generation unit 250 may generate a superimposed image by superimposing the image of the vehicle 100 and the image of the factory FC, in other words, by synthesizing the image of the vehicle 100 and the image of the factory FC. The processing unit 260 may perform a predetermined processing on the image of the vehicle 100 and the image of the factory FC before generating a superimposed image by superimposing the image of the vehicle 100 and the image of the factory FC. After generating a superimposed image by superimposing the image of the vehicle 100 and the image of the factory FC, the processing unit 260 may perform a predetermined processing on the superimposed image.

[0080] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0081] 50...system, 100...vehicle, 110...vehicle control device, 111...processor, 112...memory, 113...input / output interface, 114...internal bus, 115...driving control unit, 120...actuator group, 130...communication device, 200...server device, 201...processor, 202...memory, 203...input / output interface, 204...internal bus, 205...communication device, 206...input device, 207...display device, 210...remote control unit, 220...driving route acquisition unit, 230...driving environment information acquisition unit, 240...vehicle information acquisition unit, 250...superimposed image generation unit, 260...processing unit, 270...label information acquisition unit, 280...data set generation unit, 290...learning unit, 300...external camera

Claims

1. 1. A method comprising: A movement path acquisition step of acquiring a movement path of a moving object; a superimposed image generating step of generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path; a label information acquisition step of acquiring label information relating to a position and an orientation of the moving object; A data set generation step of generating a learning data set including the superimposed image and the label information; The method comprising:

2. 2. The method of claim 1 , The method further comprises a step of performing a predetermined processing on the image or three-dimensional data of the moving environment prior to the superimposed image generating step.

3. 2. The method of claim 1 , The method further comprises a step of performing a predetermined processing on the image or three-dimensional data of the moving object prior to the step of generating the superimposed image.

4. 2. The method of claim 1 , The method further comprises the step of performing a predetermined processing on the superimposed image.

5. 5. The method of claim 4, A method, wherein the processing includes at least one of adding or changing light irradiated onto the moving object and adding or changing a shadow of the moving object.

6. 5. The method according to claim 2, further comprising the steps of: The method, wherein the processing includes modifying at least one of brightness, saturation, and contrast.

7. 2. The method of claim 1 , The superimposed image generating step is performed a plurality of times, A method for making the position of the moving object in a direction parallel to the movement path different in at least two of a plurality of superimposed images generated by performing the superimposed image generating step a plurality of times.

8. 2. The method of claim 1 , The superimposed image generating step is performed a plurality of times, A method for making the position of the moving object different in a direction perpendicular to the movement path in at least two of a plurality of superimposed images generated by performing the superimposed image generating step a plurality of times.

9. 2. The method of claim 1 , The superimposed image generating step is performed a plurality of times, A method for generating a superimposed image by performing the superimposed image generating step a plurality of times, the superimposed images being generated such that the orientation of the moving object with respect to the travel path is made different in at least two of the superimposed images.

10. 2. The method of claim 1 , The superimposed image generating step is performed a plurality of times, A method for generating a superimposed image, the method comprising: making the color of the moving object different in at least two of the superimposed images generated by performing the superimposed image generating step a plurality of times.

11. 2. The method of claim 1 , The label information acquiring step acquires the label information from information associated with an image or three-dimensional data of the moving object.

12. 2. The method of claim 1 , The method further comprises a learning step of performing machine learning using a group of training datasets including the training dataset.

13. 13. The method of claim 12, The method further comprises a step of acquiring information regarding the position and orientation of a moving body moving in the mobile environment using a trained model generated by the learning step and a captured image of the moving body moving in the mobile environment.

14. 2. The method of claim 1 , The superimposed image generating step is performed a plurality of times, Further comprising a learning step of performing machine learning using a group of training data sets including a plurality of the training data sets; determining a specific position among a plurality of positions on the movement path at which the accuracy of the output of the trained model generated by the training step is relatively low; A method for increasing the number of training data sets corresponding to the specific location compared to other locations.

15. 2. The method of claim 1 , The superimposed image generating step is performed a plurality of times, Further comprising a learning step of performing machine learning using a group of training data sets including a plurality of the training data sets; determining a specific position among a plurality of positions on the movement path at which the accuracy of the output of the trained model generated by the training step is relatively low; The method of generating the training dataset according to an environment of the specific location.

16. 1. An apparatus comprising: A movement path acquisition unit that acquires a movement path of a moving object; a superimposed image generating unit that generates a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path; a label information acquisition unit that acquires label information relating to a position and an orientation of the moving object; a dataset generation unit that generates a learning dataset including the superimposed image and the label information; An apparatus comprising:

17. A computer program comprising: A function of acquiring images or three-dimensional data representing a moving object, or images or three-dimensional data representing an environment in which the moving object moves; A function of displaying a screen on which processing conditions for processing the acquired image or three-dimensional data can be selected; a function of performing the processing on the acquired image or three-dimensional data according to the selected processing conditions; A computer program that enables a computer to realize the above.

18. 18. A computer program according to claim 17, comprising: The processing includes changing at least one of brightness, saturation, and contrast.

19. 18. A computer program according to claim 17, comprising: A computer program, wherein the processing includes at least one of adding or changing light irradiated to the moving object and adding or changing a shadow of the moving object.

20. A computer program comprising: A function of acquiring images or three-dimensional data representing a moving object and images or three-dimensional data representing a moving environment of the moving object; a function of displaying a screen on which a superimposition condition for superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing the moving environment can be selected; a function of generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing the moving environment according to the selected superimposition condition; A computer program that enables a computer to realize the above.

21. 21. A computer program according to claim 20, comprising: The superimposition condition includes at least one of a position and an orientation of the moving object relative to the moving environment.

22. A learning dataset used in machine learning, a superimposed image generated by superimposing an image or three-dimensional data representing a moving object and an image or three-dimensional data representing a moving environment of the moving object; Information regarding the position and orientation of the moving object; A training dataset containing

Images