Method and apparatus

Abstract

Provided is a technique for reducing the labor for preparing data for learning. The method includes: a movement path acquisition step of acquiring a movement path of a moving body; a superimposed image generation step for generating a superimposed image by superimposing the image or three-dimensional data representing the moving body and the image or three-dimensional data representing a moving environment including the moving path; a label information acquisition step for acquiring label information relating to the position and orientation of the moving body; and a dataset generation step for generating a learning dataset including the superimposed image and the tag information.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus. Background Art

[0002] There is known a technique for detecting the position of a traveling vehicle using an image captured by a camera located outside the vehicle (for example, Patent Document 1). Prior Art Literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application No. 2017-538619 Summary of the invention Problems to be solved by the invention

[0004] When using artificial intelligence to detect a moving object such as a vehicle from an image captured by a camera, it is necessary to prepare a large amount of data for the artificial intelligence to learn. Therefore, a technology that reduces the effort for preparing learning data is desired. Technical means to solve problems

[0005] The present disclosure can be implemented as the following aspects.

[0006] (1) According to a first aspect of the present disclosure, a method is provided. The method comprises: a moving path acquisition step of acquiring a moving path of a moving object; a superimposed image generation step of superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path to generate a superimposed image; a label information acquisition step of acquiring label information related to the position and orientation of the moving object; and a data set generation step of generating a learning data set including the superimposed image and the label information. According to the method of this aspect, since a training data set including overlapping images is generated, the effort for preparing the training data set can be reduced. (2) The method of the above aspect may further include a step of performing a predetermined process on the image or three-dimensional data of the moving environment before the superimposed image generating step. According to the method of this aspect, it is possible to generate a learning data set including superimposed images in a desired state. (3) The method of the above aspect may further include a step of performing a predetermined process on the image or three-dimensional data of the moving object before the step of generating the superimposed image. According to the method of this aspect, it is possible to generate a learning data set including superimposed images in a desired state. (4) The method of the above aspect may further include a step of performing a predetermined process on the superimposed image. According to the method of this aspect, it is possible to generate a learning data set including superimposed images in a desired state. (5) In the method of the above aspect, the processing may include at least one of adding or changing the light irradiated to the moving object and adding or changing the shadow of the moving object. According to the method of this aspect, the moving object represented by the superimposed image can be made to look different. (6) In the method of the above aspect, the processing may include changing at least one of brightness, chroma, and contrast. According to the method of this aspect, the moving object or moving environment represented by the superimposed image can be made to look different. (7) In the method of the above aspect, the overlapping image generating step may be performed multiple times, and in at least two of the multiple overlapping images generated by performing the overlapping image generating step multiple times, the position of the moving body in the direction parallel to the moving path may be different. According to the method of this aspect, a plurality of superimposed images in which the positions of the moving object in the direction parallel to the moving path are different from each other can be generated. (8) In the method described above, the overlapping image generating step may be performed multiple times, and in at least two of the multiple overlapping images generated by performing the overlapping image generating step multiple times, the position of the moving body in the direction perpendicular to the moving path may be different. According to the method of this aspect, a plurality of superimposed images in which the positions of the moving object in the direction perpendicular to the moving path are different from each other can be generated. (9) In the method of the above aspect, the superimposed image generating step may be executed multiple times, and the orientation of the moving body relative to the moving path may be different in at least two of the multiple superimposed images generated by executing the superimposed image generating step multiple times. According to the method of this aspect, a plurality of superimposed images in which the orientations of the moving object with respect to the moving path are different from each other can be generated. (10) In the method of the above aspect, the superimposed image generating step may be executed a plurality of times, and the color of the moving object may be different in at least two of the plurality of superimposed images generated by executing the superimposed image generating step a plurality of times. According to the method of this aspect, a plurality of superimposed images of the moving object having different colors from each other can be generated. (11) In the method of the above aspect, in the tag information acquisition step, the tag information may be acquired based on information associated with the image or three-dimensional data of the moving object. According to the method of this aspect, it is possible to eliminate the time and effort of separately preparing information on the position and orientation of the moving object. (12) The method of the above aspect may further include a learning step of executing machine learning using a learning data set group including the learning data set. According to the method of this aspect, it is possible to execute machine learning using a learning data set including overlapping images. (13) The method of the above aspect may further include a step of obtaining information related to the position and orientation of the moving object moving in the moving environment using the learned model generated by the learning step and a captured image of the moving object moving in the moving environment. According to the method of this aspect, it is possible to obtain information on the position and orientation of a moving object moving in a moving environment using a learned model. (14) In the method described above, the overlapping image generating step may be executed multiple times, and there may also be a learning step for executing machine learning using a learning data set group including a plurality of the learning data sets, determining a specific position among a plurality of positions on the moving path where the output accuracy of the learned model generated by the learning step is relatively low, so that the number of the learning data sets corresponding to the specific position is greater than that of other positions. According to the method of this aspect, it is possible to improve the output accuracy of the learned model at a specific position. (15) In the method described above, the overlapping image generating step may be executed multiple times, and there is also a learning step for executing machine learning using a learning data set group including a plurality of the learning data sets, determining a specific position among a plurality of positions on the moving path where the output accuracy of the learning model generated by the learning step is relatively low, and generating the learning data set corresponding to the environment of the specific position. According to the method of this aspect, it is possible to improve the output accuracy of the learned model at a specific position. (16) According to the second aspect of the present disclosure, there is provided an apparatus comprising: a movement path acquisition unit that acquires a movement path of a moving object; a superimposed image generation unit that generates a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the movement path; a label information acquisition unit that acquires label information related to a position and orientation of the moving object; and a data set generation unit that generates a learning data set including the superimposed image and the label information. According to the device of this aspect, since a training data set including overlapping images is generated, the effort for preparing the training data set can be reduced. (17) According to a third aspect of the present disclosure, there is provided a computer program. The computer program causes a computer to implement the following functions: a function of acquiring an image or three-dimensional data representing a moving object, or an image or three-dimensional data representing a moving environment of the moving object; a function of displaying a screen capable of selecting a processing condition related to processing to be performed on the acquired image or three-dimensional data; and a function of performing the processing corresponding to the selected processing condition on the acquired image or three-dimensional data. According to the computer program of this aspect, it is possible to generate a superimposed image according to the processing conditions selected by the user. (18) In the computer program of the above aspect, the processing may include changing at least one of brightness, chroma, and contrast. According to the computer program of this aspect, a superimposed image in a desired state can be generated. (19) In the computer program of the above aspect, the processing may include at least one of adding or changing the light irradiated to the moving object and adding or changing the shadow of the moving object. According to the computer program of this aspect, a superimposed image in a desired state can be generated. (20) According to a fourth aspect of the present disclosure, a computer program is provided. The computer program causes a computer to implement the following functions: a function of acquiring an image or three-dimensional data representing a moving object and an image or three-dimensional data representing a moving environment of the moving object; a function of displaying a screen capable of selecting an overlapping condition, the overlapping condition being a condition for overlapping the image or three-dimensional data representing the moving object and the image or three-dimensional data representing the moving environment; and a function of overlapping the image or three-dimensional data representing the moving object and the image or three-dimensional data representing the moving environment to generate an overlapping image based on the selected overlapping condition. According to the computer program of this aspect, it is possible to generate a superimposed image according to the superimposed condition selected by the user. (21) In the computer program of the above aspect, the overlapping condition may include at least one of the position and orientation of the mobile object relative to the mobile environment. According to the computer program of this aspect, a superimposed image in a desired state can be generated. (22) According to a fifth aspect of the present disclosure, there is provided a learning data set for machine learning. The learning data set includes an overlapped image generated by overlapping an image or three-dimensional data representing a moving object and an image or three-dimensional data representing a moving environment of the moving object, and information related to the position and orientation of the moving object. According to the learning data set of this aspect, the learning data set can be used to generate a learned model for acquiring information on the position and orientation of a moving object. The present disclosure can also be implemented in various forms other than methods, devices, computer programs, and learning data sets, such as systems and recording media recording computer programs. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is an explanatory diagram showing the configuration of the system according to the first embodiment. Figure 2 It is an explanatory diagram showing the structure of the vehicle according to the first embodiment. Figure 3 It is an explanatory diagram showing the configuration of the server device according to the first embodiment. Figure 4 This is a flowchart showing the contents of the unmanned driving process according to the first embodiment. Figure 5 This is a flowchart showing an example of a method for generating shape data and a method for acquiring a position of a vehicle using the shape data. Figure 6 It shows execution Figure 5 Schematic diagram of various image examples of the method shown. Figure 7 This is an explanatory diagram for explaining the details of the coordinate point calculation process. Figure 8 This is an explanatory diagram for explaining the method of calculating basic coordinate points. Fig. 9 This is a first explanatory diagram for explaining the details of the position changing step. Fig.10 This is a second explanatory diagram for explaining the details of the position conversion process. Fig.11 : is a flowchart showing the content of the learned model generation method. Fig.12 It is an explanatory diagram schematically showing a state where a plant model and a vehicle model overlap. Fig.13 FIG. 2 is an explanatory diagram schematically showing a learning data set group. Fig.14 It is an explanatory diagram schematically showing a selection screen. Fig.15 It is an explanatory diagram showing the structure of a vehicle according to the second embodiment. Fig.16 It is an explanatory diagram showing the configuration of a server device according to the second embodiment. Fig.17 It is a flowchart showing the contents of the unmanned driving process according to the second embodiment. Fig.18 This is a first explanatory diagram schematically showing a learned model according to another embodiment. Fig.19 This is a second explanatory diagram schematically showing a learned model according to another embodiment. DETAILED DESCRIPTION

[0008] A. First Implementation Method: Figure 1 It is an explanatory diagram showing the configuration of the system 50 in the first embodiment. Figure 2 It is an explanatory diagram showing the configuration of a vehicle 100 in the first embodiment. Figure 3 2 is an explanatory diagram showing the configuration of the server device 200 in the first embodiment. Figure 1 As shown, in the present embodiment, the system 50 includes a vehicle 100 as a moving object, a server device 200 , and at least one external camera 300 .

[0009] In the present disclosure, a "mobile body" refers to an object that can move. The mobile body is, for example, a vehicle, an electric vertical take-off and landing aircraft (so-called flying car). The vehicle may be a vehicle that runs on wheels or a vehicle that runs on tracks. That is, there is no particular limitation on the form of the vehicle, and the vehicle may be, for example, a car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, a construction vehicle, and the like. Vehicles include electric vehicles (BEV: Battery Electric Vehicle), gasoline vehicles, hybrid electric vehicles (HEV: Hybrid Electric Vehicle), and fuel cell vehicles (FCEV: Fuel Cell Electric Vehicle), and the like. In addition, in the case where the mobile body is other than a vehicle, the expressions "vehicle" and "car" in the present disclosure may be appropriately replaced with "mobile body", and the expression "running" may be appropriately replaced with "moving".

[0010] The vehicle 100 is configured to be able to travel by unmanned driving. Here, "unmanned driving" refers to driving that does not rely on the driving operation of the passengers riding in the vehicle 100. "Driving operation" refers to an operation related to at least one of "driving", "turning" and "stopping" of the vehicle 100. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 100, or by autonomous control of the vehicle 100. In the vehicle 100 that travels by unmanned driving, passengers who do not perform driving operations can also ride. Among the passengers who do not perform driving operations, for example, there are people who only sit on the seats of the vehicle 100, and people who ride in the vehicle 100 and perform actions different from driving operations. Actions different from driving operations include, for example, assembly operations relative to components of the vehicle 100, operations to inspect the vehicle 100, and operations of switches provided on the vehicle 100. In addition, driving performed by driving operations of passengers is sometimes referred to as "manned driving".

[0011] In the present disclosure, “remote control” includes “full remote control” in which all actions of the vehicle 100 are determined completely from outside the vehicle 100, and “partial remote control” in which part of the actions of the vehicle 100 are determined from outside the vehicle 100. In addition, “autonomous control” includes “full autonomous control” in which the vehicle 100 autonomously controls its own actions without receiving any information from a device located outside the vehicle 100, and “partial autonomous control” in which the vehicle 100 autonomously controls its own actions using information received from a device located outside the vehicle 100.

[0012] In the present embodiment, the system 50 is used in a factory FC for manufacturing a vehicle 100. The reference coordinate system of the factory FC is the global coordinate system GC. That is, any position in the factory FC can be represented by the coordinates of X, Y, and Z in the global coordinate system GC. The factory FC includes a first place PL1 and a second place PL2. The first place PL1 and the second place PL2 are connected by a driving road TR on which the vehicle 100 can travel. In the factory FC, a plurality of external cameras 300 are arranged along the driving road TR. The external camera 300 is provided with a communication device not shown in the figure, and can communicate with the server device 200 by wired communication or wireless communication. The position and orientation of each external camera 300 in the factory FC are adjusted in advance. The vehicle 100 moves from the first place PL1 to the second place PL2 through the driving road TR by unmanned driving.

[0013] like Figure 2 As shown, the vehicle 100 includes a vehicle control device 110 for controlling various parts of the vehicle 100, an actuator group 120 driven under the control of the vehicle control device 110, and a communication device 130 for communicating with the server device 200 through wireless communication. The actuator group 120 includes at least one actuator. In this embodiment, the actuator group 120 includes an actuator of a driving device for accelerating the vehicle 100, an actuator of a steering device for changing the direction of travel of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The driving device includes a battery, a driving motor driven by the power of the battery, and a wheel rotated by the driving motor. The driving motor is included in the actuator of the driving device.

[0014] The vehicle control device 110 is constituted by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected to each other for bidirectional communication via the internal bus 114. The input / output interface 113 is connected to the actuator group 120 and the communication device 130.

[0015] The processor 111 functions as a travel control unit 115 by executing a computer program PG1 pre-stored in the memory 112. The travel control unit 115 drives the vehicle 100 by controlling the actuator group 120. When a passenger is on the vehicle 100, the travel control unit 115 controls the actuator group 120 according to the operation of the passenger, thereby enabling the vehicle 100 to drive. In the present embodiment, regardless of whether a passenger is on the vehicle 100, the travel control unit 115 controls the actuator group 120 according to the travel control signal received from the server device 200, thereby enabling the vehicle 100 to drive. In the present embodiment, the travel control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In addition, in other embodiments, the travel control signal may include the speed of the vehicle 100 as a parameter instead of the acceleration of the vehicle 100, or may include the speed of the vehicle 100 as a parameter in addition to the acceleration of the vehicle 100.

[0016] like Figure 3 As shown, the server device 200 is composed of a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 so as to be capable of bidirectional communication. The input / output interface 203 is connected to a communication device 205 for communicating with the vehicle 100 by wireless communication. In the present embodiment, the communication device 205 can communicate with the external camera 300 by wired communication or wireless communication. In addition to the communication device 205, the input / output interface 203 is also connected to an input device 206 and a display device 207. The input device 206 is, for example, a mouse, a keyboard, etc. The display device 207 is, for example, a liquid crystal display, etc.

[0017] The processor 201 functions as a remote control unit 210, a driving path acquisition unit 220, a driving environment information acquisition unit 230, a vehicle information acquisition unit 240, a superimposed image generation unit 250, a processing unit 260, a label information acquisition unit 270, a data set generation unit 280, and a learning unit 290 by executing a computer program PG2 pre-stored in the memory 202. In addition, the driving path acquisition unit 220 is sometimes referred to as a moving path acquisition unit, the driving environment information acquisition unit 230 is sometimes referred to as a moving environment information acquisition unit, and the vehicle information acquisition unit 240 is sometimes referred to as a moving body information acquisition unit.

[0018] The remote control unit 210 remotely controls the vehicle 100 to drive the vehicle 100. In the present embodiment, the remote control unit 210 remotely controls the vehicle 100 so that the vehicle 100 drives along a reference route RR pre-stored in the memory 202. The remote control unit 210 uses the learned model DM pre-stored in the memory 202 and the captured image obtained from the external camera 300 to obtain vehicle position information including the position and orientation of the vehicle 100, and remotely controls the vehicle 100 using the vehicle position information and the reference route RR. The learned model DM is prepared in advance for each external camera 300.

[0019] The travel route acquisition unit 220 acquires the travel route of the vehicle 100 . In the present embodiment, the travel route acquisition unit 220 acquires a reference route RR stored in advance in the memory 202 as the travel route of the vehicle 100 .

[0020] The driving environment information acquisition unit 230 acquires driving environment information indicating the appearance of the driving environment of the vehicle 100. The driving environment information is an image indicating the appearance of the driving environment of the vehicle 100 or three-dimensional data indicating the appearance of the driving environment of the vehicle 100. In the present embodiment, the driving environment information acquisition unit 230 acquires a factory model MF pre-stored in the memory 202 as the driving environment information. The factory model MF is three-dimensional data indicating the appearance of the factory FC as the driving environment of the vehicle 100. The appearance of the factory FC includes the appearance of the building of the factory FC, the appearance of various equipment installed inside and outside the building of the factory FC, the appearance of the road surface of the driving road TR in the factory FC, and the like.

[0021] The vehicle information acquisition unit 240 acquires vehicle information indicating the appearance of the vehicle 100. The vehicle information is an image indicating the appearance of the vehicle 100 or three-dimensional data indicating the appearance of the vehicle 100. In the present embodiment, the vehicle information acquisition unit 240 acquires a vehicle model MV pre-stored in the memory 202 as the vehicle information. The vehicle model MV is three-dimensional data indicating the appearance of the vehicle 100.

[0022] The vehicle model MV and the factory model MF are generated using, for example, 3D CAD software or 3D CG software. The vehicle model MV and the factory model MF may be generated on the server device 200 or on a computer different from the server device 200. The appearance of the vehicle model MV preferably reproduces the appearance of the vehicle 100 as accurately as possible, and the appearance of the factory model MF preferably reproduces the appearance of the factory FC as accurately as possible.

[0023] The superimposed image generation unit 250 generates a superimposed image by superimposing the vehicle information and the running environment information. In the present embodiment, the superimposed image generation unit 250 generates a superimposed image by superimposing the vehicle model MV and the plant model MF, and outputs the generated superimposed image.

[0024] The processing unit 260 processes at least one of the vehicle information, the driving environment information, and the overlapping image. The processing may include at least one of adding or changing the light irradiated on the vehicle 100 and adding or changing the shadow of the vehicle 100. The processing may also include changing at least one of the brightness, chroma, and contrast. In the present embodiment, the processing unit 260 can change the color and texture of at least a portion of the vehicle 100 represented by the vehicle model MV by processing the vehicle model MV. The processing unit 260 can change the color and texture of at least a portion of the factory FC represented by the factory model MF by processing the factory model MF. The processing unit 260 can change at least one of the brightness, chroma, and contrast of at least a portion of the overlapping image by processing the overlapping image.

[0025] The label information acquisition unit 270 acquires label information related to at least one of the position and the orientation of the vehicle 100. The label information is used as a correct answer label in supervised machine learning.

[0026] The data set generation unit 280 generates a learning data set including a superimposed image and label information associated with the superimposed image.

[0027] The learning unit 290 generates a learned model DM by executing machine learning using a learning data set group including at least one learning data set.

[0028] Figure 4 is a flowchart showing the contents of the unmanned driving process in the present embodiment. In the present embodiment, the unmanned driving process is performed by the remote control unit 210 of the server device 200 and the driving control unit 115 of the vehicle 100. In step S110, the remote control unit 210 obtains the vehicle position information of the vehicle 100 using the captured image output from the external camera 300 and the learned model DM. The vehicle position information is position information that serves as a basis for generating a driving control signal. In the present embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC.

[0029] Specifically, in step S110, the remote control unit 210 obtains shape data related to the outline of the vehicle 100 from the captured image using, for example, a learned model DM using artificial intelligence, calculates the coordinates of the positioning point of the vehicle 100 in the coordinate system of the captured image, that is, the local coordinate system of the factory FC, and converts the calculated coordinates into coordinates in the global coordinate system GC, thereby obtaining the position of the vehicle 100. In the present embodiment, the learned model DM is pre-generated by a learned model generation method described later and stored in the memory 202. The remote control unit 210 estimates the direction of the movement vector of the vehicle 100 calculated from the position change of the feature point of the vehicle 100 between frames of the captured image, for example, using the optical flow method, thereby obtaining the direction of the vehicle 100.

[0030] In step S120, the remote control unit 210 determines the target position to which the vehicle 100 should go next. In the present embodiment, the target position is represented by the coordinates of X, Y, and Z in the global coordinate system GC. In the memory 202, a reference path RR is pre-stored as the path on which the vehicle 100 should travel. The path is represented by a node showing a departure point, a node showing a passing point, a node showing a destination, and a link connecting each node. The remote control unit 210 uses the vehicle position information and the reference path RR to determine the target position to which the vehicle 100 should go next. The remote control unit 210 determines the target position on the reference path RR that is ahead of the current position of the vehicle 100.

[0031] In step S130, the remote control unit 210 generates a travel control signal for causing the vehicle 100 to travel toward the determined target position. The remote control unit 210 calculates the travel speed of the vehicle 100 based on the change in the position of the vehicle 100, and compares the calculated travel speed with a predetermined target speed of the vehicle 100. In general, the remote control unit 210 determines the acceleration so as to accelerate the vehicle 100 when the travel speed is lower than the target speed, and determines the acceleration so as to decelerate the vehicle 100 when the travel speed is higher than the target speed. When the vehicle 100 is on the reference path RR, the remote control unit 210 determines the acceleration and the steering angle so as not to deviate from the reference path RR, and when the vehicle 100 is not on the reference path RR, in other words, when the vehicle 100 deviates from the reference path RR, determines the acceleration and the steering angle so as to return the vehicle 100 to the reference path RR.

[0032] In step S140, the remote control unit 210 transmits the generated travel control signal to the vehicle 100. The remote control unit 210 repeatedly acquires the position of the vehicle 100, determines the target position, generates the travel control signal, transmits the travel control signal, etc. at a predetermined cycle.

[0033] In step S150, the travel control unit 115 receives the travel control signal transmitted from the server device 200. In step S160, the travel control unit 115 controls the actuator group 120 using the travel control signal received from the server device 200, thereby causing the vehicle 100 to travel at the acceleration and steering angle indicated by the travel control signal. The travel control unit 115 repeatedly receives the travel control signal and controls the actuator group 120 at a predetermined cycle. According to the system 50 in this embodiment, the vehicle 100 can be driven by remote control, so the vehicle 100 can be moved without using a transport device such as a crane or a conveyor.

[0034] right Figure 4 The method of obtaining the vehicle position information in step S110 is described in more detail. Figure 5 This is a flowchart showing an example of a method for generating shape data and a method for acquiring the position of the vehicle 100 using the shape data. Figure 6 It shows execution Figure 5 Schematic diagram of examples of various images Im1 to Im6 in the case of the method shown. Figure 6 It is shown in Figure 5 In the present embodiment, the positioning point 10e of the vehicle 100 is the rear end of the left side surface of the vehicle 100. In addition, the positioning point 10e of the vehicle 100 may be other than the rear end of the left side surface of the vehicle 100.

[0035] exist Figure 5 In the image acquisition process (step S410) shown in FIG. 1 , the remote control unit 210 acquires the original image Im1 output from the external camera 300. In the distortion correction process (step S420), the remote control unit 210 generates a corrected image Im2 as a processed image by correcting the distortion of the original image Im1. In the rotation process process (step S430), the remote control unit 210 rotates the corrected image Im2. Thus, the remote control unit 210 generates a rotated image Im3 as a processed image. In the cropping process process (step S440), the remote control unit 210 deletes the unnecessary area A2 other than the necessary area A1 composed of the vehicle 100 and the surrounding area of ​​the vehicle 100 from the rotated image Im3 among the areas constituting the rotated image Im3. Thus, the remote control unit 210 generates a processed image Im4 as a processed image. Each of steps S420 to S440 is a pre-processing step for improving the detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4. Therefore, at least one of the steps S420 to S440 may be omitted.

[0036] In the detection process (step S450), the remote control unit 210 inputs the processed image Im4 to the learned model DM. Thus, the remote control unit 210 detects the vehicle 100 from the processed image Im4 and obtains the first mask image Im5 as shape data Da. The first mask image Im5 is an image in which a mask area Ms is added to the processed image Im4 by masking the area showing the vehicle 100 among the areas constituting the processed image Im4.

[0037] In the perspective transformation process (step S460), the remote control unit 210 generates a second mask image Im6 by performing perspective transformation on the first mask image Im5. The remote control unit 210 uses, for example, predetermined perspective transformation parameters to perspective transform the first mask image Im5 into a bird's-eye view image observed from a viewpoint above the vehicle 100 that is approximately perpendicular to the road surface Rs. The perspective transformation parameters are, for example, parameters related to the position information and internal parameters of the external camera 300 obtained through calibration. Thus, the remote control unit 210 generates a second mask image Im6 represented by an image coordinate system from the first mask image Im5 represented by the camera coordinate system. The camera coordinate system is a coordinate system that takes the focus of the external camera 300 as the origin and has coordinate axes represented by the Xc axis and the Yc axis orthogonal to the Xc axis. The image coordinate system is a coordinate system that takes a point on the image plane as the origin and has coordinate axes represented by the Xi axis and the Yi axis orthogonal to the Xi axis.

[0038] In the coordinate point calculation step (step S470 ), the remote control unit 210 calculates an image coordinate point indicating the position of the vehicle 100 in the image coordinate system. Figure 7 1 is an explanatory diagram for explaining the details of the coordinate point calculation process. To calculate the image coordinate point, the remote control unit 210 first calculates the basic coordinate point P0 based on the first circumscribed rectangle R1 set for the mask region Ms in the first mask image Im5 which is the image before perspective transformation. Figure 8: is an explanatory diagram for explaining the calculation method of the basic coordinate point P0. In order to calculate the basic coordinate point P0, the remote control unit 210 sets a basic circumscribed rectangle R0 for the mask area Ms in the first mask image Im5. Next, the remote control unit 210 rotates the first mask image Im5 by a desired amount of rotation, using the center of gravity C of the mask area Ms as the rotation center, in such a way that the direction of the movement vector V of the vehicle 100 corresponding to the mask area Ms in the first mask image Im5 is oriented in a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device displaying the first mask image Im5. Next, the remote control unit 210 sets a first circumscribed rectangle R1 for the mask area Ms of the rotated first mask image Im5 in such a way that the long side is parallel to the movement vector V. Next, the remote control unit 210 rotates the first mask image Im5 to which the first circumscribed rectangle R1 is attached by the above-mentioned amount of rotation, using the center of gravity C of the mask area Ms as the rotation center. Thus, the remote control unit 210 sets the coordinate point of one of the four vertices of the first circumscribed rectangle R1 , which has coordinates closest to the positioning point 10 e of the vehicle 100 , as the basic coordinate point P0 .

[0039] Then, if Figure 7 As shown, the remote control unit 210 performs perspective transformation on the first mask image Im5 after reverse rotation, in other words, the first mask image Im5 after calculating the basic coordinate point P0. As a result, the remote control unit 210 sets the coordinate point corresponding to the basic coordinate point P0 in the first circumscribed rectangle R1 deformed by the perspective transformation as the first coordinate point P1.

[0040] Next, the remote control unit 210 sets a second circumscribed rectangle R2 for the mask area Ms in the second mask image Im6 after perspective transformation of the first mask image Im5. Then, the remote control unit 210 sets the vertex of the second circumscribed rectangle R2 that represents the same position as the first coordinate point P1 as the second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points that represent the same position, so they have a correlation relationship with each other.

[0041] Next, the remote control unit 210 performs a correction to replace the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 according to the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the coordinate values ​​of the second coordinate point P2. When the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the remote control unit 210 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 in the Yi direction of the first coordinate point P1 is greater than the coordinate value Yi2 in the Yi direction of the second coordinate point P2 (Yi1>Yi2), the remote control unit 210 replaces the coordinate value Yi1 in the Yi direction of the first coordinate point P1 with the coordinate value Yi2 in the Yi direction of the second coordinate point P2. Figure 7 In the example shown, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. In addition, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is less than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has the coordinates (Xi2, Yi1). In this way, the remote control unit 210 calculates the image coordinate point P3 representing the position of the vehicle 100 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2.

[0042] In the position conversion process (step S480), the remote control unit 210 converts the image coordinate point P3 into a vehicle coordinate point, thereby calculating the vehicle coordinate point representing the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC. The remote control unit 210 converts the image coordinate point P3 into a vehicle coordinate point using the relational equations (1) to (3) described later, which include the vehicle coordinate point as a target variable and include the image coordinate point P3, the shooting parameter, and the vehicle parameter as explanatory variables. The shooting parameter is a parameter related to the distance of the external camera 300 from a predetermined reference point. In the present embodiment, the shooting parameter is the height H (described later) of the external camera 300 from the road surface Rs. Fig. 9 The vehicle parameter is a parameter related to the distance between the positioning point 10e of the vehicle 100 and the reference point. In the present embodiment, the vehicle parameter is the height h (described later) of the positioning point 10e of the vehicle 100 from the road surface Rs. Fig. 9 ).

[0043] Fig. 9 This is a first explanatory diagram for explaining the details of the position conversion process. Fig. 9 , the diagram shows a state where the vehicle 100 is viewed from the left side. Fig.10 2 is a second explanatory diagram for explaining the details of the position conversion process. Fig.10 , the diagram shows a state where the vehicle 100 is viewed from the roof side. Fig. 9 and Fig.10 The global coordinate system GC shown is a coordinate system having a fixed coordinate point Pf representing an arbitrary reference position of the road surface Rs as an origin and having coordinate axes represented by an Xg axis and a Yg axis orthogonal to the Xg axis. The photographing coordinate point Pc is a coordinate point representing the position of the external camera 300 outputting the original image Im1 used in calculating the image coordinate point P3 in the global coordinate system GC. The fixed coordinate point Pf and the photographing coordinate point Pc are stored in advance in the memory 202, for example.

[0044] like Fig. 9 As shown, the observation distance on the XgYg plane between the position of the external camera 300 and the position of the vehicle 100 (image coordinate point P3) is set to Do, the observation error is set to ΔD, the height [m] of the external camera 300 from the road surface Rs as a shooting parameter is set to H, and the height [m] of the positioning point 10e of the vehicle 100 from the road surface Rs as a vehicle parameter is set to h. In this case, the observation error ΔD is expressed by the following formula (1). ΔD=h / H×Do…(1) That is, the larger the observation distance Do is, the larger the observation error ΔD is.

[0045] When the actual distance between the position of the external camera 300 and the position of the anchor point 10 e of the vehicle 100 is set as the first distance D, the first distance D is expressed by the following equation (2). D=Do×(1-h / H)…(2) That is, the first distance D is determined by the observation distance Do, the height H of the external camera 300 as a photographing parameter, and the height h of the positioning point 10 e of the vehicle 100 as a vehicle parameter.

[0046] like Fig.10 As shown, when the distance between the reference position and the estimated position of the vehicle 100 is set as the estimated distance Dp and the actual distance between the reference position and the vehicle 100 is set as the second distance Dt, the second distance Dt is expressed by the following equation (3). Dt=Dp×(1-h / H)…(3)

[0047] Here, the estimated distance Dp can be calculated using the third distance Dc, which is the actual distance obtained from the fixed coordinate point Pf and the photographed coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the remote control unit 210 can calculate the vehicle coordinate point Pv using the second distance Dt and the fixed coordinate point Pf obtained by correcting the estimated distance Dp using the above formula (3). At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC, and therefore corresponds to the position of the vehicle 100 in the actual space.

[0048] Fig.11 1 is a flowchart showing the contents of a learned model generation method including a learning data set generation method in the present embodiment. Fig.12 It is an explanatory diagram schematically showing a state where the plant model MF and the vehicle model MV overlap. Fig.13 Schematically shows a diagram of a learning data set group DG. In the present embodiment, the learning completed model generation method is executed by the processor 201 of the server device 200. First, in step S510, the driving path acquisition unit 220 acquires the driving path of the vehicle 100 in the real space. In the present embodiment, the driving path acquisition unit 220 acquires the reference path RR stored in the memory 202 as the driving path of the vehicle 100 in the real space. In addition, step S510 is sometimes referred to as a driving path acquisition process or a moving path acquisition process.

[0049] In step S520, the driving environment information acquisition unit 230 acquires driving environment information indicating the appearance of the driving environment of the vehicle 100, and the vehicle information acquisition unit 240 acquires vehicle information indicating the appearance of the vehicle 100. In the present embodiment, the driving environment information acquisition unit 230 acquires the factory model MF pre-stored in the memory 202 as the driving environment information, and the vehicle information acquisition unit 240 acquires the vehicle model MV pre-stored in the memory 202 as the vehicle information.

[0050] In step S530, as a pre-processing, the processing unit 260 processes at least one of the vehicle model MV and the factory model MF as needed. The processing unit 260 can change the color and texture of at least a portion of the vehicle 100 represented by the vehicle model MV by, for example, processing the vehicle model MV. For example, the processing unit 260 can also change the color of the door mirrors and the roof of the vehicle model MV. For example, the processing unit 260 can also perform processing to change the texture of the vehicle model MV from a metal system to a matte system through texture mapping. By changing the texture of the vehicle model MV, it is possible to change the highlights and shadows generated on the surface of the vehicle model MV due to the light irradiated from the virtual light source VL to the vehicle model MV in the rendering described later. The processing unit 260 can change the color and texture of at least a portion of the factory FC represented by the factory model MF by, for example, processing the factory model MF. In addition, the process of step S530 can also be omitted.

[0051] In step S540, the overlapping image generating unit 250 overlaps the vehicle model MV and the factory model MF to generate an overlapping image GS. Fig.12 As shown, the overlapping image generating unit 250 overlaps the vehicle model MV and the factory model MF by configuring the factory model MF and the vehicle model MV in the virtual space VS. In the following description, the three-dimensional data obtained by overlapping the vehicle model MV and the factory model MF is referred to as the overlapping model MS. The overlapping image generating unit 250 generates the overlapping image GS from the overlapping model MS by rendering. Rendering refers to generating a two-dimensional image based on three-dimensional data. The rendering technique used in the generation of the overlapping image GS is preferably physically based rendering. According to physically based rendering, the behavior of light in the real space can be reproduced as accurately as possible. In the virtual space VS, in addition to the vehicle model MV and the factory model MF, a virtual viewpoint VC and a virtual light source VL are also configured. The virtual viewpoint VC is configured so that the relative position and orientation of the factory model MF and the virtual viewpoint VC in the virtual space VS are the same as the relative position and orientation of the factory FC and the external camera 300 in the real space. The virtual light source VL is preferably configured at the position of the light source in the real space. Multiple virtual light sources VL can also be configured in the virtual space VS. For example, when a plurality of lighting devices are arranged in the factory FC in the real space, a virtual light source VL can be arranged at the position of each lighting device. In the overlapping image GS, the overlapping model MS observed from the virtual viewpoint VC is shown. In the overlapping image GS, the shadow caused by the light irradiated from the virtual light source VL to the overlapping model MS is shown. In addition, step S540 is sometimes referred to as the overlapping image generation process.

[0052] In step S550, as post-processing, the processing unit 260 processes the overlapped image GS as needed. For example, the processing unit 260 can change at least one of the brightness, saturation, and contrast of at least a portion of the overlapped image GS by processing the overlapped image GS. In addition, the process of step S550 can also be omitted.

[0053] In step S560, the label information acquisition unit 270 acquires label information LB related to the position and orientation of the vehicle 100. In the present embodiment, the label information acquisition unit 270 acquires information of a mask area Ms for acquiring the position and orientation of the vehicle 100 as the label information LB. The label information acquisition unit 270 can acquire the mask area Ms by, for example, acquiring information of pixels that depict the vehicle model MV among a plurality of pixels constituting the overlapped image GS. In addition, step S560 is sometimes referred to as a label information acquisition process.

[0054] In step S570, the data set generation unit 280 generates a learning data set DS including the overlapped image GS and the label information LB. Fig.13 As shown, in the present embodiment, the learning data set DS includes one overlapped image GS and one piece of label information LB corresponding to the overlapped image GS. In addition, step S570 may be referred to as a learning data set generating step.

[0055] In step S580, the data set generation unit 280 determines whether to generate other types of learning data sets DS. For example, the data set generation unit 280 determines to generate other types of learning data sets DS when the generation of the predetermined number of learning data sets DS has not been completed, and determines not to generate other types of learning data sets DS when the generation of the predetermined number of learning data sets DS has been completed.

[0056] If it is determined in step S580 that another type of learning data set DS is to be generated, the processing from step S530 to step S580 is performed to generate another type of learning data set DS. The process from step S530 to step S580 is repeated until it is determined in step S580 that another type of learning data set DS is not to be generated. The superimposed images GS included in the other types of learning data sets DS are different from each other. In the present embodiment, the superimposed image generating unit 250 makes the position of the vehicle 100 in the direction parallel to the travel path different in at least two of the plurality of superimposed images GS. In addition, in the present embodiment, the superimposed image generating unit 250 also makes the position of the vehicle 100 in the direction perpendicular to the travel path different in at least two of the plurality of superimposed images GS. The direction perpendicular to the travel path mentioned here is not a direction perpendicular to the road surface of the travel road TR, but a direction parallel to the road surface of the travel road TR. In addition, in the present embodiment, the superimposed image generating unit 250 also makes the orientation of the vehicle 100 relative to the travel path different in at least two of the plurality of superimposed images GS. The superimposed image generation unit 250 changes the position and orientation of the vehicle 100 in the superimposed image GS by changing the position and orientation of the vehicle model MV relative to the plant model MF. For example, the superimposed image generation unit 250 generates 10 superimposed images in which the position of the vehicle 100 in the direction parallel to the driving path is different, 10 superimposed images in which the position of the vehicle 100 in the direction perpendicular to the driving path is different, and 10 superimposed images in which the orientation of the vehicle 100 at each position is different, totaling 10×10×10=1000 superimposed images. The processing unit 260 may also make at least a part of the color of at least two of the plurality of superimposed images GS different. In this case, the superimposed image generation unit 250 generates, for example, 10 superimposed images in which the color is further changed in addition to the 1000 superimposed images in which the position and orientation of the vehicle 100 is different in 1000, totaling 1000×10=10000 superimposed images. The label information acquisition unit 270 acquires label information corresponding to each superimposed image GS.

[0057] In step S590, the learning unit 290 performs machine learning by using a learning data set group DG including a plurality of learning data sets DS, thereby generating a learned model DM. In the present embodiment, the learning unit 290 stores the generated learned model DM in the memory 202. The learned model DM can use, for example, a convolutional neural network (hereinafter referred to as CNN) that implements either semantic segmentation or instance segmentation. When learning the CNN, the parameters of the CNN are updated in a manner that reduces the error between the output result of the learned model DM and the correct answer label, for example, by back propagation (error back propagation method). In addition, step S590 is sometimes referred to as a learning process.

[0058] Fig.14 : is an explanatory diagram schematically showing the selection screen SW. The selection screen SW includes an overlap condition selection area RS for selecting an overlap condition and a processing condition selection area RK for selecting a processing condition. The overlap condition is a condition when the vehicle information and the driving environment information are overlapped. Fig.11 In step S540, the overlapping image generation unit 250 overlaps the vehicle information and the driving environment information according to the overlapping conditions selected in the overlapping condition selection area RS. The overlapping conditions include, for example, conditions related to the position of the vehicle model MV relative to the factory model MF and conditions related to the orientation of the vehicle model MV relative to the factory model MF. The processing conditions are conditions for processing at least one of the vehicle information, the driving environment information, and the overlapping image. Fig.11 In step S530 and step S550, the processing unit 260 processes at least one of the vehicle information, the driving environment information, and the overlapping image according to the processing conditions selected in the processing condition selection area RK. The processing conditions include, for example, conditions related to the color of the vehicle 100. The processing conditions may also include conditions related to the weather of the factory FC. The processing condition selection area RK may be configured to allow selection of, for example, sunny days and rainy days. The processing unit 260 may dim the virtual light source VL when a rainy day is selected compared to when a sunny day is selected. Alternatively, the processing condition selection area RK may be configured to allow selection of the longitude and latitude, time, and date of the factory FC. The processing unit 260 may obtain meteorological information related to the position of the sun from the outside according to the conditions selected in the processing condition selection area RK, and change the position of the virtual light source VL based on the meteorological information. In this case, it is possible to suppress the reduction in the detection accuracy of the vehicle 100 based on the learned model DM due to changes in the weather. In addition, the processing condition selection area RK can also be configured to select conditions such as the size of raindrops, the interval between raindrops, the falling speed of raindrops, whether it is water or ice, etc. In this case, it is possible to suppress the reduction in the detection accuracy of the vehicle 100 based on the learned model DM when the vehicle 100 is blocked by raindrops. By selecting the falling speed of raindrops, the length of the afterimage of raindrops presented in the superimposed image can be selected.

[0059] According to the system 50 in the present embodiment described above, the overlapping image generation unit 250 overlaps the vehicle model MV and the plant model MF to generate the overlapping image GS, and the data set generation unit 280 generates the learning data set DS including the overlapping image GS. Therefore, for example, the effort of acquiring the learning image by photographing the vehicle 100 arranged in the plant FC in the real space can be eliminated. Therefore, the effort of generating the learning data set DS can be reduced. In addition, the overlapping image GS can also be used to express a phenomenon that appears less frequently in the real space.

[0060] B. Second Implementation Method: Fig.15 It is an explanatory diagram showing the configuration of a vehicle 100 in the second embodiment. Fig.16 It is an explanatory diagram showing the configuration of a server device 200 in the second embodiment. Fig.17 1 is a flowchart showing the contents of the unmanned driving process in the second embodiment. In the system 50 in this embodiment, the difference from the first embodiment is that the server device 200 does not have the remote control unit 210, and the vehicle 100 travels by autonomous control rather than by remote control. The other configurations are the same as those in the first embodiment unless otherwise specified.

[0061] like Fig.15 As shown, the reference path RR and the learned model DM are pre-stored in the memory 112 of the vehicle control device 110. The communication device 130 can communicate with the external camera 300 through wireless communication. The processor 111 of the vehicle control device 110 functions as a driving control unit 115 by executing a computer program PG1 pre-stored in the memory 112. In this embodiment, the driving control unit 115 does not control the actuator group 120 using a driving control signal obtained from outside the vehicle 100, but controls the actuator group 120 using a driving control signal generated by itself. Fig.16 As shown, the server device 200 functions as a driving route acquisition unit 220, a driving environment information acquisition unit 230, a vehicle information acquisition unit 240, a superimposed image generation unit 250, a processing unit 260, a label information acquisition unit 270, a data set generation unit 280, and a learning unit 290 by executing a computer program PG2 pre-stored in a memory 202. The learned model DM generated by the learning unit 290 can be transmitted to the vehicle 100 by wireless communication before the vehicle 100 travels and stored in the memory 112 of the vehicle control device 110, for example.

[0062] like Fig.17As shown, in the present embodiment, the unmanned driving process is performed by the driving control unit 115. In step S210, the driving control unit 115 obtains the vehicle position information using the captured image output from the external camera 300 and the learned model DM. In step S220, the driving control unit 115 determines the target position to which the vehicle 100 should go next using the vehicle position information and the reference path RR. In step S230, the driving control unit 115 generates a driving control signal for driving the vehicle 100 toward the determined target position. In step S240, the driving control unit 115 controls the actuator group 120 using the generated driving control signal, so that the vehicle 100 drives at the acceleration and steering angle indicated by the driving control signal. The driving control unit 115 repeatedly performs the acquisition of the position information of the vehicle 100, the determination of the target position, the generation of the driving control signal, and the control of the actuator group 120 at a predetermined cycle.

[0063] According to the system 50 in the present embodiment described above, the vehicle 100 can be driven by the autonomous control of the vehicle 100 even without external remote control of the vehicle 100.

[0064] C. Other implementations: (C1) In the first embodiment, the server device 200 performs the processing from the acquisition of the vehicle position information to the generation of the travel control signal. In contrast, at least a part of the processing from the acquisition of the vehicle position information to the generation of the travel control signal may be performed by the vehicle 100. For example, the following methods (1) to (3) may be used.

[0065] (1) The server device 200 can obtain vehicle position information, determine the target position to which the vehicle 100 should go next, and generate a path from the current position of the vehicle 100 indicated by the obtained vehicle position information to the target position. The server device 200 can generate a path to the target position between the current position and the destination, or can generate a path to the destination. The server device 200 can send the generated path to the vehicle 100. The vehicle 100 can generate a travel control signal in a manner such that the vehicle 100 travels on the path received from the server device 200, and use the generated travel control signal to control the actuator group 120.

[0066] (2) The server device 200 may also obtain vehicle position information and transmit the obtained vehicle position information to the vehicle 100. The vehicle 100 may also determine a target position to which the vehicle 100 should go next, generate a path from the current position of the vehicle 100 indicated by the received vehicle position information to the target position, generate a travel control signal so that the vehicle 100 travels on the generated path, and control the actuator group 120 using the generated travel control signal.

[0067] (3) In the above-mentioned methods (1) and (2), the vehicle 100 may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used in at least one of the generation of the route and the generation of the driving control signal. The internal sensor is a sensor mounted on the vehicle 100. The internal sensor may include, for example, a sensor for detecting the motion state of the vehicle 100, a sensor for detecting the motion state of each part of the vehicle 100, and a sensor for detecting the environment around the vehicle 100. Specifically, the internal sensor may include, for example, a camera, a LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyroscope sensor, etc. For example, in the above-mentioned method (1), the server device 200 may also obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the route. In the above-mentioned method (1), the vehicle 100 may also obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the driving control signal when generating the driving control signal. In the above-mentioned method (2), the vehicle 100 may also obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the route. In the above-mentioned aspect (2), the vehicle 100 may obtain the detection result of the internal sensor and reflect the detection result of the internal sensor in the travel control signal when generating the travel control signal.

[0068] (C2) In each of the above embodiments, the vehicle 100 may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used in at least one of the generation of the route and the generation of the travel control signal. For example, the vehicle 100 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the route when generating the route. The vehicle 100 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor in the travel control signal when generating the travel control signal.

[0069] (C3) In the second embodiment described above, the vehicle 100 uses the detection result of the external camera 300 to obtain the vehicle position information. In contrast, the vehicle 100 may be equipped with an internal sensor, and the vehicle 100 may use the detection result of the internal sensor to obtain the vehicle position information, determine the target position to which the vehicle 100 should go next, generate a path from the current position of the vehicle 100 indicated by the obtained vehicle position information to the target position, generate a travel control signal for traveling on the generated path, and control the actuator group 120 using the generated travel control signal. In this case, the vehicle 100 can travel without using the detection result of the external camera 300 at all. In addition, the vehicle 100 may also obtain the target arrival time and / or congestion information from the outside of the vehicle 100, so that the target arrival time and / or congestion information is reflected in at least one of the path and the travel control signal. In addition, the functional configuration of the system 50 may also be entirely provided on the vehicle 100. That is, the processing implemented by the system 50 in the present disclosure may also be implemented by the vehicle 100 alone.

[0070] (C4) In the first embodiment described above, the server device 200 automatically generates a travel control signal to be transmitted to the vehicle 100. In contrast, the server device 200 may generate a travel control signal to be transmitted to the vehicle 100 in accordance with an operation of an external operator located outside the vehicle 100. For example, the external operator may operate an operating device including a display for displaying a captured image output from the external camera 300, a steering wheel for remotely operating the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server device 200 via wired communication or wireless communication, and the server device 200 may generate a travel control signal corresponding to the operation applied to the operating device.

[0071] (C5) In each of the above embodiments, the vehicle 100 only needs to have a structure capable of moving by unmanned driving, for example, it may be a form having a stand having the structure described below. Specifically, in order for the vehicle 100 to perform the three functions of "driving", "turning" and "stopping" by unmanned driving, it is sufficient to have at least a vehicle control device 110 and an actuator group 120. In the case where the vehicle 100 obtains information from the outside for unmanned driving, the vehicle 100 may also have a communication device 130. That is, on the vehicle 100 capable of moving by unmanned driving, at least a part of the interior parts such as the driver's seat and the instrument panel may not be installed, at least a part of the exterior parts such as the bumper and the fender may not be installed, and the body shell may not be installed. In this case, the remaining parts such as the body shell may be installed on the vehicle 100 before the vehicle 100 is shipped from the factory FC, or the remaining parts such as the body shell may be installed on the vehicle 100 after the vehicle 100 is shipped from the factory FC in a state where the remaining parts such as the body shell are not installed on the vehicle 100. Each component can be installed from any direction such as the upper side, lower side, front side, rear side, right side or left side of the vehicle 100, and can be installed from the same direction or from different directions. In addition, the position of the stand can also be determined in the same way as the vehicle 100 in the first embodiment.

[0072] (C6) The vehicle 100 can be manufactured by combining a plurality of modules. A module refers to a unit composed of a plurality of components that are assembled according to the location and function of the vehicle 100. For example, the rack of the vehicle 100 can be manufactured by combining a front module that constitutes the front part of the rack, a central module that constitutes the central part of the rack, and a rear module that constitutes the rear part of the rack. In addition, the number of modules that constitute the rack is not limited to 3, and may be 2 or less or 4 or more. In addition, in addition to the components that constitute the rack, components that constitute a portion of the vehicle 100 that is different from the rack may also be modularized, or components that constitute a portion of the vehicle 100 that is different from the rack may be modularized instead of the components that constitute the rack. In addition, various modules may also include any exterior components such as a bumper or grille, and any interior components such as a seat or console. In addition, not limited to the vehicle 100, any type of mobile body may also be manufactured by combining a plurality of modules. Such a module may be manufactured, for example, by joining a plurality of components by welding or a fixing tool, or may be manufactured by integrally molding at least a portion of the components that constitute the module into one component by casting. The molding method of integrally molding a component, especially a larger component, is also called one-piece casting (Giga-casting or Mega-casting). For example, the front module, the central module, and the rear module mentioned above can also be manufactured using Giga-casting.

[0073] (C7) The transportation of the vehicle 100 by the driving of the unmanned vehicle 100 is also referred to as "self-propelled transportation". In addition, the configuration for realizing self-propelled transportation is also referred to as "vehicle remote control autonomous driving transportation system". In addition, the production method of producing the vehicle 100 by self-propelled transportation is also referred to as "self-propelled production". In self-propelled production, for example, in the factory FC that manufactures the vehicle 100, at least a part of the transportation of the vehicle 100 is realized by self-propelled transportation.

[0074] (C8) In the above-mentioned embodiments, part or all of the functions and processes implemented in software may be implemented in hardware. In addition, part or all of the functions and processes implemented in hardware may be implemented in software. As hardware for implementing the various functions in the above-mentioned embodiments, various circuits such as integrated circuits or discrete circuits may be used.

[0075] (C9) Fig.18 This is a first explanatory diagram schematically showing a learned model DM in another embodiment. Fig.19 2 is a second explanatory diagram schematically showing a learned model DM in another embodiment. In the above embodiments, the learned model DM is configured to output a mask image when a captured image is input. In contrast, the learned model DM may be configured to output coordinate points related to the outline of the vehicle 100 when a captured image is input. For example, Fig.18 As shown, it is configured to output four coordinate points. Fig.19 As shown, eight coordinate points are output. In these cases, the label information includes the position coordinates of the coordinate points. The remote control unit 210 and the travel control unit 115 can obtain the vehicle position information by using the coordinate points output from the learned model DM, for example.

[0076] (C10) In each of the above embodiments, the processor 201 may determine a specific position among a plurality of positions on the driving path of the vehicle 100 where the detection accuracy of the vehicle 100 based on the learned model DM is relatively low, and the number of learning data sets DS included in the learning data set group DG corresponding to the specific position is larger than that of other positions. For example, in a darker place, a place with many obstacles, or a place where the position and orientation of the vehicle 100 are easily changed due to the influence of interference, the detection accuracy of the vehicle 100 based on the learned model DM is likely to be low. Therefore, when the processor 201 generates the learned model DM for acquiring the vehicle position information using the captured images of the external camera 300 disposed in a darker place, a place with many obstacles, or a place where the position and orientation of the vehicle 100 are easily changed due to the influence of interference, the number of learning data sets DS included in the learning data set group DG is increased compared to when the learned model DM for acquiring the vehicle position information is generated using the captured images of the external camera 300 disposed in a brighter place, a place with few obstacles, or a place where the position and orientation of the vehicle 100 are not easily changed due to the influence of interference. For example, the number of learning data sets DS included in the learning data set group DG may be increased in such a manner that the types of brightness, the types of positions of the vehicle 100, and the types of orientations of the vehicle 100 are increased. In this case, the detection accuracy of the vehicle 100 can be improved in dark places, places with many obstacles, and places where the position and orientation of the vehicle 100 are easily changed due to the influence of interference.

[0077] (C11) In each of the above-mentioned embodiments, the processor 201 may determine a specific position among a plurality of positions on the driving path of the vehicle 100 where the detection accuracy of the vehicle 100 based on the learned model DM is relatively low, and generate a learning data set DS corresponding to the environment of the specific position. For example, in a darker place, a place with many obstacles, or a place where the position and orientation of the vehicle 100 are easily changed due to the influence of interference, the detection accuracy of the vehicle 100 based on the learned model DM is likely to be low. Therefore, when the processor 201 generates the learned model DM for obtaining the vehicle position information using the captured image of the external camera 300 disposed in a darker place, the brightness in the overlapped image GS may be lowered compared to when the learned model DM for obtaining the vehicle position information is generated using the captured image of the external camera 300 disposed in a brighter place. For places where the position and orientation of the vehicle 100 are easily changed, the influence of interference may be considered in the position and orientation of the vehicle 100. In this case, it is possible to improve the detection accuracy of the vehicle 100 in a dark place, a place with many obstacles, or a place where the position and direction of the vehicle 100 are likely to change due to the influence of interference.

[0078] (C12) In the first embodiment, the remote control unit 210 may be provided on a computer different from the server device 200. In this case, a communication device for communicating with the vehicle 100 by wireless communication is provided on the computer having the remote control unit 210, and the computer having the remote control unit 210 obtains the learned model DM generated on the server device 200, and obtains the position and orientation of the vehicle 100 using the learned model DM.

[0079] (C13) In each of the above-mentioned embodiments, the driving environment information acquisition unit 230 acquires the factory model MF as driving environment information, the vehicle information acquisition unit 240 acquires the vehicle model MV as vehicle information, and the superimposed image generation unit 250 superimposes the vehicle model MV and the factory model MF to generate a superimposed image. In contrast, the driving environment information acquisition unit 230 may acquire an image representing the appearance of the factory FC including the driving route as driving environment information. The image of the factory FC may be a photographed image generated by photographing the appearance of the factory FC including the driving route in real space, or an image generated by rendering using the factory model MF. The vehicle information acquisition unit 240 may acquire an image representing the appearance of the vehicle 100 as vehicle information. The image of the vehicle 100 may be a photographed image generated by photographing the appearance of the vehicle 100 in real space, or an image generated by rendering using the vehicle model MV. The superimposed image generation unit 250 may generate a superimposed image by superimposing the image of the vehicle 100 and the image of the factory FC, in other words, synthesizing the image of the vehicle 100 and the image of the factory FC. The processing unit 260 may perform a predetermined processing on the image of the vehicle 100 and the image of the factory FC before superimposing the image of the vehicle 100 and the image of the factory FC to generate a superimposed image. The processing unit 260 may perform a predetermined processing on the superimposed image after superimposing the image of the vehicle 100 and the image of the factory FC to generate a superimposed image.

[0080] The present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various configurations within the scope of its main purpose. For example, the technical features in the embodiments corresponding to the technical features in each method described in the invention content section can be appropriately replaced or combined in order to solve part or all of the above-mentioned problems, or to achieve part or all of the above-mentioned effects. In addition, as long as the technical feature is not described as a necessary part in this specification, it can be appropriately deleted. Description of Reference Numerals

[0081] 50…system, 100…vehicle, 110…vehicle control device, 111…processor, 112…memory, 113…input / output interface, 114…internal bus, 115…driving control unit, 120…actuator group, 130…communication device, 200…server device, 201…processor, 202…memory, 203…input / output interface, 204…internal bus, 205…communication device, 206…input device, 207…display device, 210…remote control unit, 220…driving path acquisition unit, 230…driving environment information acquisition unit, 240…vehicle information acquisition unit, 250…overlapping image generation unit, 260…processing unit, 270…label information acquisition unit, 280…data set generation unit, 290…learning unit, 300…external camera.

Claims

1. A method comprising: A moving path obtaining step of obtaining a moving path of the moving body; a superimposed image generating step of generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path; a tag information acquisition step of acquiring tag information related to the position and orientation of the moving object; and The data set generating step generates a learning data set including the overlapped image and the label information.

2. The method according to claim 1, wherein: Prior to the superimposed image generating step, there is further included a step of performing a predetermined processing on the image or three-dimensional data of the moving environment.

3. The method according to claim 1, wherein: Prior to the step of generating the superimposed image, there is further included a step of performing a predetermined process on the image or three-dimensional data of the moving object.

4. The method according to claim 1, wherein: The method further includes performing a predetermined process on the superimposed image.

5. The method according to claim 4, wherein: The processing includes at least one of adding or changing the light irradiated to the moving object and adding or changing the shadow of the moving object.

6. The method according to claim 4, wherein: The processing includes changing at least one of brightness, chroma, and contrast.

7. The method according to claim 1, wherein: The overlapping image generation process is performed multiple times. In at least two of the plurality of superimposed images generated by executing the superimposed image generating step a plurality of times, the position of the moving object in the direction parallel to the moving path is made different.

8. The method according to claim 1, wherein: The overlapping image generation process is performed multiple times. In at least two of the plurality of superimposed images generated by executing the superimposed image generating step a plurality of times, the position of the moving object in a direction perpendicular to the moving path is made different.

9. The method according to claim 1, wherein: The overlapping image generation process is performed multiple times. In at least two of the plurality of superimposed images generated by executing the superimposed image generating step a plurality of times, the orientation of the moving object with respect to the moving path is made different.

10. The method according to claim 1, wherein: The overlapping image generation process is performed multiple times. In at least two of the plurality of superimposed images generated by executing the superimposed image generating step a plurality of times, the color of the moving object is made different.

11. The method according to claim 1, wherein: In the tag information acquisition step, the tag information is acquired based on information associated with the image or three-dimensional data of the moving object.

12. The method according to claim 1, wherein: The method further includes a learning step of executing machine learning using a learning data set group including the learning data set.

13. The method according to claim 12, wherein: The method further includes a step of acquiring information on the position and orientation of the moving object moving in the moving environment using the learned model generated in the learning step and a captured image of the moving object moving in the moving environment.

14. The method according to claim 1, wherein: The overlapping image generation process is performed multiple times. The method further comprises a learning step of executing machine learning using a learning data set group including a plurality of the learning data sets. determining a specific position among a plurality of positions on the movement path at which the output accuracy of the learned model generated by the learning step is relatively low, The number of the training data sets corresponding to the specific position is made larger than that corresponding to other positions.

15. A device comprising: a moving path acquisition unit that acquires a moving path of the moving object; a superimposed image generating unit for generating a superimposed image by superimposing an image or three-dimensional data representing the moving object and an image or three-dimensional data representing a moving environment including the moving path; a tag information acquisition unit that acquires tag information related to the position and orientation of the moving object; as well as A data set generating unit generates a learning data set including the overlapped image and the label information.

Images