Machine learning system

Abstract

To provide a technology that can properly train a machine learning model used to move a moving object in an unmanned operation.SOLUTION: A machine learning system includes: a range finder that acquires three-dimensional point group information that represents a moving object that can move in an unmanned operation; an external camera that photographs an image including the moving object; a calculation unit that calculates a location and orientation of the moving object using the three-dimensional point group information; and a learning unit that trains a machine learning model that outputs a signal generation parameter when the photographed image is input, the learning unit training the machine learning model using one or more learning data sets that associate the photographed image with a correct label generated using the position and the orientation calculated by the calculation unit. The signal generation parameter is used when generating a control signal that regulates an operation of the moving object to move the moving object in an unmanned operation.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to machine learning systems. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a technique is known in which a vehicle is made to travel automatically by remote control by monitoring the travel of the vehicle using a camera outside the vehicle (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to move a moving object such as a vehicle in an unmanned driving manner, an image including the moving object captured by a camera may be input to a machine learning model to obtain parameters for generating a control signal that defines the operation of the moving object. In this case, a method for appropriately training the machine learning model is desired. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, a machine learning system is provided. The machine learning system includes a distance measuring device that detects a moving body that can move by unmanned driving from the outside and obtains three-dimensional point cloud information representing the moving body, an external camera that captures an image including the moving body by capturing an image of the moving body from the outside, a calculation unit that calculates a position and orientation of the moving body using the three-dimensional point cloud information, and a learning unit that trains a machine learning model that outputs a signal generation parameter when the captured image is input, the learning unit training the machine learning model using one or more training data sets that associate the captured image with a correct answer label generated using the position and orientation calculated by the calculation unit, and the signal generation parameter is a parameter used when generating a control signal that specifies the operation of the moving body to move the moving body by unmanned driving. According to this aspect, the machine learning system can train the machine learning model using one or more training data sets that associate a correct answer label generated using the position and orientation of the vehicle calculated from the three-dimensional point cloud information with the captured image. In this way, the machine learning model can be trained without the need for a person to manually assign a correct answer label to the captured image. This reduces the time and labor required for preparation to train the machine learning model. Therefore, the machine learning model can be trained appropriately. (2) In the above embodiment, the position and the orientation calculated by the calculation unit may be used in a signal generation process for generating the control signal and a label generation process for generating the correct label. According to this embodiment, the machine learning system can use the position and orientation of the vehicle calculated using the three-dimensional point cloud information in the signal generation process and the label generation process. In this way, during the period when the unmanned driving control is being executed, a correct label can be generated, a learning dataset can be created, and a machine learning model can be learned. (3) In the above embodiment, the signal generation parameters may be at least one of rectangular coordinates and three-dimensional coordinates, the rectangular coordinates being coordinates of four vertices of a rectangle that is set in the captured image so as to surround an area occupied by the moving object when the moving object is projected onto a road surface on which the moving object moves, and the three-dimensional coordinates being coordinates of eight vertices of a rectangular parallelepiped that is set in the captured image so as to surround the moving object. According to this embodiment, the machine learning system can train a machine learning model that outputs at least one of rectangular coordinates and three-dimensional coordinates as signal generation parameters when a captured image is input. (4) In the above embodiment, the training dataset may further include supplementary information associated with the captured image and the correct label, the supplementary information being related to factors that affect the output result of the machine learning model. According to this embodiment, the machine learning system can train the machine learning model using one or more training datasets in which supplementary information related to factors that affect the output result of the machine learning model is associated with the captured image and the correct label. This can improve the accuracy of the signal generation parameters output from the machine learning model. (5) In the above embodiment, the learning unit may execute at least one of an individual learning process for training a plurality of the machine learning models for each type of the moving object, and a simultaneous learning process for training one of the machine learning models using the training data set in which the captured image and the correct answer label are associated with supplementary information related to the type of the moving object. According to this embodiment, the machine learning system can train a plurality of machine learning models for each type of vehicle by executing the individual learning process. In addition, the machine learning system can train a machine learning model using one or more training data sets in which supplementary information related to the type of vehicle is associated with the captured image and the correct answer label by executing the simultaneous learning process. In this way, the accuracy of the signal generation parameters output from the machine learning model can be improved. The present disclosure can be realized in various forms other than the above machine learning system, for example, in the form of a distance measuring device, an external camera, a machine learning device, a moving body, a manufacturing method thereof, a control method thereof, a computer program for realizing the control method, a non-transitory recording medium on which the computer program is recorded, etc., which are included in the machine learning system. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a configuration of a traveling system in a first embodiment. [Diagram 2] FIG. 13 is a diagram for explaining rectangular coordinates. [Diagram 3] 4 is a flowchart showing a vehicle travel control procedure in the first embodiment. [Figure 4] 5 is a flowchart showing the processing content of a learning period in the first embodiment. [Diagram 5] 10 is a flowchart showing the processing content of a learned period in the first embodiment. [Figure 6] FIG. 11 is a block diagram showing the configuration of a traveling system in a second embodiment. [Figure 7] FIG. 2 is a diagram for explaining three-dimensional coordinates. [Figure 8] 13 is a flowchart showing the processing content of a learning period in the second embodiment. [Figure 9] 13 is a first flowchart showing the processing content of a learned period in the second embodiment. [Figure 10] 13 is a second flowchart showing the processing content in a learned period in the second embodiment. [Figure 11] FIG. 11 is a block diagram showing the configuration of a traveling system according to a third embodiment. [Figure 12] 13 is a flowchart showing a vehicle travel control procedure in a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: 1 is a block diagram showing the configuration of a traveling system 50 in the first embodiment. The traveling system 50 is a system for moving a moving body without the need for a driving operation by a passenger on the moving body. The traveling system 50 includes one or more vehicles 100 as moving bodies, a machine learning system 7, and a remote control device 80.

[0009] The machine learning system 7 is a system for learning a first learning model DM1 as a machine learning model DM used to move the vehicle 100 by unmanned driving. The machine learning model DM outputs a signal generation parameter when a captured image including the vehicle 100 is input. The signal generation parameter is a parameter used when generating a driving control signal. The driving control signal is a control signal that specifies the operation of the vehicle 100 to move the vehicle 100 by unmanned driving. The machine learning system 7 includes an external LiDAR 310 and an external camera 320 as the external sensor 300, and a machine learning device 70. In this embodiment, the function of the machine learning device 70 and the function of the remote control device 80 are realized by the server 200.

[0010] In the present disclosure, the term "mobile body" refers to an object that can move, such as a vehicle 100 or an electric vertical take-off and landing aircraft (a so-called flying car). The vehicle 100 may be a vehicle 100 that runs on wheels or a vehicle 100 that runs on caterpillar tracks, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, or a construction vehicle. The vehicle 100 includes an electric vehicle (BEV: Battery Electric Vehicle), a gasoline-powered vehicle, a hybrid vehicle, and a fuel cell vehicle. When the mobile body is other than the vehicle 100, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "running" can be appropriately replaced with "movement".

[0011] The vehicle 100 is configured to be capable of traveling by unmanned driving. "Unmanned driving" means driving without the driving operation of a passenger. Driving operation means at least one of the operations of "running", "turning" and "stopping" of the vehicle 100. Unmanned driving is realized by automatic or manual remote control using a device located outside the vehicle 100, or by autonomous control of the vehicle 100. The vehicle 100 traveling by unmanned driving may have a passenger who does not perform driving operation on board. The passenger who does not perform driving operation includes, for example, a person who simply sits in the seat of the vehicle 100, and a person who performs work other than driving operation, such as assembly, inspection, and operation of switches, while riding on the vehicle 100. Note that driving by a passenger who performs driving operation is sometimes called "manned driving".

[0012] In this specification, "remote control" includes "full remote control" in which all of the operations of the vehicle 100 are completely determined from outside the vehicle 100, and "partial remote control" in which some of the operations of the vehicle 100 are determined from outside the vehicle 100. In addition, "autonomous control" includes "full autonomous control" in which the vehicle 100 autonomously controls its own operations without receiving any information from devices external to the vehicle 100, and "partial autonomous control" in which the vehicle 100 autonomously controls its own operations using information received from devices external to the vehicle 100.

[0013] In this embodiment, the traveling system 50 is used in a factory that manufactures the vehicle 100. The reference coordinate system of the factory is a global coordinate system. That is, any position in the factory is expressed by X, Y, and Z coordinates in the global coordinate system. The factory has a first location and a second location. The first location and the second location are connected by a track on which the vehicle 100 can travel. In the factory, a plurality of external sensors 300 are installed along the track. The position of each external sensor 300 in the factory is adjusted in advance. The vehicle 100 moves from the first location to the second location through the track by unmanned driving.

[0014] The vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 including one or more actuators that are driven under the control of the vehicle control device 110, and a communication device 130 for communicating via wireless communication with an external device such as a server 200. The actuator group 120 includes an actuator of a drive device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100.

[0015] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected via the internal bus 114 to enable bidirectional communication. The input / output interface 113 is connected to an actuator group 120 and a communication device 130. The processor 111 executes a program PG1 stored in the memory 112 to realize various functions including a function as a vehicle control unit 115.

[0016] The vehicle control unit 115 controls the actuator group 120 to drive the vehicle 100. The vehicle control unit 115 controls the actuator group 120 using a driving control signal received from the server 200 to drive the vehicle 100. The driving control signal is a control signal for driving the vehicle 100. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0017] The external sensor 300 is a sensor located outside the vehicle 100. In this embodiment, the external sensor 300 is a sensor that captures the vehicle 100 from outside the vehicle 100. The external sensor 300 includes a communication device (not shown) and can communicate with other devices such as the server 200 by wired communication or wireless communication.

[0018] Specifically, the external sensor 300 is composed of an external LiDAR 310 and an external camera 320. The external LiDAR 310 (Light Detection And Ranging) as the external sensor 300 detects the vehicle 100 from the outside, acquires three-dimensional point cloud information representing the vehicle 100, and outputs the three-dimensional point cloud information as a detection result. The LiDAR is an example of a distance measuring device. In other embodiments, the distance measuring device may be another sensor such as a stereo camera. The external camera 320 as the external sensor 300 acquires a captured image including the vehicle 100 by capturing an image of the vehicle 100 from the outside, and outputs the captured image as a detection result.

[0019] The server 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected to each other via the internal bus 204 so as to be able to communicate in both directions. The input / output interface 203 is connected to a communication device 205 for communicating with various devices outside the server 200. The communication device 205 can communicate with the vehicle 100 by wireless communication, and can communicate with each external sensor 300 by wired communication or wireless communication. The processor 201 executes a program PG2 stored in the memory 202 to realize various functions including functions as a first calculation unit 211, a label generation unit 212, a data set creation unit 213, a learning unit 214, a second calculation unit 215, and a remote control unit 216. The server 200 may include, as a functional unit, a point cloud acquisition unit that acquires three-dimensional point cloud information representing the vehicle 100 acquired by detecting the vehicle 100 from the outside using the external LiDAR 310. The server 200 may also include an image acquisition unit that acquires an image acquired by capturing an image of the vehicle 100 from the outside using the external camera 320.

[0020] The first calculation unit 211 acquires vehicle position information by calculating the position and orientation of the vehicle 100 using the three-dimensional point cloud information output from the external LiDAR 310. In this embodiment, during a learning period from when the external LiDAR 310 starts detecting the vehicle 100 to when learning of the first learning model DM1 is completed, the first calculation unit 211 calculates the position and orientation of the vehicle 100 using the three-dimensional point cloud information. The vehicle position information is position information that is the basis for generating a driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system of the factory. In this embodiment, the position of the vehicle 100 is represented by rectangular coordinates. In addition, in this embodiment, the orientation of the vehicle 100 is represented by the direction of a vector that faces from the rear side to the front side of the vehicle 100 along the longitudinal axis of the vehicle 100.

[0021] 2 is a diagram for explaining rectangular coordinates CR1 to CR4. The rectangular coordinates CR1 to CR4 are the coordinates of four vertices VC1 to VC4 of a vehicle circumscribing rectangle RC. The vehicle circumscribing rectangle RC is a quadrangle that is set so as to surround the area occupied by the vehicle 100 when the vehicle 100 is projected onto the road surface RS of the track TR along which the vehicle 100 travels, based on the detection results output from the external sensor 300. The vehicle circumscribing rectangle RC may be determined taking into consideration the wheelbase of the vehicle 100.

[0022] The first calculation unit 211 first calculates the coordinates of the positioning point of the vehicle 100 in the LiDAR coordinate system by template matching using the three-dimensional point cloud information output from the external LiDAR 310 and the reference point cloud information prepared in advance. The reference point cloud information is, for example, information that virtually reproduces the external shape of the vehicle 100 by a point cloud. The reference point cloud information is, for example, three-dimensional CAD data of the vehicle 100. For example, an algorithm such as ICP (Iterative Closest Point) or NDT (Normal Distribution Transform) is used for matching the three-dimensional point cloud information and the reference point cloud information. The LiDAR coordinate system is a local coordinate system with the focal point of the external LiDAR 310 as the origin. The positioning point is a detection point by the external sensor 300 that is set in advance for a specific part of the vehicle 100. Next, the first calculation unit 211 converts the coordinates of the positioning point of the vehicle 100 in the LiDAR coordinate system into coordinates in the global coordinate system using the LiDAR parameters RP stored in the memory 202 of the server 200. The LiDAR parameters RP are parameters related to the external LiDAR 310. The LiDAR parameters RP are, for example, the installation position and installation attitude of the external LiDAR 310. Next, the first calculation unit 211 calculates rectangular coordinates CR1 to CR4 from the coordinates of the positioning point of the vehicle 100 using the rectangle database DC stored in the memory 202 of the server 200. The rectangle database DC is a database that indicates the relative positional relationship between each vertex VC1 to VC4 of the vehicle circumscribing rectangle RC and the positioning point. In this way, the first calculation unit 211 calculates the rectangular coordinates CR1 to CR4 in the global coordinate system as the position of the vehicle 100.

[0023] Further, the first calculation unit 211 calculates the orientation of the vehicle 100 using the rectangular coordinates CR1 to CR4. Specifically, the first calculation unit 211 calculates the orientation of the vehicle 100 using the coordinates of the first central position CN11 and the coordinates of the second central position CN21. The first central position CN11 is the central position of a first side SC1 that is aligned along the vehicle width direction on the front side of the vehicle 100 among the four sides SC1 to SC4 of the vehicle circumscribing rectangle RC. The second central position CN21 is the central position of a second side SC2 that is aligned along the vehicle width direction on the rear side of the vehicle 100 among the four sides SC1 to SC4 of the vehicle circumscribing rectangle RC.

[0024] The first calculation unit 211 may calculate the position and orientation of the vehicle 100 by a method different from the above. For example, the first calculation unit 211 may calculate, as the orientation of the vehicle 100, a direction from a third rectangular coordinate CR3 located on the rear side of the vehicle 100 to a first rectangular coordinate CR1 located on the front side of the vehicle 100, among the four vertices VC1 to VC4 of the vehicle circumscribing rectangle RC.

[0025] The label generating unit 212 shown in FIG. 1 generates a correct label according to the output result of the first learning model DM1 using the position and orientation of the vehicle 100 calculated by the first calculating unit 211. That is, the correct label is generated using the position and orientation of the vehicle 100 calculated from the three-dimensional point cloud information. In this embodiment, the first learning model DM1 outputs rectangular coordinates CR1 to CR4 in the image coordinate system as signal generation parameters. The image coordinate system is a coordinate system with a point on the image plane as the origin. Therefore, the label generating unit 212 generates, as a correct label, a rectangular correct label that represents, in the image coordinate system, the coordinates of four vertices VC1 to VC4 of the vehicle circumscribing rectangle RC set for the vehicle 100 included in the captured image as the training image acquired by the external camera 320. Specifically, the label generation unit 212 converts the rectangular coordinates CR1 to CR4 in the global coordinate system, which are the position of the vehicle 100 calculated by the first calculation unit 211, into the image coordinate system using the orientation of the vehicle 100 calculated by the first calculation unit 211 and the camera parameter CP. In this manner, the label generation unit 212 generates a rectangular correct answer label indicating the rectangular coordinates CR1 to CR4 in the image coordinate system. In the rectangular correct answer label, each coordinate is associated with additional information indicating which of the four vertices VC1 to VC4 of the vehicle circumscribing rectangle RC the coordinate corresponds to.

[0026] The label generating unit 212 may generate the rectangular correct label by a method different from the above. The label generating unit 212 may generate the rectangular correct label by, for example, executing the following process. Specifically, the label generating unit 212 first converts the coordinates of the positioning point of the vehicle 100 in the LiDAR coordinate system calculated using the three-dimensional point cloud information into coordinates in the camera coordinate system based on the relative positional relationship between the external LiDAR 310 and the external camera 320. The camera coordinate system is a local coordinate system with the focus of the external camera 320 as the origin. Next, the label generating unit 212 executes a dimension conversion process for converting three-dimensional data into two-dimensional data, and a coordinate system conversion process for converting from the camera coordinate system to the image coordinate system by perspective conversion or the like. As a result, the label generating unit 212 converts the coordinates of the positioning point of the vehicle 100 in the camera coordinate system into coordinates in the image coordinate system. Next, the label generation unit 212 calculates rectangular coordinates CR1 to CR4 from the coordinates of the positioning point of the vehicle 100, using the rectangular database DC stored in the memory 202 of the server 200. In this manner, the label generation unit 212 may generate rectangular correct answer labels indicating the rectangular coordinates CR1 to CR4 in the image coordinate system.

[0027] The data set creation unit 213 creates a first training data set by associating a rectangle correct label with a captured image as a training image. In this embodiment, the data set creation unit 213 creates a first training data set for each external camera 320.

[0028] The learning unit 214 trains a first learning model DM1 utilizing artificial intelligence through supervised learning using one or more first learning data sets. For the first learning model DM1, for example, a convolutional neural network (hereinafter, CNN) that realizes either semantic segmentation or instance segmentation can be used. During training of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce an error between an output result by the first learning model DM1 and a rectangle ground truth label. In this embodiment, the learning unit 214 trains the first learning model DM1 for each external camera 320.

[0029] The second calculation unit 215 acquires vehicle position information by calculating the position and orientation of the vehicle 100 using the captured image output from the external camera 320. In this embodiment, after the learning of the first learning model DM1 is completed, the second calculation unit 215 calculates the position and orientation of the vehicle 100 using the captured image.

[0030] The second calculation unit 215 first inputs a captured image to the trained first learning model DM1 to obtain rectangular coordinates CR1 to CR4 in the image coordinate system as signal generation parameters. Next, the second calculation unit 215 converts the rectangular coordinates CR1 to CR4 in the image coordinate system into coordinates in the global coordinate system using the camera parameters CP stored in the memory 202 of the server 200. In this manner, the second calculation unit 215 calculates the rectangular coordinates CR1 to CR4 in the global coordinate system as the position of the vehicle 100. Similarly to the first calculation unit 211, the second calculation unit 215 calculates the orientation of the vehicle 100 using the rectangular coordinates CR1 to CR4.

[0031] The second calculation unit 215 may calculate the position and orientation of the vehicle 100 by a method different from the above. For example, the second calculation unit 215 may use an optical flow method to calculate the orientation of the vehicle 100 based on the orientation of a movement vector of the vehicle 100 calculated from the position side of a feature point of the vehicle 100 between frames of captured images.

[0032] The remote control unit 216 generates a driving control signal for controlling the actuator group 120 of the vehicle 100, and transmits the driving control signal to the vehicle 100, thereby causing the vehicle 100 to drive by remote control.

[0033] The remote control unit 216 first determines a target position to which the vehicle 100 should next head. In this embodiment, the target position is represented by X, Y, and Z coordinates in a global coordinate system. A reference route RR, which is a route along which the vehicle 100 should travel, is stored in advance in the memory 202 of the server 200. The route is represented by nodes indicating a departure point, nodes indicating passing points, nodes indicating a destination, and links connecting the nodes. The remote control unit 216 uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. The remote control unit 216 determines a target position on the reference route RR that is ahead of the current location of the vehicle 100.

[0034] The remote control unit 216 generates a travel control signal for driving the vehicle 100 toward the determined target position. The remote control unit 216 calculates the travel speed of the vehicle 100 from the transition of the position of the vehicle 100, and compares the calculated travel speed with the target speed. When the travel speed is lower than the target speed as a whole, the remote control unit 216 determines the acceleration so that the vehicle 100 accelerates, and when the travel speed is higher than the target speed, the remote control unit 216 determines the acceleration so that the vehicle 100 decelerates. In addition, when the vehicle 100 is located on the reference route RR, the remote control unit 216 determines the steering angle and acceleration so that the vehicle 100 does not deviate from the reference route RR, and when the vehicle 100 is not located on the reference route RR, in other words, when the vehicle 100 deviates from the reference route RR, the remote control unit 216 determines the steering angle and acceleration so that the vehicle 100 returns to the reference route RR.

[0035] The remote control unit 216 transmits the generated driving control signal to the vehicle 100. The processor 201 of the server 200 repeats, at a predetermined cycle, obtaining the position of the vehicle 100, determining a target position, generating the driving control signal, and transmitting the driving control signal.

[0036] Fig. 3 is a flowchart showing a processing procedure for driving control of the vehicle 100 in the first embodiment. The flow shown in Fig. 3 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts to drive in an unmanned manner.

[0037] In step 101, the calculation units 211, 215 of the server 200 acquire vehicle position information of the vehicle 100 using the detection results output from the external sensor 300. In step 102, the remote control unit 216 uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. In step 103, the remote control unit 216 generates a driving control signal for driving the vehicle 100 toward the determined target position. In step 104, the remote control unit 216 transmits the generated driving control signal to the vehicle 100.

[0038] In step 105, the vehicle control device 110 receives a driving control signal transmitted from the server 200. In step 106, the vehicle control unit 115 of the vehicle control device 110 controls the actuator group 120 using the received driving control signal, thereby causing the vehicle 100 to drive at the acceleration and steering angle represented by the driving control signal. The processor 111 of the vehicle control device 110 repeats receiving the driving control signal and controlling the actuator group 120 at a predetermined cycle. According to the driving system 50 in this embodiment, the vehicle 100 can be driven by remote control, and the vehicle 100 can be moved without using transportation equipment such as a crane or a conveyor.

[0039] 4 is a flowchart showing the processing contents during the learning period in the first embodiment. In step 201, the external LiDAR 310 acquires three-dimensional point cloud information. In step 202, the external LiDAR 310 transmits three-dimensional point cloud information in which the vehicle identifier and the point cloud identifier are associated to the server 200. The vehicle identifier is a unique identifier assigned to each vehicle 100 so as to identify multiple vehicles 100. The point cloud identifier is a unique identifier assigned to each three-dimensional point cloud information so as to identify multiple three-dimensional point cloud information.

[0040] In step 203, the external camera 320 acquires a captured image. In step 204, the external camera 320 transmits the captured image associated with the vehicle identifier and the image identifier to the server 200. The image identifier is a unique identifier assigned to each captured image in order to identify a plurality of captured images.

[0041] In step 205, the first calculation unit 211 of the server 200 acquires vehicle position information using the three-dimensional point cloud information. After step 205, a label generation process shown in step 206 and a signal generation process shown in steps 210 and 211 are respectively executed.

[0042] After step 205, in step 206, the label generation unit 212 of the server 200 generates a rectangular correct answer label using the vehicle position information acquired using the three-dimensional point cloud information. In step 207, the dataset creation unit 213 associates the rectangular correct answer label with the captured image corresponding to the acquisition timing of the three-dimensional point cloud information used to generate the rectangular correct answer label, using the vehicle identifier, the point cloud identifier, and the image identifier. In this way, the dataset creation unit 213 creates a first learning dataset in which the correct answer label generated using the position and orientation of the vehicle 100 calculated by the first calculation unit 211 is associated with the captured image corresponding to the acquisition timing of the three-dimensional point cloud information used to generate the correct answer label. Each step from step 201 to step 207 is repeatedly executed until N first learning datasets are created. If step 208 is "Yes," that is, if N first learning datasets are created, in step 209, the learning unit 214 trains the first learning model DM1 using the N first learning datasets. Note that the learning unit 214 may train the first learning model DM1 every time one first learning data set is created.

[0043] After step 205, in step 210, the remote control unit 216 determines a target position to which the vehicle 100 should next head, using the vehicle position information acquired using the three-dimensional point cloud information and the reference route RR. In step 211, the remote control unit 216 generates a driving control signal for driving the vehicle 100 toward the determined target position. In step 212, the remote control unit 216 transmits the generated driving control signal to the vehicle 100.

[0044] When the vehicle control device 110 mounted on the vehicle 100 receives a driving control signal transmitted from the server 200, the vehicle control unit 115 of the vehicle control device 110 executes the following process in step 213. In this case, the vehicle control unit 115 controls the actuator group 120 using the received driving control signal to cause the vehicle 100 to drive at the acceleration and steering angle represented in the driving control signal. During the learning period, steps 201, 202, 205, and steps 210 to 214 are repeatedly executed at predetermined time intervals.

[0045] Fig. 5 is a flowchart showing the processing contents of the learned period in the first embodiment. The flow shown in Fig. 5 is repeatedly executed at predetermined time intervals during the learned period, which is the period from the time when learning of the first learning model DM1 is completed to the time when unmanned driving ends.

[0046] In step 301, the external camera 320 acquires a captured image. In step 302, the external camera 320 transmits to the server 200 the captured image in which the vehicle identifier and the image identifier are associated.

[0047] In step 303, the second calculation unit 215 of the server 200 inputs the captured image into the trained first learning model DM1 to obtain rectangular coordinates CR1 to CR4 in the image coordinate system. In step 304, the second calculation unit 215 performs various processes such as converting the rectangular coordinates CR1 to CR4 in the image coordinate system into the global coordinate system to obtain vehicle position information. In step 305, the remote control unit 216 uses the vehicle position information obtained using the captured image and the reference route RR to determine a target position to which the vehicle 100 should next head. In step 306, the remote control unit 216 generates a driving control signal for driving the vehicle 100 toward the determined target position. In step 307, the remote control unit 216 transmits the generated driving control signal to the vehicle 100.

[0048] When the vehicle control device 110 mounted on the vehicle 100 receives a driving control signal transmitted from the server 200, the vehicle control unit 115 of the vehicle control device 110 executes the following process in step 308. In this case, the vehicle control unit 115 controls the actuator group 120 using the received driving control signal to make the vehicle 100 drive at the acceleration and steering angle indicated in the driving control signal.

[0049] According to the first embodiment, the machine learning system 7 can calculate the position and orientation of the vehicle 100 using the three-dimensional point cloud information output from the external LiDAR 310. Then, the machine learning system 7 can generate a rectangular answer label using the position and orientation of the vehicle 100 calculated from the three-dimensional point cloud information. Then, the machine learning system 7 can create a first learning dataset by associating the rectangular answer label with the captured image as a training image. As a result, the machine learning system 7 can train a first learning model DM1 that outputs rectangular coordinates CR1 to CR4 as signal generation parameters when a captured image is input, using one or more first learning datasets. In this way, the server 200 can automatically generate a rectangular answer label and associate it with the captured image using identification information or the like, without a person manually attaching a rectangular answer label to the captured image, thereby allowing the server 200 to automatically create the first learning dataset. That is, the creation of the first learning dataset can be automated. As a result, the load of time and man-hours required for creating the first learning dataset can be reduced. In addition, when a person manually labels a captured image with a rectangle answer label, there is a risk that the accuracy of the rectangle answer label may decrease due to human error. In contrast, in the first embodiment, the creation of the first learning data set can be automated, so that a more appropriate first learning data set can be prepared. Therefore, the first learning model DM1 can be appropriately trained.

[0050] Furthermore, according to the first embodiment, the machine learning system 7 can generate a rectangular correct answer label using the position and orientation of the vehicle 100 calculated by matching between the three-dimensional point cloud information output from the external LiDAR 310 and the reference point cloud information. That is, the machine learning system 7 can generate a rectangular correct answer label without performing machine learning. In this way, it is possible to reduce the load of time and labor required to create the first learning dataset. This allows the first learning model DM1 to be more appropriately trained.

[0051] According to the first embodiment, the machine learning system 7 can use the position and orientation of the vehicle 100 calculated by the first calculation unit 211 in the signal generation process for generating a driving control signal and the label generation process for generating a rectangular answer label. That is, the machine learning system 7 can use the position and orientation of the vehicle 100 calculated using the three-dimensional point cloud information in both the signal generation process and the label generation process. In this way, during the period when the unmanned driving control is being performed, it is possible to generate a rectangular answer label, create a first learning dataset, and learn the first learning model DM1.

[0052] Furthermore, according to the first embodiment, when the learning of the first learning model DM1 is completed, the machine learning system 7 can acquire vehicle position information using the rectangular coordinates CR1 to CR4 acquired by inputting the captured image to the first learning model DM1. In this way, when the learning of the first learning model DM1 is completed, the use of the external LiDAR 310 can be stopped and removed. In general, LiDAR is more expensive than a camera. Therefore, when the learning of the first learning model DM1 is completed, the cost required for unmanned driving can be reduced by removing the external LiDAR 310.

[0053] Further, according to the first embodiment, when the learning of the first learning model DM1 is completed, the machine learning system 7 can acquire vehicle position information using the rectangular coordinates CR1 to CR4 acquired by inputting the captured image to the first learning model DM1. That is, when the learning of the first learning model DM1 is completed, the machine learning system 7 can acquire vehicle position information using the captured image. When the captured image is a color image, the captured image has information about the color of an object present in the detection range of the external camera 320 when the captured image is acquired, according to the number of gradations of the brightness and saturation of each pixel constituting the captured image. On the other hand, the reflectance of the laser light irradiated from the external LiDAR 310 differs depending on the color of the object present in the detection range of the external LiDAR 310, and therefore the density of the point cloud representing the object in the three-dimensional point cloud information differs. Therefore, the three-dimensional point cloud information represents information about the color of the object present in the detection range of the external LiDAR 310 when the three-dimensional point cloud information is acquired by the density of the point cloud. Therefore, depending on the number of gradations of each pixel constituting the captured image, the captured image may have more information about the color of an object present in the detection range of the external sensor 300 than the three-dimensional point cloud information. In this way, the captured image may have a larger amount of information than the three-dimensional point cloud information. In this case, when the learning of the first learning model DM1 is completed, the machine learning system 7 can acquire vehicle position information using the captured image having a larger amount of information than the three-dimensional point cloud information.

[0054] The signal generation parameters output from the machine learning model DM may be in the LiDAR coordinate system, the camera coordinate system, or the global coordinate system. The machine learning system 7 may calculate the signal generation parameters without calculating the coordinates of the positioning point of the vehicle 100. At least one of the correct answer label and the learning data set may be prepared outside the machine learning system 7, and the machine learning system 7 may learn the machine learning model DM by acquiring at least one of the correct answer label and the learning data set from another device.

[0055] B. Second embodiment: FIG. 6 is a block diagram showing the configuration of the traveling system 50a in the second embodiment. The traveling system 50a includes one or more vehicles 100, a machine learning system 7a, and a remote control device 80a. The machine learning system 7a includes an external LiDAR 310 and an external camera 320 as the external sensor 300, and a machine learning device 70a. The functions of the machine learning device 70a and the functions of the remote control device 80a are realized by the server 200a. In this embodiment, the type of the machine learning model DM and the processing contents after the learning of the machine learning model DM is completed are different from those in the first embodiment. In this embodiment, the machine learning model DM is a second learning model DM2. The other configurations of the traveling system 50a are the same as those in the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those in the first embodiment, and the description is omitted.

[0056] The second learning model DM2 outputs three-dimensional coordinates in the image coordinate system as signal generation parameters. Fig. 7 is a diagram for explaining the three-dimensional coordinates CB1 to CB8. The three-dimensional coordinates CB1 to CB8 are the coordinates of eight vertices VB1 to VB8 of a rectangular parallelepiped RB circumscribing the vehicle. The rectangular parallelepiped RB circumscribing the vehicle is a rectangular parallelepiped that is set so as to surround the vehicle 100 in the detection result output from the external sensor 300. In this embodiment, the position of the vehicle 100 is represented by the three-dimensional coordinates CB1 to CB8.

[0057] 6 is configured by a computer including a processor 201a, a memory 202a, an input / output interface 203, an internal bus 204, and a notification unit 209. The processor 201a executes a program PG2a stored in the memory 202a. In this way, the processor 201a realizes various functions including functions as a first calculation unit 211a, a label generation unit 212a, a data set creation unit 213a, a learning unit 214a, a second calculation unit 215a, a determination unit 217, a remote control unit 216a, and an execution unit 218.

[0058] The first calculation unit 211a acquires vehicle position information by calculating the position and orientation of the vehicle 100 using the three-dimensional point cloud information during the learning period and the learned period. Specifically, the first calculation unit 211a first calculates the coordinates of the positioning points of the vehicle 100 in the LiDAR coordinate system by matching using the three-dimensional point cloud information and the reference point cloud information. Next, the first calculation unit 211a converts the coordinates of the positioning points of the vehicle 100 in the LiDAR coordinate system into coordinates in the global coordinate system using the LiDAR parameters RP stored in the memory 202a of the server 200a. Next, the first calculation unit 211a calculates three-dimensional coordinates CB1 to CB8 from the coordinates of the positioning points of the vehicle 100 using the three-dimensional database DB stored in the memory 202a of the server 200a. The three-dimensional database DB is a database that indicates the relative positional relationship between each vertex VB1 to VB8 of the vehicle circumscribed rectangular parallelepiped RB and the positioning points. In this manner, the first calculation unit 211a calculates the three-dimensional coordinates CB1 to CB8 as the position of the vehicle 100 in the global coordinate system.

[0059] Further, the first calculation unit 211a calculates the orientation of the vehicle 100 using the three-dimensional coordinate CB1. Specifically, the first calculation unit 211a calculates the orientation of the vehicle 100 using the coordinates of the first center position CN12 and the coordinates of the second center position CN22 shown in Fig. 7. The first center position CN12 is the center position of the first side SB1 that is aligned along the vehicle width direction at the front side and bottom side of the vehicle 100 among the twelve sides SB1 to SB12 of the vehicle circumscribing rectangular parallelepiped RB. The second center position CN22 is the center position of the second side SB2 that is aligned along the vehicle width direction at the rear side and bottom side of the vehicle 100 among the twelve sides SB1 to SB12 of the vehicle circumscribing rectangular parallelepiped RB.

[0060] The first calculation unit 211a may calculate the position and orientation of the vehicle 100 by a method different from the above. The first calculation unit 211a may calculate the orientation of the vehicle 100 by using, for example, the coordinates of the third center position CN32 and the coordinates of the fourth center position CN42. The third center position CN32 is the center position of the third side SB3 that is aligned along the vehicle width direction on the front side and ceiling side of the vehicle 100 among the twelve sides SB1 to SB12 of the vehicle circumscribing rectangular parallelepiped RB. The fourth center position CN42 is the center position of the fourth side SB4 that is aligned along the vehicle width direction on the rear side and ceiling side of the vehicle 100 among the twelve sides SB1 to SB12 of the vehicle circumscribing rectangular parallelepiped RB.

[0061] The label generating unit 212a shown in FIG. 6 generates a correct answer label according to the output result of the second learning model DM2 using the position and orientation of the vehicle 100 calculated by the first calculating unit 211a. The label generating unit 212a generates, as a correct answer label, a three-dimensional correct answer label that represents the coordinates of eight vertices VB1 to VB8 of a vehicle circumscribed rectangular parallelepiped RB shown in FIG. 7 set for the vehicle 100 included in the captured image as a training image acquired by the external camera 320, in the image coordinate system. Specifically, the label generating unit 212a converts the three-dimensional coordinates CB1 to CB8 in the global coordinate system, which are the position of the vehicle 100 calculated by the first calculating unit 211a, into the image coordinate system, using the orientation of the vehicle 100 calculated by the first calculating unit 211a and the camera parameter CP. In this way, the label generating unit 212a generates a three-dimensional correct answer label that indicates the three-dimensional coordinates CB1 to CB8 in the image coordinate system. In the solid correct label, each coordinate is associated with additional information indicating which of the eight vertices VB1 to VB8 of the vehicle circumscribing rectangular solid RB the coordinates belong to. The label generating unit 212a may generate the solid correct label by a method different from the above.

[0062] The data set creation unit 213a creates a second training data set by associating a stereoscopic answer label with a captured image as a training image. In this embodiment, the data set creation unit 213a creates a second training data set for each external camera 320.

[0063] The learning unit 214a trains the second learning model DM2 by supervised learning using one or more second learning datasets. For the second learning model DM2, for example, a CNN that realizes either semantic segmentation or instance segmentation can be used. During training of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce an error between the output result by the second learning model DM2 and the stereo ground truth label. In this embodiment, the learning unit 214a trains the second learning model DM2 for each external camera 320.

[0064] The second calculation unit 215a acquires vehicle position information by calculating the position and orientation of the vehicle 100 using the captured image during the learned period. Specifically, the second calculation unit 215a first acquires three-dimensional coordinates CB1 to CB8 in the image coordinate system as signal generation parameters by inputting the captured image to the learned second learning model DM2. Next, the second calculation unit 215a converts the three-dimensional coordinates CB1 to CB8 in the image coordinate system into coordinates in the global coordinate system using the camera parameters CP stored in the memory 202a of the server 200a. In this manner, the second calculation unit 215a calculates the three-dimensional coordinates CB1 to CB8 in the global coordinate system as the position of the vehicle 100. Similarly to the first calculation unit 211a, the second calculation unit 215a calculates the orientation of the vehicle 100 using the three-dimensional coordinates CB1 to CB8. That is, in this embodiment, both the acquisition of vehicle position information using the three-dimensional point cloud information and the acquisition of vehicle position information using the captured image are executed during the learned period. The second calculation unit 215a may calculate the position and orientation of the vehicle 100 by a method different from the above.

[0065] The judgment unit 217 judges whether or not a defect occurs in the second learning model DM2 using the accuracy of the second learning model DM2 during the learned period. The accuracy of the second learning model DM2 is acquired by comparing the output result by the second learning model DM2 with the three-dimensional correct label. The accuracy of the second learning model DM2 is, for example, any of the accuracy rate, the matching rate, the recall rate, and the F value for the second learning model DM2. When the accuracy of the second learning model DM2 is equal to or greater than a predetermined threshold, the remote control unit 216a judges that no defect occurs in the second learning model DM2. On the other hand, when the accuracy of the second learning model DM2 is less than a predetermined threshold, the judgment unit 217 judges that a defect occurs in the second learning model DM2. In other words, the judgment unit 217 judges that a defect occurs in the second learning model DM2 when a difference between the vehicle position information acquired using the three-dimensional point cloud information and the vehicle position information acquired using the captured image is equal to or greater than a predetermined value.

[0066] When the determination unit 217 determines that there is no defect in the second learning model DM2, the remote control unit 216a uses the vehicle position information and the reference route RR to determine the target position to which the vehicle 100 should next head. At this time, in order to determine the target position, the remote control unit 216a uses, for example, the average value of the vehicle position information acquired using the three-dimensional point cloud information and the vehicle position information acquired using the captured image as the vehicle position information. In order to determine the target position, the remote control unit 216a may use either the vehicle position information acquired using the three-dimensional point cloud information or the vehicle position information acquired using the captured image. Then, the remote control unit 216a generates a driving control signal for driving the vehicle 100 toward the determined target position, and transmits the generated driving control signal to the vehicle 100.

[0067] On the other hand, if it is determined that a defect occurs in the second learning model DM2, at least one of the first process, the second process, the third process, and the fourth process is executed.

[0068] The first process is an additional learning process for additionally learning the second learning model DM2 that has already been learned. The additional learning process is, for example, at least one of a re-learning process and a transfer learning process. In the re-learning process, the learning unit 214a updates at least a part of the trained parameters of the second learning model DM2. In the re-learning process, the learning unit 214a may execute a process for adding a layer of CNN that constitutes the second learning model DM2. In the transfer learning process, the learning unit 214a adds, for example, a layer of CNN that constitutes the second learning model DM2. Then, the learning unit 214a updates parameters related to the added layer. Note that, in the transfer learning process, at least a part of the parameter update work may be performed manually by a person. That is, the second learning model DM2 may be transfer-learned by a user correcting a part of the parameters mechanically determined by the learning unit 214a. In this case, for example, the user may correct the parameters of the layer that has learned more features according to the type of the vehicle 100. In this way, it is possible to prepare a second learning model DM2 that can more reliably capture the features desired by the user.

[0069] The second process is a process of notifying the user of error information indicating that the accuracy of the second learning model DM2 may be reduced. The error information may be, for example, text information or audio information. In the second process, the execution unit 218 notifies the user of the error information via the notification unit 209. The notification unit 209 may be, for example, a display that displays the error information as text information or a speaker that reproduces the error information as audio information.

[0070] The third process is a process of storing an error code indicating that the accuracy of the second learning model DM2 may be degraded in the memory 202a of the server 200a. In the third process, the execution unit 218 stores the error code in the memory 202a of the server 200a.

[0071] The fourth process is a process for stopping the vehicle 100. In the fourth process, the remote control unit 216a generates a stop signal, which is a travel control signal for stopping the vehicle 100. The stop signal includes, for example, the acceleration of the vehicle 100 as a parameter. For example, the remote control unit 216a calculates the traveling speed of the vehicle 100 from the transition of the position of the vehicle 100, and determines the acceleration so that the vehicle 100 decelerates below the calculated traveling speed. Note that, in order to stop the vehicle 100 at a desired location such as a road shoulder, the remote control unit 216a may generate a stop signal including the steering angle of the vehicle 100 as a parameter.

[0072] 8 is a flowchart showing the processing contents during the learning period in the second embodiment. In step 401, the external LiDAR 310 acquires three-dimensional point cloud information. In step 402, the external LiDAR 310 transmits three-dimensional point cloud information in which the vehicle identifier and the point cloud identifier are associated with each other to the server 200a. In step 403, the external camera 320 acquires a captured image. In step 404, the external camera 320 transmits the captured image in which the vehicle identifier and the image identifier are associated with each other to the server 200a.

[0073] In step 405, the first calculation unit 211a of the server 200a acquires vehicle position information using the three-dimensional point cloud information. After step 405, a label generation process shown in step 406 and a signal generation process shown in steps 410 and 411 are respectively executed.

[0074] After step 405, in step 406, the label generating unit 212a of the server 200a generates a stereoscopic correct label using the vehicle position information acquired using the three-dimensional point cloud information. In step 407, the dataset creating unit 213a associates the stereoscopic correct label with the captured image corresponding to the acquisition timing of the three-dimensional point cloud information used to generate the stereoscopic correct label using the vehicle identifier, the point cloud identifier, and the image identifier. In this way, the dataset creating unit 213a creates a second learning dataset. Each step from step 401 to step 407 is repeatedly executed until N second learning datasets are created. If step 408 is "Yes", that is, if N second learning datasets are created, in step 409, the learning unit 214a trains the second learning model DM2 using the N second learning datasets.

[0075] After step 405, in step 410, the remote control unit 216a determines a target position to which the vehicle 100 should next head, using the vehicle position information acquired using the three-dimensional point cloud information and the reference route RR. In step 411, the remote control unit 216a generates a driving control signal for driving the vehicle 100 toward the determined target position. In step 412, the remote control unit 216a transmits the generated driving control signal to the vehicle 100.

[0076] When the vehicle control device 110 mounted on the vehicle 100 receives a driving control signal transmitted from the server 200a, the vehicle control unit 115 of the vehicle control device 110 executes the following process in step 413. In this case, the vehicle control unit 115 controls the actuator group 120 using the received driving control signal to make the vehicle 100 drive with the acceleration and steering angle represented in the driving control signal. During the learning period, steps 401, 402, 405, and steps 410 to 413 are repeatedly executed at predetermined time intervals.

[0077] Fig. 9 is a first flowchart showing the processing contents in the learned period in the second embodiment. Fig. 10 is a second flowchart showing the processing contents in the learned period in the second embodiment. The flows shown in Figs. 9 and 10 are repeatedly executed at predetermined time intervals in the learned period.

[0078] As shown in Fig. 9, in step 501, the external LiDAR 310 acquires three-dimensional point cloud information. In step 502, the external LiDAR 310 transmits the three-dimensional point cloud information in which the vehicle identifier and the point cloud identifier are associated to the server 200a. In step 503, the external camera 320 acquires a captured image. In step 504, the external camera 320 transmits the captured image in which the vehicle identifier and the image identifier are associated to the server 200a.

[0079] In step 505, the first calculation unit 211a of the server 200a acquires vehicle position information using the three-dimensional point cloud information. In step 506, the second calculation unit 215a acquires three-dimensional coordinates CB1 to CB8 in the image coordinate system by inputting the captured image into the trained second learning model DM2. In step 507, the second calculation unit 215a acquires vehicle position information by performing various processes such as converting the three-dimensional coordinates CB1 to CB8 in the image coordinate system into the global coordinate system. In step 508, the judgment unit 217 acquires the accuracy of the second learning model DM2.

[0080] As shown in FIG. 10, if step 509 is "Yes", that is, if the accuracy of the second learning model DM2 is equal to or greater than a predetermined threshold, in step 510, the judgment unit 217 judges that there is no defect in the second learning model DM2. If it is judged that there is no defect in the second learning model DM2, in step 511, the remote control unit 216a uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. In step 512, the remote control unit 216a generates a driving control signal for driving the vehicle 100 toward the determined target position. In step 513, the remote control unit 216a transmits the generated driving control signal to the vehicle 100.

[0081] When the vehicle control device 110 mounted on the vehicle 100 receives a driving control signal transmitted from the server 200a, the vehicle control unit 115 of the vehicle control device 110 executes the following process in step 514. In this case, the vehicle control unit 115 controls the actuator group 120 using the received driving control signal to make the vehicle 100 drive at the acceleration and steering angle indicated in the driving control signal.

[0082] If step 509 is "No", i.e., if the accuracy of the second learning model DM2 is less than the threshold, the judgment unit 217 judges that a deficiency occurs in the second learning model DM2 in step 515. If it is judged that a deficiency occurs in the second learning model DM2, in step 516, the processor 201a of the server 200a executes at least one of the first process, the second process, the third process, and the fourth process.

[0083] According to the second embodiment, the machine learning system 7a can generate a stereoscopic correct label by using the position and orientation of the vehicle 100 calculated from the three-dimensional point cloud information. The machine learning system 7a can create a second learning data set by associating the stereoscopic correct label with a captured image as a training image. This allows the machine learning system 7a to train a second learning model DM2 that outputs stereoscopic coordinates CB1 to CB8 as signal generation parameters when a captured image is input, using one or more second learning data sets.

[0084] Furthermore, according to the second embodiment, when the learning of the second learning model DM2 is completed, the machine learning system 7a can acquire the accuracy of the second learning model DM2 based on the difference between the position and orientation of the vehicle 100 calculated by the first calculation unit 211a and the position and orientation of the vehicle 100 calculated by the second calculation unit 215a. When the accuracy of the second learning model DM2 is less than a threshold, the machine learning system 7a can determine that a defect occurs in the second learning model DM2 and execute at least one of the first process, the second process, the third process, and the fourth process. This makes it possible to address the defect in the second learning model DM2.

[0085] C. Third embodiment: FIG. 11 is a block diagram showing the configuration of a traveling system 50v in the third embodiment. The traveling system 50v includes one or more vehicles 100v and a machine learning system 7v. The machine learning system 7v includes an external LiDAR 310 and an external camera 320 as an external sensor 300, and a machine learning device 70v. The function of the machine learning device 70v is realized by a server 200v. In this embodiment, the traveling system 50v differs from the first embodiment in that it does not include a remote control device 80. In addition, the vehicle 100v in this embodiment can travel by autonomous control of the vehicle 100v. The other configurations of the traveling system 50v are the same as those of the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those of the first embodiment, and the description is omitted.

[0086] The server 200v is configured by a computer including a processor 201v, a memory 202v, an input / output interface 203, and an internal bus 204. The processor 201v executes a program PG2v stored in the memory 202v. In this way, the processor 201v realizes various functions including functions as a first calculation unit 211, a label generation unit 212, a dataset creation unit 213, and a learning unit 214.

[0087] The vehicle control device 110v is configured by a computer including a processor 111v, a memory 112v, an input / output interface 113, and an internal bus 114. The processor 111v executes a program PG1v stored in the memory 112v. As a result, the processor 111v realizes various functions as a position information acquisition unit 116, a signal generation unit 117, and a vehicle control unit 115v.

[0088] The position information acquisition unit 116 acquires vehicle position information. In this embodiment, during the learning period, the position information acquisition unit 116 acquires three-dimensional point cloud information from the external LiDAR 310. Then, the position information acquisition unit 116 acquires vehicle position information by calculating the position and orientation of the vehicle 100v using the three-dimensional point cloud information. During the learned period, the position information acquisition unit 116 acquires a captured image from the external camera 320. Then, the position information acquisition unit 116 acquires rectangular coordinates CR1 to CR4 in the image coordinate system by inputting the captured image to the learned first learning model DM1 created in the server 200v and stored in the memory 112v of the vehicle control device 110v. Then, the position information acquisition unit 116 calculates the position of the vehicle 100v by converting the rectangular coordinates CR1 to Cr4 in the image coordinate system into coordinates in the global coordinate system using the camera parameters CP stored in the memory 112v of the vehicle control device 110v. Furthermore, the position information acquisition unit 116 calculates the orientation of the vehicle 100v using the rectangular coordinates CR1 to CR4. In this manner, during the learned period, the position information acquisition unit 116 calculates the position and orientation of the vehicle 100v using the captured image, thereby acquiring vehicle position information.

[0089] The signal generating unit 117 determines a target position to which the vehicle 100v should next travel, using the vehicle position information and the reference route RR stored in the memory 112v of the vehicle control device 110v. Then, the signal generating unit 117 generates a travel control signal for causing the vehicle 100v to travel toward the determined target position.

[0090] The vehicle control unit 115 controls the actuator group 120 using the generated driving control signal, thereby causing the vehicle 100v to drive in accordance with the parameters represented in the driving control signal.

[0091] Fig. 12 is a flowchart showing a processing procedure for driving control of the vehicle 100v in the third embodiment. The flow shown in Fig. 12 is repeatedly executed at predetermined time intervals from the time when the vehicle 100v starts to drive in an unmanned manner.

[0092] In step 601, the position information acquisition unit 116 of the vehicle control device 110v acquires vehicle position information using the detection result output from the external sensor 300. In step 602, the signal generation unit 117 determines a target position to which the vehicle 100v should next head. In step 603, the signal generation unit 117 generates a driving control signal for driving the vehicle 100v toward the determined target position. In step 604, the vehicle control unit 115v controls the actuator group 120 using the generated driving control signal, thereby causing the vehicle 100v to drive according to parameters represented in the driving control signal. The processor 111v repeats the acquisition of vehicle position information, the determination of the target position, the generation of the driving control signal, and the control of the actuators at a predetermined cycle.

[0093] According to the third embodiment, the traveling system 50v can cause the vehicle 100v to travel by autonomous control of the vehicle 100v without remote control of the vehicle 100v by the server 200v.

[0094] D. Other embodiments: D-1. Other embodiment 1: At least a part of the functions of the servers 200, 200a, 200v may be a function of the vehicle control devices 110, 110v, or a function of the external sensor 300. Furthermore, the machine learning devices 70, 70v may be configured with a learning dataset creation device having functions of the first calculation units 211, 211a, the label generation units 212, 212a, and the dataset creation units 213, 213a, etc., and a learning device having functions of the learning units 214, 214a, etc.

[0095] D-2. Other embodiment 2: The learning dataset used for learning the machine learning model DM may further include incidental information associated with the captured image as the training image and the correct answer label, and may include incidental information on factors that affect the output result of the machine learning model DM. The factors that affect the output result of the machine learning model DM are, for example, at least one of the specification values ​​of the vehicles 100 and 100v and the imaging conditions when the captured image as the training image was acquired. The specification values ​​of the vehicles 100 and 100v are, for example, information on the vehicle class that represents the size of the vehicle body such as the overall length, width, and height of the vehicles 100 and 100v. The information on the vehicle class may be an actual value, a design value, or the length and size on the data output from the external sensor 300. The specification values ​​of the vehicles 100 and 100v may be information on the exterior color of the vehicles 100 and 100v, information on the type of the vehicles 100 and 100v, or information on the product name, model, and specifications of the vehicles 100 and 100v. The specification values ​​of the vehicle 100, 100v may be information on the body type of the vehicle 100, 100v. The body type is a group of vehicle types when a plurality of types of the vehicle 100, 100v are classified according to the vehicle class and the external shape of the vehicle 100, 100v. The body types are, for example, "SUV", "sedan", "station wagon", "minivan", "mini-box", "compact car", and "light car". The imaging conditions when the captured image is acquired are, for example, the time, the time period such as morning, afternoon, and night, the weather, the season, and the like. In this case, the second calculation unit 215, 215a and the position information acquisition unit 116 input the captured image and information corresponding to the incidental information to the trained machine learning model DM to acquire the signal generation parameters. In this form, the machine learning system 7, 7a, 7v can train the machine learning model DM using one or more learning datasets in which incidental information on factors that affect the output result of the machine learning model DM is associated with the captured image and the correct answer label. This makes it possible to improve the accuracy of the signal generation parameters output from the machine learning model DM.

[0096] D-3. Other embodiment 3: The machine learning systems 7, 7a, and 7v may train the machine learning model DM by executing at least one of an individual learning process and a simultaneous learning process. The individual learning process is a process of training a plurality of machine learning models DM for each type of the vehicles 100 and 100v. The simultaneous learning process is a process of training one machine learning model DM using a learning data set in which the captured images as training images and the correct answer labels are associated with additional information related to the types of the vehicles 100 and 100v. The additional information related to the vehicle types of the vehicles 100 and 100v is acquired by, for example, using management information to identify the types of the vehicles 100 and 100v represented by the captured images. The management information is information indicating the travel order of the plurality of vehicles 100 and 100v traveling within the detection range of the external sensor 300 in association with the types of the vehicles 100 and 100v. The management information is created using, for example, vehicle position information, a transmission history of a driving control signal to the vehicle 100, 100v, and an installation location of the external sensor 300, and is stored in the memory 112v, 202, 202a. In this form, the machine learning systems 7, 7a, 7v can train a plurality of machine learning models DM for each type of the vehicle 100, 100v by executing an individual learning process. In addition, the machine learning systems 7, 7a, 7v can train the machine learning model DM using one or more learning data sets in which the auxiliary information related to the type of the vehicle 100, 100v is associated with the captured image and the correct answer label by executing a simultaneous learning process. In this way, the accuracy of the signal generation parameters output from the machine learning model DM can be improved. Note that the machine learning systems 7, 7a, 7v may execute either the individual learning process or the simultaneous learning process, or may execute both the individual learning process and the simultaneous learning process. The machine learning systems 7, 7a, and 7v may, for example, perform individual learning processes for a first body type and a second body type, and perform simultaneous learning processes for the other body types.

[0097] D-4. Other embodiment 4: The machine learning system 7, 7a, 7v may train one machine learning model DM commonly used for the captured images acquired by each of the multiple external cameras 320. In this case, the data set creation unit 213, 213a creates a learning data set in which the captured images as training images are associated with a correct answer label and additional information related to the external camera 320 that acquired the captured images. Then, the learning unit 214, 214a trains the machine learning model DM using the learning data set in which the captured images as training images are associated with a correct answer label and additional information related to the external camera 320 that acquired the captured images. At this time, the second calculation unit 215, 215a may acquire the signal generation parameters by inputting the captured images and the additional information related to the external camera 320 that acquired the captured images to the trained machine learning model DM. Even in this form, the machine learning system 7, 7a, 7v can train the machine learning model DM using a learning dataset in which correct labels generated using the position and orientation of the vehicle 100 calculated from the three-dimensional point cloud information are associated with captured images.

[0098] D-5. Other embodiment 5: The machine learning system 7, 7a, 7v may be a third learning model that is a machine learning model DM that outputs shape data as a signal generation parameter when a captured image is input. The shape data is data indicating the external shape of the vehicle 100, 100v. The shape data is generated by detecting the vehicle 100, 100v from the captured image. The shape data is, for example, a mask image in which a mask area is added to the captured image by masking an area representing the vehicle 100, 100v among the areas constituting the captured image. Each area constituting the captured image is, for example, one pixel constituting the captured image. In this case, the label generating unit 212, 212a generates an area correct answer label that is a correct answer label according to the output result of the third learning model using the position and orientation of the vehicle 100, 100v calculated by the first calculating unit 211, 211a. The area correct answer label is a correct answer label that indicates whether each area in the training image is an area representing the vehicle 100, 100v or an area other than the vehicle 100, 100v. The data set creation unit 213, 213a creates a third learning data set. The third learning data set has, for example, a captured image as a training image including the vehicle 100, 100v, and a region answer label. The learning unit 214, 214a trains a third learning model by supervised learning using one or more third learning data sets. After the learning of the third learning model is completed, the second calculation unit 215, 215a and the position information acquisition unit 116 input the captured image to the trained third learning model, thereby detecting the outer shape of the vehicle 100, 100v from the captured image and acquiring shape data. Then, the second calculation unit 215, 215a and the position information acquisition unit 116 calculate, for example, the coordinates of the positioning points of the vehicle 100, 100v in the coordinate system of the captured image, i.e., the local coordinate system, and convert the calculated coordinates into coordinates in the global coordinate system to acquire the position of the vehicle 100. In this embodiment, the machine learning system 7, 7a, 7v can train a third learning model using a third learning data set in which a region answer label generated using the position and orientation of the vehicle 100 calculated from the three-dimensional point cloud information is associated with a captured image.

[0099] D-6. Other embodiment 6: In the first and second embodiments, the servers 200 and 200a execute the processes from obtaining the vehicle position information to generating the driving control signal. In contrast, at least a part of the processes from obtaining the vehicle position information to generating the driving control signal may be executed by the vehicle 100. For example, the following forms (1) to (3) may be used.

[0100] (1) The server 200, 200a may acquire vehicle position information, determine a target position to which the vehicle 100 should next head, and generate a route from the current location of the vehicle 100 indicated in the acquired vehicle position information to the target position. The server 200, 200a may generate a route to a target position between the current location and the destination, or may generate a route to the destination. The server 200, 200a may transmit the generated route to the vehicle 100. The vehicle 100 may generate a driving control signal so that the vehicle 100 drives on the route received from the server 200, 200a, and control the actuator group 120 using the generated driving control signal.

[0101] (2) Server 200, 200a may acquire vehicle position information and transmit the acquired vehicle position information to vehicle 100. Vehicle 100 may determine a target position to which vehicle 100 should next head, generate a route from the current location of vehicle 100 represented in the received vehicle position information to the target position, generate a driving control signal so that vehicle 100 travels on the generated route, and control actuator group 120 using the generated driving control signal.

[0102] (3) In the above-mentioned (1) and (2) modes, the vehicle 100 may be equipped with an internal sensor, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. The internal sensor is a sensor equipped in the vehicle 100. The internal sensor may include, for example, a sensor for detecting a motion state of the vehicle 100, a sensor for detecting an operating state of each part of the vehicle 100, and a sensor for detecting an environment around the vehicle 100. Specifically, the internal sensor may include, for example, a camera, a LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyro sensor, and the like. For example, in the above-mentioned (1) mode, the server 200 may acquire a detection result of the internal sensor, and may reflect the detection result of the internal sensor on the route when generating a route. In the above-mentioned (1) mode, the vehicle 100 may acquire a detection result of the internal sensor, and may reflect the detection result of the internal sensor on the driving control signal when generating a driving control signal. In the above-mentioned (2) mode, the vehicle 100 may acquire a detection result of the internal sensor, and may reflect the detection result of the internal sensor on the route when generating a route. In the above embodiment (2), the vehicle 100 may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor in the driving control signal when generating the driving control signal.

[0103] D-7. Other embodiment 7: In the above third embodiment, an internal sensor may be mounted on the vehicle 100v, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. For example, the vehicle 100v may acquire a detection result of the internal sensor, and when generating a route, may reflect the detection result of the internal sensor in the route. The vehicle 100v may acquire a detection result of the internal sensor, and when generating a driving control signal, may reflect the detection result of the internal sensor in the driving control signal.

[0104] D-8. Other embodiment 8: In the third embodiment, the vehicle 100v acquires the vehicle position information using the detection result of the external sensor 300. In contrast, the vehicle 100v may be equipped with an internal sensor, acquire vehicle position information using the detection result of the internal sensor, determine a target position to which the vehicle 100v should next head, generate a route from the current location of the vehicle 100v represented in the acquired vehicle position information to the target position, generate a travel control signal for traveling the generated route, and control the actuator group 120 using the generated travel control signal. In this case, the vehicle 100v can travel without using any of the detection results of the external sensor 300. The vehicle 100v may acquire a target arrival time or traffic congestion information from outside the vehicle 100v, and reflect the target arrival time or traffic congestion information in at least one of the route and the travel control signal.

[0105] D-9. Other embodiment 9: In the above-mentioned other embodiment 7 and the above-mentioned other embodiment 8, when an obstacle occurs in the traveling of the vehicle 100v by the autonomous control, the vehicle 100v may travel by remote control by the server 200v by receiving any of the following from the server 200v. In this case, the vehicle 100v receives at least any of a traveling control signal, vehicle position information, and a route along which the vehicle 100v should travel from the server 200v. In other words, when an obstacle occurs in the traveling of the vehicle 100v by the autonomous control, the driving mode of the vehicle 100v may be switched from the driving mode by the autonomous control to the driving mode by the remote control. In this form, the server 200v can support the traveling of the vehicle 100v according to the traveling situation of the vehicle 100v traveling by the autonomous control.

[0106] D-10. Other embodiment 10: In the first and second embodiments, the server 200, 200a automatically generates a driving control signal to be transmitted to the vehicle 100. In contrast, the server 200, 200a may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a steering device including a display for displaying an image output from the external camera 320 as the external sensor 300, a steering wheel for remotely operating the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the server 200, 200a by wired communication or wireless communication, and the server 200, 200a may generate a driving control signal in accordance with the operation applied to the steering device.

[0107] D-11. Other embodiment 11: In each of the above embodiments, the vehicle 100, 100v may have a configuration capable of moving by unmanned driving, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100, 100v may have at least the vehicle control device 110, 110v and the actuator group 120 in order to perform the three functions of "running", "turning" and "stopping" by unmanned driving. When the vehicle 100, 100v acquires information from the outside for unmanned driving, the vehicle 100, 100v may further have a communication device 130. That is, the vehicle 100, 100v capable of moving by unmanned driving may not be equipped with at least a part of interior parts such as a driver's seat and a dashboard, may not be equipped with at least a part of exterior parts such as a bumper and a fender, and may not be equipped with a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100, 100v before the vehicle 100, 100v is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 100, 100v after the vehicle 100 is shipped from the factory without the remaining parts such as the body shell being attached to the vehicle 100, 100v. Each part may be attached from any direction such as the upper side, lower side, front side, rear side, right side, or left side of the vehicle 100, 100v, and may be attached from the same direction or from different directions. Note that the position of the platform shape may be determined in the same manner as the vehicle 100, 100v in the first embodiment.

[0108] D-12. Other embodiment 12: The vehicle 100, 100v may be manufactured by combining a plurality of modules. The module means a unit composed of a plurality of parts grouped according to the parts and functions of the vehicle 100, 100v. For example, the platform of the vehicle 100, 100v may be manufactured by combining a front module constituting the front part of the platform, a central module constituting the central part of the platform, and a rear module constituting the rear part of the platform. The number of modules constituting the platform is not limited to three, and may be two or less or four or more. In addition to or instead of the parts constituting the platform, parts constituting parts of the vehicle 100, 100v other than the platform may be modularized. The various modules may include any exterior parts such as a bumper or a grille, or any interior parts such as a seat or a console. In addition to the vehicle 100, 100v, any type of moving body may be manufactured by combining a plurality of modules. Such a module may be manufactured, for example, by joining a plurality of parts by welding or a fastener, or may be manufactured by integrally molding at least a part of the parts constituting the module as one part by casting. The molding method for integrally molding a single component, particularly a relatively large component, is also called gigacast or megacast. For example, the front module, the center module, and the rear module may be manufactured using gigacast.

[0109] D-13. Other embodiment 13: Transporting vehicles 100, 100v using unmanned driving of vehicles 100, 100v is also called "self-propelled transport." Also, a configuration for realizing self-propelled transport is also called a "vehicle remote-controlled autonomous driving transport system." Also, a production method for producing vehicles 100, 100v using self-propelled transport is also called "self-propelled production." In self-propelled production, for example, at a factory where vehicles 100, 100v are manufactured, at least a portion of the transportation of vehicles 100, 100v is realized by self-propelled transport.

[0110] D-14. Other embodiment 14: In each of the above embodiments, some or all of the functions and processes implemented by software may be implemented by hardware. Also, some or all of the functions and processes implemented by hardware may be implemented by software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0111] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0112] 7, 7a, 7v...machine learning system, 50, 50a, 50v...driving system, 70, 70a, 70v...machine learning device, 80, 80a...remote control device, 100, 100v...vehicle, 110, 110v...vehicle control device, 111, 111v...vehicle control device processor, 112, 112v...vehicle control device memory, 113...vehicle control device input / output interface, 114...vehicle control device internal bus, 115, 115v...vehicle control unit, 116...position information acquisition unit , 117...signal generation unit, 120...actuator group, 130...vehicle communication device, 200, 200a, 200v...server, 201, 201a, 201v...server processor, 202, 202a, 202v...server memory, 203...server input / output interface, 204...server internal bus, 205...server communication device, 209...notification unit, 211, 211a...first calculation unit, 212, 212a...label generation unit, 213, 213a...data set creation unit , 214, 214a... learning unit, 215, 215a... second calculation unit, 216, 216a... remote control unit, 217... judgment unit, 218... execution unit, 300... external sensor, 310... external LiDAR, 320... external camera, CB1 to CB8... three-dimensional coordinates, CN11, CN12... first central position, CN21, CN22... second central position, CN32... third central position, CN42... fourth central position, CP... camera parameters, CR1 to CR4... rectangular coordinates, DB... three-dimensional database, D C...rectangle database, DM...machine learning model, DM1...first learning model, DM2...second learning model, PG1,PG1v,PG2,PG2a,PG2v...program, RB...vehicle circumscribing rectangle, RC...vehicle circumscribing rectangle, RP...LiDAR parameters, RR...reference route, RS...road surface, SB1~SB12...sides of vehicle circumscribing rectangle, SC1~SC4...sides of vehicle circumscribing rectangle, TR...road, VB1~VB8...vertices of vehicle circumscribing rectangle, VC1~VC4...vertices of vehicle circumscribing rectangle

Claims

1. A machine learning system, a distance measuring device that acquires three-dimensional point cloud information representing the moving object by detecting, from the outside, a moving object that can move by autonomous driving; an external camera that acquires a captured image including the moving object by imaging the moving object from the outside; a calculation unit that calculates the position and orientation of the moving object using the three-dimensional point cloud information; a learning unit that learns a machine learning model that outputs signal generation parameters when the captured image is input, the learning unit learning the machine learning model using one or more learning data sets in which the captured image is associated with a correct label generated using the position and the orientation calculated by the calculation unit; The signal generation parameter is a parameter used when generating a control signal that defines the operation of the moving object in order to move the moving object by autonomous driving. A machine learning system.

2. The machine learning system according to claim 1, wherein the position and the orientation calculated by the calculation unit are respectively used for a signal generation process for generating the control signal and a label generation process for generating the correct label. A machine learning system.

3. The machine learning system according to claim 1, wherein the signal generation parameter is at least one of rectangular coordinates and three-dimensional coordinates, the rectangular coordinates are coordinates of four vertices of a rectangle set so as to surround a region occupied by the moving object when the moving object is projected onto a road surface on which the moving object moves in the captured image, and the three-dimensional coordinates are coordinates of eight vertices of a rectangular parallelepiped set so as to surround the moving object in the captured image. A machine learning system.

4. The machine learning system according to claim 1, wherein the learning data set further includes additional information associated with the captured image and the correct label, the additional information being information regarding an element that affects the output result of the machine learning model. A machine learning system.

5. The machine learning system according to claim 1, wherein the learning unit performs individual learning processing for learning a plurality of the machine learning models for each type of the moving object, A machine learning system that executes at least one of batch learning processes for training one of the machine learning models using the learning dataset in which the additional information regarding the type of the moving body is associated with the captured image and the correct label.

Images