Information processing device, information processing system, and information processing method

Abstract

To provide a technique for suppressing decrease in accuracy of a machine learning model when an appearance state in an imaging range of a camera has changed from the appearance state at the time when the machine learning model has trained.SOLUTION: An information processing device used to move a moving object that is capable of moving by unmanned driving includes: an accuracy acquisition unit which acquires accuracy of a machine learning model: a training data acquisition unit which acquires a training data group used to train the machine learning model when the accuracy is less than a predetermined threshold; and a training unit which performs a training process for training the machine learning model using the training data group.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a technique is known in which a vehicle is made to travel automatically by remote control by monitoring the travel of the vehicle using a camera outside the vehicle (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to move a moving object such as a vehicle in an unmanned driving manner, an image including the moving object captured by a camera may be input to a machine learning model to obtain at least one of a control signal that specifies the operation of the moving object and a parameter for generating the control signal. However, if the appearance state of the camera's imaging area changes from when the machine learning model was learned, the accuracy of the machine learning model may decrease. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, an information processing device is provided. The information processing device used to move a moving body that can move by unmanned driving includes: an accuracy acquisition unit that acquires the accuracy of a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body to move the moving body by unmanned driving and a parameter used to generate the control signal when an image acquired by imaging the moving body by an external camera installed at a location different from the moving body is input; a learning data acquisition unit that acquires a group of learning data used to train the machine learning model when the accuracy is less than a predetermined threshold; and a learning unit that executes a learning process to train the machine learning model using the acquired group of learning data, the group of learning data including one or more training data sets that correspond to a background image representing a target area including a passageway along which the moving body moves and a surrounding area of ​​the passageway, a moving body image representing the moving body, and a correct answer label. According to this aspect, the information processing device can train the machine learning model using the acquired training data set when the accuracy of the machine learning model is less than a threshold. In this way, it is possible to prevent a decrease in the accuracy of the machine learning model when the appearance state of the imaging area of ​​the external camera has changed since the time when the machine learning model was learned. (2) In the above-mentioned form, the machine learning model is trained in advance using an initial training data group, and the training data acquisition unit acquires an additional training data group as the training data group used for additional training of the machine learning model when the accuracy is less than the threshold when the captured images captured by some of the multiple external cameras are input to the machine learning model, and the learning unit executes an additional training process for additionally training the machine learning model using the acquired additional training data group, and the initial training data group may include a plurality of initial training data sets corresponding to each imaging area of ​​the multiple external cameras, the initial training data set including a training image including the background image and the moving object image corresponding to the imaging area, and a correct answer label, and the additional training data group may include one or more additional training data sets corresponding to the training image including the background image and the moving object image corresponding to the imaging area of ​​the external camera that acquired the captured image with the accuracy less than the threshold, and the correct answer label. According to this embodiment, a machine learning model is prepared by using a plurality of initial learning data sets corresponding to the imaging areas of the plurality of external cameras, in which a training image including a background image and a moving object image corresponding to the imaging area and a correct answer label are associated with each other. The information processing device can execute the following process when the accuracy is less than a threshold when an image captured by a part of the plurality of external cameras is input to the trained machine learning model. In this case, the information processing device can additionally train the machine learning model using an additional learning data set in which a training image including a background image and a moving object image corresponding to the imaging area of ​​the external camera that captured the image with the accuracy less than the threshold and a correct answer label are associated with each other. In this way, when the appearance state in the imaging area of ​​a part of the plurality of external cameras has changed since the time when the machine learning model was learned, it is possible to suppress a decrease in the accuracy of the machine learning model. (3) In the above aspect, the machine learning model is trained in advance using an initial training data group, and the training data acquisition unit acquires a new training data group as the training data group used to newly train a machine learning model different from the machine learning model when the accuracy is less than the threshold when the captured images captured by some of the multiple external cameras are input to the machine learning model, and the training unit executes a new training process to train the new machine learning model using the acquired new training data group, and the initial training data group may include a plurality of initial training data sets corresponding to each imaging area of ​​the multiple external cameras, the plurality of initial training data sets correlating a training image including the background image and the moving object image corresponding to the imaging area with a correct answer label, and the new training data group may include one or more new training data sets correlating the training image including the background image and the moving object image corresponding to the imaging area of ​​the external camera that acquired the captured image with the accuracy less than the threshold with the correct answer label. According to this embodiment, a machine learning model is prepared by using a plurality of initial learning data sets corresponding to the imaging areas of the plurality of external cameras, in which a training image including a background image and a moving object image corresponding to the imaging area and a correct answer label are associated with each other. When an image captured by a part of the plurality of external cameras is input to a trained machine learning model, the information processing device can execute the following process if the accuracy is less than a threshold. In this case, the information processing device can newly train the machine learning model using a new learning data set in which a training image including a background image and a moving object image corresponding to the imaging area of ​​the external camera that captured the image with the accuracy less than the threshold and a correct answer label are associated with each other. In this way, when the appearance state in the imaging area of ​​a part of the plurality of external cameras has changed since the time when the machine learning model was trained, it is possible to suppress a decrease in the accuracy of the machine learning model. (4) In the above embodiment, the training image may be at least one of the captured image and a composite image, and the composite image may be generated by combining a virtual moving body image, which is a virtual reproduction of the moving body, with either a real background image acquired by capturing an image of the target area with the external camera or a virtual background image, which is a virtual reproduction of the target area. According to this embodiment, the information processing device can use, as a training image used for learning a machine learning model, a composite image generated by combining a virtual moving body image with either a real background image or a virtual background image. (5) The above embodiment may further include an appearance acquisition unit that acquires appearance information indicating an appearance state of the moving body, and the learning unit may execute the learning process using the virtual moving body image that virtually reproduces the moving body in the appearance state specified by the appearance information. According to this embodiment, the information processing device may train a machine learning model using a training image including a virtual moving body image that virtually reproduces a moving body in the appearance state specified by the appearance information. In this way, when training the machine learning model, a feature amount according to the appearance state of the target moving body may be trained. This may improve the accuracy of the machine learning model. (6) In the above aspect, the learning unit may execute the learning process during a period in which control of the unmanned driving is being performed. According to this aspect, the information processing device can execute a learning process to train a machine learning model during a period in which the moving body is controlling the unmanned driving. (7) The above aspect may further include a control information acquisition unit that acquires at least one of control information of the control signal and the parameter by inputting the captured image to the machine learning model, and when the learning process is completed, the control information acquisition unit may acquire the control information by inputting the captured image to the machine learning model after the learning process. According to this aspect, when the learning process is completed, the information processing device can acquire the control information by inputting the captured image to the machine learning model after the learning process. This makes it possible to suppress a decrease in accuracy of the control information acquired using captured images acquired by some of the external cameras among the multiple external cameras. (8) The above embodiment further includes a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image to the machine learning model, and the accuracy acquisition unit may further acquire accuracy of the machine learning model after the learning process, and when the learning process is completed and the accuracy of the machine learning model after the learning process is equal to or greater than the threshold, the control information acquisition unit may acquire the control information by inputting the captured image to the machine learning model after the learning process. According to this embodiment, when the learning process is completed and the accuracy of the machine learning model after the learning process is equal to or greater than the threshold, the information processing device can input the captured image to the machine learning model after the learning process to acquire the control information. This makes it possible to further suppress a decrease in accuracy of the control information acquired using captured images acquired by some of the external cameras among the multiple external cameras. (9) In the above embodiment, the accuracy acquisition unit may acquire the accuracy of the machine learning model after the learning process using a plurality of captured images acquired by capturing an image of the moving object moving behind the moving object included in the training image by the external camera, and when a ratio of the number of captured images in which the accuracy of the machine learning model after the learning process is equal to or greater than the threshold to a total number of the captured images used to calculate the accuracy of the machine learning model after the learning process is equal to or greater than a reference value, the control information acquisition unit may acquire the control information by inputting the captured images to the machine learning model after the learning process. According to this embodiment, when a ratio of the number of captured images in which the accuracy of the machine learning model after the learning process is equal to or greater than a threshold to a total number of the captured images used to calculate the accuracy of the machine learning model after the learning process is equal to or greater than a reference value, the information processing device can input the captured images to the machine learning model after the learning process to acquire the control information. This can further suppress a decrease in accuracy of the control information acquired using captured images captured by some of the external cameras among the plurality of external cameras. (10) The above embodiment further includes a control information acquisition unit that acquires at least one of the control information of the control signal and the parameter by inputting the captured image to the machine learning model, and a transmission unit that transmits the control signal to the moving body, and the training image is the captured image acquired by capturing an image of the moving body that is performing the control of the unmanned driving with the external camera, and when acquiring the captured image to be used as the training image in the learning process, the transmission unit may transmit to the moving body the control signal different from the control signal acquired by the control information acquisition unit and the control signal generated from the parameter acquired by the control information acquisition unit. According to this embodiment, the information processing device can transmit to the moving body the control signal acquired using the machine learning model and the control signal different from the control signal generated from the parameter acquired using the machine learning model. This can reduce the load required for preparing training images and make it easier to generate desired training images. (11) According to a second aspect of the present disclosure, an information processing system is provided. The information processing system used to move a moving body capable of moving by unmanned driving includes the moving body, a plurality of external cameras installed in a location different from the moving body, and the information processing device described in the above aspect. According to this aspect, when the accuracy of the machine learning model is below a threshold, the information processing system can train the machine learning model using the acquired training dataset. In this way, when the appearance state of the imaging area of ​​the external camera changes from when the machine learning model was trained, it is possible to suppress a decrease in the accuracy of the machine learning model. (12) According to a third aspect of the present disclosure, an information processing method is provided. The information processing method used to move a moving body capable of moving by unmanned driving includes: an accuracy acquisition step of acquiring accuracy of a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body to move the moving body by unmanned driving and a parameter used to generate the control signal when an image acquired by imaging the moving body by an external camera installed at a location different from the moving body is input; a learning data acquisition step of acquiring a learning data group used for training the machine learning model when the accuracy is less than a predetermined threshold; and a learning step of executing a learning process for training the machine learning model using the acquired learning data group, the learning data group including one or more learning data sets in which a training image including a background image representing a target area including a passageway along which the moving body moves and a surrounding area of ​​the passageway, a moving body image representing the moving body, and a correct answer label are associated with each other. According to this aspect, when the accuracy of the machine learning model is less than a threshold, the machine learning model can be trained using the acquired learning data set. In this way, it is possible to prevent a decrease in the accuracy of the machine learning model when the appearance state of the imaging area of ​​the external camera has changed since the time when the machine learning model was learned. The present disclosure can be realized in various forms other than the above-mentioned information processing device, information processing system, and information processing method. For example, the present disclosure can be realized in the form of an information processing device, an information processing system, a manufacturing method for a moving body, an information processing device, an information processing system, a control method for a moving body, a computer program for realizing the control method, a non-transitory recording medium on which the computer program is recorded, etc. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a conceptual diagram showing a configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an information processing system according to a first embodiment. [Diagram 3] 4 is a flowchart showing a processing procedure for vehicle travel control in the first embodiment. [Figure 4] 4 is a flowchart showing a learning processing method in the first embodiment. [Diagram 5] 5 is a flowchart showing a method for acquiring shape data in the first embodiment. [Figure 6] 5 is a flowchart showing an example of a vehicle position acquisition method using shape data. [Figure 7] 7A to 7C are schematic diagrams showing examples of various images obtained when the method shown in FIG. 6 is executed. [Figure 8] FIG. 11 is a diagram for explaining details of a coordinate point calculation step. [Figure 9] FIG. 4 is a diagram for explaining a method of calculating a base coordinate point. [Figure 10] FIG. 1 is a diagram for explaining details of a position conversion process. [Figure 11] FIG. 2 for explaining the details of the position conversion process. [Figure 12] FIG. 11 is a block diagram showing the configuration of an information processing system according to a second embodiment. [Figure 13] 10 is a flowchart showing a learning processing method in the second embodiment. [Figure 14] 10 is a flowchart showing a method for acquiring shape data in the second embodiment. [Figure 15] FIG. 13 is a block diagram showing the configuration of an information processing system according to a third embodiment. [Figure 16] 13 is a flowchart showing a learning processing method in the third embodiment. [Figure 17] 13 is a flowchart showing a method for acquiring shape data in the third embodiment. [Figure 18] FIG. 13 is a block diagram showing the configuration of an information processing system according to a fourth embodiment. [Figure 19] FIG. 4 is a diagram for explaining rectangular coordinate values. [Figure 20] 13 is a flowchart showing a learning processing method in the fourth embodiment. [Figure 21] 11 is a flowchart showing a method for acquiring rectangular coordinate values. [Figure 22] FIG. 13 is a block diagram showing the configuration of an information processing system according to a fifth embodiment. [Diagram 23] FIG. 4 is a diagram for explaining three-dimensional coordinate values. [Figure 24] 13 is a flowchart showing a learning processing method in the fifth embodiment. [Diagram 25] 11 is a flowchart showing a method for acquiring three-dimensional coordinate values. [Figure 26] FIG. 13 is a block diagram showing the configuration of an information processing system according to a sixth embodiment. [Figure 27] 13 is a flowchart showing a learning processing method in the sixth embodiment. [Figure 28] 13 is a flowchart showing a method for acquiring vehicle position information using a fourth learning model. [Figure 29] FIG. 13 is a block diagram showing the configuration of an information processing system according to a seventh embodiment. [Diagram 30] 23 is a flowchart showing a learning processing method in the seventh embodiment. [Diagram 31] 13 is a flowchart showing a method of acquiring a driving control signal using a fifth learning model. [Diagram 32] FIG. 13 is a block diagram showing the configuration of an information processing system according to an eighth embodiment. [Diagram 33]A figure showing an example of a method for generating training images used for learning the sixth learning model. [Diagram 34] 23 is a flowchart showing a learning processing method in the eighth embodiment. [Diagram 35] 23 is a flowchart showing a method for acquiring shape data in the eighth embodiment. [Diagram 36] FIG. 13 is a block diagram showing the configuration of an information processing system according to a ninth embodiment. [Figure 37] 13 is a flowchart showing a learning processing method according to the ninth embodiment. [Figure 38] 13 is a flowchart showing a method for acquiring shape data in the ninth embodiment. [Figure 39] FIG. 23 is a block diagram showing the configuration of an information processing system according to a tenth embodiment. [Diagram 40] 23 is a flowchart showing a processing procedure for vehicle driving control in a tenth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: A-1. Information processing system configuration: 1 is a conceptual diagram showing the configuration of an information processing system 50 in the first embodiment. The information processing system 50 is a system for moving a moving body without the need for a driving operation by a passenger on the moving body. The information processing system 50 includes one or more moving bodies, a plurality of external cameras 300 installed at locations different from the moving bodies, and an information processing device 200.

[0009] In the present disclosure, the term "mobile body" refers to an object that can move, such as a vehicle 100 or an electric vertical take-off and landing aircraft (a so-called flying car). In the present embodiment, the mobile body is the vehicle 100. The vehicle 100 may be a vehicle 100 that runs on wheels or a vehicle 100 that runs on caterpillar tracks, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, or a construction vehicle. The vehicle 100 includes an electric vehicle (BEV: Battery Electric Vehicle), a gasoline vehicle, a hybrid vehicle, and a fuel cell vehicle. When the mobile body is other than the vehicle 100, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "running" can be appropriately replaced with "movement".

[0010] The vehicle 100 is configured to be capable of traveling by unmanned driving. "Unmanned driving" means driving without the driving operation of a passenger. Driving operation means at least one of the operations of "running", "turning" and "stopping" of the vehicle 100. Unmanned driving is realized by automatic or manual remote control using a device located outside the vehicle 100, or by autonomous control of the vehicle 100. The vehicle 100 traveling by unmanned driving may have a passenger who does not perform driving operation on board. The passenger who does not perform driving operation includes, for example, a person who simply sits in the seat of the vehicle 100, and a person who performs work other than driving operation, such as assembly, inspection, and operation of switches, while riding on the vehicle 100. Note that driving by a passenger who performs driving operation is sometimes called "manned driving".

[0011] In this specification, "remote control" includes "full remote control" in which all of the operations of the vehicle 100 are completely determined from outside the vehicle 100, and "partial remote control" in which some of the operations of the vehicle 100 are determined from outside the vehicle 100. In addition, "autonomous control" includes "full autonomous control" in which the vehicle 100 autonomously controls its own operations without receiving any information from devices external to the vehicle 100, and "partial autonomous control" in which the vehicle 100 autonomously controls its own operations using information received from devices external to the vehicle 100.

[0012] In this embodiment, the information processing system 50 is used in a factory FC where the vehicle 100 is manufactured. The reference coordinate system of the factory FC is a global coordinate system GC. That is, any position in the factory FC is expressed by X, Y, and Z coordinates in the global coordinate system GC. The factory FC includes a first location PL1 and a second location PL2. The first location PL1 and the second location PL2 are connected by an aisle TR along which the vehicle 100 can travel. In the factory FC, a plurality of external cameras 300 are installed along the aisle TR. The position of each external camera 300 in the factory FC is adjusted in advance. The vehicle 100 moves from the first location PL1 to the second location PL2 through the aisle TR by unmanned driving.

[0013] 2 is a block diagram showing the configuration of the information processing system 50. The vehicle 100 includes a vehicle control device 110 for controlling each part of the vehicle 100, an actuator group 120 including one or more actuators that are driven under the control of the vehicle control device 110, and a communication device 130 for communicating with an external device such as an information processing device 200 by wireless communication. The actuator group 120 includes an actuator of a driving device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100.

[0014] The vehicle control device 110 is configured by a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected via the internal bus 114 to enable bidirectional communication. The input / output interface 113 is connected to an actuator group 120 and a communication device 130. The processor 111 functions as a vehicle control unit 115 by executing a program PG1 stored in the memory 112.

[0015] The vehicle control unit 115 controls the actuator group 120 to drive the vehicle 100. The vehicle control unit 115 controls the actuator group 120 using a driving control signal received from the information processing device 200 to drive the vehicle 100. The driving control signal is a control signal for driving the vehicle 100. The driving control signal specifies the operation of the vehicle 100 to move the vehicle 100 by unmanned driving. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0016] The information processing device 200 is used to drive the vehicle 100 in an unmanned driving manner. The information processing device 200 is, for example, a server installed at a location different from the vehicle 100. The information processing device 200 learns a machine learning model DM, or acquires and uses a learned machine learning model DM to drive the vehicle 100 in an unmanned driving manner. When a captured image is input, the machine learning model DM outputs at least one of a driving control signal and a signal generation parameter. The captured image is acquired by capturing an image of an image area including the passage TR on which the vehicle 100 shown in FIG. 1 travels, the surrounding area SA of the passage TR, and the vehicle 100 traveling on the passage TR, using the external camera 300. The captured image is either an original image as a detection result output from the external camera 300, or a processed image obtained by processing the original image. The signal generation parameter is a parameter used when generating a driving control signal. The signal generation parameters are, for example, vehicle position information, coordinate values ​​used to acquire at least one of the position and orientation of the vehicle 100, or shape data used to acquire at least one of the position and orientation of the vehicle 100. The vehicle position information is position information that is the basis for generating a travel control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. The shape data is data that indicates the external shape of the vehicle 100. The shape data is generated by detecting the vehicle 100 from the captured image. The shape data is, for example, a mask image in which a mask area is added to the captured image by masking an area representing the vehicle 100 among the areas that constitute the captured image. Each area that constitutes the captured image is, for example, one pixel that constitutes the captured image.

[0017] As shown in FIG. 2, in this embodiment, a first learning model DM1 is used as a machine learning model DM to drive the vehicle 100 in an unmanned driving manner. When a captured image is input, the first learning model DM1 outputs shape data as a parameter for signal generation. Thus, the information processing device 200 inputs the captured image to the first learning model DM1 to acquire shape data. Then, the information processing device 200 acquires vehicle position information using the shape data. Then, the information processing device 200 generates a driving control signal using the vehicle position information and the like, and transmits the signal to the vehicle 100. As a result, the information processing device 200 drives the vehicle 100 by remote control.

[0018] Here, the appearance state in the imaging area of ​​some of the external cameras 300 may change from when the first learning model DM1 was learned. In this case, there is a risk that the detection accuracy of detecting the vehicle 100 from the captured images acquired by some of the external cameras 300 among the multiple external cameras 300 may decrease. Therefore, when the captured images acquired by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, the information processing device 200 executes the following process if the accuracy of the first learning model DM1 is less than a predetermined threshold. In this case, the information processing device 200 executes a re-learning process to re-learn the learned first learning model DM1 using a training image corresponding to the imaging area of ​​the external camera 300 that acquired the captured image whose accuracy is less than the threshold. Re-learning is an example of additional learning. Therefore, the re-learning process is an example of additional learning process.

[0019] The element affecting the change in the appearance state in the imaging area may be, for example, dirt occurring in the target area TA. As shown in FIG. 1, the target area TA is an area including the passage TR along which the vehicle 100 travels and the surrounding area SA of the passage TR. The element affecting the change in the appearance state in the imaging area may be unevenness occurring in the target area TA. The unevenness occurring in the target area TA may be, for example, a depression occurring in the passage TR due to the road surface Rs being scraped away due to an increase in the number of times the vehicle 100 travels. The unevenness occurring in the target area TA may be a protrusion occurring in the passage TR due to the road surface Rs rising. The element affecting the change in the appearance state in the imaging area may be an object other than the vehicle 100 that is movably arranged in the target area TA. The object movably arranged in the target area TA may be, for example, a manufacturing facility, a device such as a road cone, and a signboard, the position of which can be appropriately changed. The object movably arranged in the target area TA may be a flying object such as a fallen leaf. The element affecting the change in the appearance state in the imaging area may be the road surface color of the passage TR. The road surface color of the passage TR may be, for example, the color of a dividing line such as a white line, the paint color of the floor surface including the passage TR, or the color of the paved road surface Rs. Furthermore, the factor that affects the change in the appearance state in the imaging area may be the illuminance condition of the lighting that illuminates the passage TR. The illuminance condition changes due to, for example, the difference in illuminance that differs depending on the accumulated lighting time from the start of use of the lighting, the decrease in illuminance due to the interruption of the power supply due to a power outage, etc.

[0020] The information processing device 200 is configured by a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 to be able to communicate bidirectionally. A communication device 205 for communicating with various devices outside the information processing device 200 is connected to the input / output interface 203. The communication device 205 can communicate with the vehicle 100 by wireless communication, and can communicate with each external camera 300 by wired communication or wireless communication.

[0021] The memory 202 stores various information including a program PG2 that controls the operation of the information processing device 200, a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300. The memory 202 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0022] The first learning model DM1 is a trained machine learning model DM that utilizes artificial intelligence to detect the external shape of the vehicle 100 from the captured image and output shape data. The first learning model DM1 is trained in advance by supervised learning using a first initial learning data group. The first initial learning data group includes a plurality of first initial learning data sets corresponding to the respective image capture areas of the plurality of external cameras 300. In each of the plurality of first initial learning data sets, a training image including a background image and a vehicle image corresponding to the image capture area of ​​the external camera 300 is associated with a region correct answer label. The background image is an image that represents the target region TA. The background image includes the target region TA without including the vehicle 100. The vehicle image is an image that represents the vehicle 100. In this embodiment, the training image is an image acquired by capturing an image of the vehicle 100 that is performing unmanned driving control by the external camera 300. The region correct answer label indicates whether each region in the training image is an area indicating the vehicle 100 or an area indicating something other than the vehicle 100. The region correct label is a correct label according to the shape data output from the first learning model DM1. For example, a convolutional neural network (hereinafter, CNN) that realizes either semantic segmentation or instance segmentation can be used as the first learning model DM1. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the first learning model DM1 and the region correct label.

[0023] By executing program PG2 stored in memory 202, processor 201 functions as an accuracy acquisition unit 211, a learning data acquisition unit 212, a learning unit 213, a control information acquisition unit 214, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217.

[0024] The accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 that has been trained in advance using the first initial learning data group. The accuracy of the first learning model DM1 is acquired by comparing the output result of the first learning model DM1 with the area correct label. The accuracy of the first learning model DM1 is, for example, any one of the accuracy rate, the matching rate, the recall rate, and the F value for the first learning model DM1. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211 acquires the accuracy of each of the multiple first learning models DM1 corresponding to each external camera 300.

[0025] The learning data acquisition unit 212 executes the following process when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images captured by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning data acquisition unit 212 acquires a first additional learning data group used for re-learning the first learning model DM1 as additional learning. The first additional learning data group includes one or more first additional learning data sets in which a training image including a background image and a vehicle image corresponding to an image capture area of ​​the external camera 300 that captured the captured image whose accuracy is less than the threshold, and a region correct answer label are associated with each other. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212 acquires a first additional learning data group for re-learning the first learning model DM1 whose accuracy is less than the threshold.

[0026] The learning unit 213 executes the following process when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured image acquired by some of the external cameras 300 among the multiple external cameras 300 is input to the first learning model DM1. In this case, the learning unit 213 executes a re-learning process to re-learn the first learning model DM1. As the re-learning process, the learning unit 213 updates at least some of the trained parameters of the first learning model DM1. As the re-learning process, the learning unit 213 may execute a process to add a layer of CNN constituting the first learning model DM1. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Therefore, the learning unit 213 re-learns the first learning model DM1 corresponding to the external camera 300 that acquired the captured image whose accuracy is less than the threshold, using the acquired first additional learning data group. At this time, the learning unit 213 executes the re-learning process during the period when the unmanned driving control is being executed.

[0027] The control information acquisition unit 214 acquires at least one of control information of a driving control signal and a parameter for signal generation by inputting a captured image to the first learning model DM1. In this embodiment, the first learning model DM1 is prepared for each external camera 300. The first learning model DM1 is a model that outputs shape data as a parameter for signal generation. Therefore, the control information acquisition unit 214 acquires shape data by inputting a captured image to the first learning model DM1 corresponding to the external camera 300 that acquired the captured image. At this time, when the re-learning process is completed, the control information acquisition unit 214 acquires shape data by inputting a captured image to the first learning model DM1 after the re-learning process.

[0028] The position acquisition unit 215 acquires vehicle position information. In this embodiment, the position acquisition unit 215 acquires the vehicle position information by using shape data. In this embodiment, the position of the vehicle 100 is the position of a positioning point that is set in advance for a specific part of the vehicle 100. The orientation of the vehicle 100 is a direction represented by a vector that faces from the rear side to the front side of the vehicle 100 along the longitudinal axis that passes through the center of gravity of the vehicle 100.

[0029] The signal generating unit 216 generates a driving control signal for controlling the actuator group 120 of the vehicle 100. The transmitting unit 217 transmits the driving control signal to the vehicle 100. As a result, the information processing device 200 drives the vehicle 100 by remote control. The processor 201 of the information processing device 200 may generate and output not only the driving control signal but also a control signal for controlling, for example, various auxiliary devices provided in the vehicle 100 and actuators for operating various equipment such as wipers, power windows, and lamps. That is, the processor 201 of the information processing device 200 may operate such various equipment and various auxiliary devices by remote control.

[0030] A-2. Vehicle driving control method: Fig. 3 is a flowchart showing a processing procedure of driving control of the vehicle 100 in the first embodiment. The flow shown in Fig. 3 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts driving in an unmanned manner, for example.

[0031] In step S111, the position acquisition unit 215 acquires vehicle position information of the vehicle 100 using the detection result output from the external camera 300. Specifically, in step S111, the position acquisition unit 215 acquires the vehicle position information using a captured image acquired from the camera that is the external camera 300.

[0032] In detail, in step S111, the position acquisition unit 215 calculates the coordinates of the positioning point of the vehicle 100 in the coordinate system of the captured image, i.e., the local coordinate system, using shape data acquired by inputting the captured image to the first learning model DM1, and converts the calculated coordinates into coordinates in the global coordinate system GC, thereby acquiring the position of the vehicle 100. In addition, the position acquisition unit 215 acquires the orientation of the vehicle 100 by calculating based on the orientation of the movement vector of the vehicle 100 calculated from the positional change of the feature points of the vehicle 100 between frames of the captured image, using, for example, an optical flow method.

[0033] In step S112, the signal generating unit 216 determines a target position to which the vehicle 100 should next head. In this embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system GC. A reference route RR, which is a route along which the vehicle 100 should travel, is stored in advance in the memory 202 of the information processing device 200. The route is represented by nodes indicating a departure point, nodes indicating passing points, nodes indicating a destination, and links connecting the nodes. The signal generating unit 216 uses the vehicle position information and the reference route RR to determine a target position to which the vehicle 100 should next head. The signal generating unit 216 determines a target position on the reference route RR that is ahead of the current location of the vehicle 100.

[0034] In step S113, the signal generating unit 216 generates a driving control signal for driving the vehicle 100 toward the determined target position. The signal generating unit 216 calculates the driving speed of the vehicle 100 from the transition of the position of the vehicle 100, and compares the calculated driving speed with the target speed. When the driving speed is lower than the target speed as a whole, the signal generating unit 216 determines the acceleration so that the vehicle 100 accelerates, and when the driving speed is higher than the target speed, the signal generating unit 216 determines the acceleration so that the vehicle 100 decelerates. In addition, when the vehicle 100 is located on the reference route RR, the signal generating unit 216 determines the steering angle and acceleration so that the vehicle 100 does not deviate from the reference route RR, and when the vehicle 100 is not located on the reference route RR, in other words, when the vehicle 100 deviates from the reference route RR, the signal generating unit 216 determines the steering angle and acceleration so that the vehicle 100 returns to the reference route RR.

[0035] In step S114, the transmission unit 217 transmits the generated driving control signal to the vehicle 100. The processor 201 repeats, at a predetermined cycle, obtaining the position of the vehicle 100, determining the target position, generating the driving control signal, and transmitting the driving control signal.

[0036] In step S115, the vehicle control unit 115 of the vehicle 100 receives the traveling control signal transmitted from the information processing device 200. In step S116, the vehicle control unit 115 controls the actuator group 120 using the received traveling control signal, thereby causing the vehicle 100 to travel at the acceleration and steering angle represented in the traveling control signal. The processor 111 of the vehicle 100 repeats receiving the traveling control signal and controlling the actuator group 120 at a predetermined cycle. According to the information processing system 50 in this embodiment, the vehicle 100 can be caused to travel by remote control, and the vehicle 100 can be moved without using transportation equipment such as a crane or a conveyor.

[0037] A-3. Data processing method: Fig. 4 is a flowchart showing the learning processing method in the first embodiment. The flow shown in Fig. 4 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling in an unmanned driving mode, for example.

[0038] In the accuracy acquisition step, the accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 previously trained using the first initial learning data group (step S211). When the captured images acquired by some of the multiple external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold (step S212: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212 acquires a first additional learning data group (step S213). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213 executes a re-learning process to re-train the first learning model DM1 using the acquired first additional learning data group (step S214). On the other hand, when the captured images captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S212: No), the processor 201 ends the flow without executing the learning data acquisition process and the learning process.

[0039] Fig. 5 is a flowchart showing a method for acquiring shape data in the first embodiment. The flow shown in Fig. 5 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0040] In the control information acquisition step, the control information acquisition unit 214 executes the following process to acquire shape data. Specifically, when the captured image acquired by some of the external cameras 300 among the multiple external cameras 300 is input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold (step S311: Yes), and the re-learning process is completed (step S312: Yes), the control information acquisition unit 214 acquires shape data by inputting the captured image to the first learning model DM1 after the re-learning process (step S313). On the other hand, when the captured image acquired by some of the external cameras 300 among the multiple external cameras 300 is input to the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S311: No), the control information acquisition unit 214 acquires shape data by inputting the captured image to the first learning model DM1 on which the re-learning process has not been executed (step S314).

[0041] A-4. Vehicle position acquisition method: FIG. 6 is a flowchart showing an example of a method for generating shape data and a method for acquiring the position of the vehicle 100 using the shape data. FIG. 7 is a schematic diagram showing examples of various images Im1 to Im6 when the method shown in FIG. 6 is executed. In FIG. 7, step numbers corresponding to the steps in FIG. 6 are assigned. In this embodiment, the positioning point 10e of the vehicle 100 is the rear end on the left side of the vehicle 100. Note that the positioning point 10e of the vehicle 100 may be other than the rear end on the left side of the vehicle 100.

[0042] In an image acquisition step (step S410), the control information acquisition unit 214 acquires an original image Im1 output from the external camera 300. In a distortion correction step (step S420), the control information acquisition unit 214 corrects the distortion of the original image Im1 to generate a corrected image Im2 as a processed image. In a rotation processing step (step S430), the control information acquisition unit 214 rotates the corrected image Im2. As a result, the control information acquisition unit 214 generates a rotated image Im3 as a processed image. In a crop processing step (step S440), the control information acquisition unit 214 deletes an unnecessary area A2 other than a necessary area A1 constituted by the vehicle 100 and the surrounding area of ​​the vehicle 100 from the rotated image Im3 among the areas constituting the rotated image Im3. As a result, the control information acquisition unit 214 generates a processed image Im4 as a processed image. The steps shown from step S420 to step S440 are pre-processing steps for improving the detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4. Therefore, at least any of the steps shown from step S420 to step S440 may be omitted.

[0043] In the detection step (step S450), the control information acquisition unit 214 inputs the processed image Im4 to the first learning model DM1. As a result, the control information acquisition unit 214 detects the vehicle 100 from the processed image Im4 and acquires a first mask image Im5 as shape data Da. The first mask image Im5 is an image in which a mask area Ms is added to the processed image Im4 by masking the area indicating the vehicle 100 among the areas constituting the processed image Im4.

[0044] In the perspective transformation step (step S460), the position acquisition unit 215 performs perspective transformation on the first mask image Im5 to generate a second mask image Im6. The position acquisition unit 215 performs perspective transformation on the first mask image Im5 into a bird's-eye image seen from a viewpoint above the vehicle 100 that is approximately perpendicular to the road surface Rs, for example, using a predetermined perspective transformation parameter. The perspective transformation parameter is, for example, a parameter related to the position information and internal parameters of the external camera 300 obtained by calibration. In this way, the position acquisition unit 215 generates a second mask image Im6 expressed in the image coordinate system from the first mask image Im5 expressed in the camera coordinate system. The camera coordinate system is a coordinate system having the focal point of the external camera 300 as the origin and coordinate axes represented by the Xc axis and the Yc axis perpendicular to the Xc axis. The image coordinate system is a coordinate system that has a point on the image plane as its origin and has coordinate axes indicated by the Xi axis and the Yi axis that is perpendicular to the Xi axis.

[0045] In the coordinate point calculation step (step S470), the position acquisition unit 215 calculates an image coordinate point indicating the position of the vehicle 100 in the image coordinate system. FIG. 8 is a diagram for explaining the details of the coordinate point calculation step. In order to calculate the image coordinate point, the position acquisition unit 215 first calculates a base coordinate point P0 from a first circumscribing rectangle R1 set in the mask area Ms in the first mask image Im5, which is an image before perspective transformation. FIG. 9 is a diagram for explaining a calculation method of the base coordinate point P0. In order to calculate the base coordinate point P0, the position acquisition unit 215 sets a base circumscribing rectangle R0 for the mask area Ms in the first mask image Im5. Next, the position acquisition unit 215 rotates the first mask image Im5 by a required rotation amount with the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 100 corresponding to the mask area Ms in the first mask image Im5 faces a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device that displays the first mask image Im5. Next, the position acquisition unit 215 sets a first circumscribing rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the movement vector V. Next, the position acquisition unit 215 rotates the first mask image Im5 to which the first circumscribing rectangle R1 has been added in the reverse direction by the above-mentioned rotation amount, with the center of gravity C of the mask area Ms as the rotation center. As a result, the position acquisition unit 215 sets the coordinate point of one of the four vertices of the first circumscribing rectangle R1, which has coordinates that are closest to the positioning point 10e of the vehicle 100, as the base coordinate point P0.

[0046] Next, the position acquisition unit 215 performs perspective transformation on the first mask image Im5 after reverse rotation, i.e., the first mask image Im5 after calculating the base coordinate point P0, as shown in Fig. 8. As a result, the position acquisition unit 215 sets the coordinate point corresponding to the base coordinate point P0 as the first coordinate point P1 in the first circumscribed rectangle R1 deformed by the perspective transformation.

[0047] Next, the position acquisition unit 215 sets a second circumscribing rectangle R2 for a mask region Ms in a second mask image Im6 obtained by performing perspective transformation on the first mask image Im5. Then, the position acquisition unit 215 sets, among the vertices of the second circumscribing rectangle R2, a vertex indicating the same position as the first coordinate point P1 as a second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation with each other.

[0048] Next, the position acquisition unit 215 performs correction to replace the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 depending on the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the position acquisition unit 215 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction (Yi1>Yi2), the position acquisition unit 215 replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. In the example shown in FIG. 8, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Also, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has coordinates (Xi2, Yi1). In this way, the position acquisition unit 215 calculates the image coordinate point P3 indicating the position of the vehicle 100 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2.

[0049] In the position conversion step (step S480), the position acquisition unit 215 converts the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point indicating the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC. The position acquisition unit 215 converts the image coordinate point P3 into the vehicle coordinate point using relational expressions (1) to (3) described later, which use the vehicle coordinate point as a response variable and include the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables. The imaging parameters are parameters related to the distance of the external camera 300 from a predetermined reference point. In this embodiment, the imaging parameter is the height H of the external camera 300 from the road surface Rs (FIG. 10 described later). The vehicle parameter is a parameter related to the distance of the positioning point 10e of the vehicle 100 from the reference point. In this embodiment, the vehicle parameter is the height h of the positioning point 10e of the vehicle 100 from the road surface Rs (FIG. 10 described later).

[0050] FIG. 10 is a first diagram for explaining the details of the position conversion process. FIG. 10 illustrates the state when the vehicle 100 is viewed from the left side. FIG. 11 is a second diagram for explaining the details of the position conversion process. FIG. 11 illustrates the state when the vehicle 100 is viewed from the roof side. The global coordinate system GC shown in FIG. 10 and FIG. 11 is a coordinate system having a fixed coordinate point Pf indicating an arbitrary reference position of the road surface Rs as the origin and coordinate axes indicated by the Xg axis and the Yg axis perpendicular to the Xg axis. The imaging coordinate point Pc is a coordinate point indicating the position in the global coordinate system GC of the external camera 300 that output the original image Im1 used in calculating the image coordinate point P3. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in advance in the memory 202 of the information processing device 200, for example.

[0051] As shown in Fig. 10, the observation distance on the XgYg plane between the position of the external camera 300 and the position of the vehicle 100 (image coordinate point P3) is set to Do. The observation error is set to ΔD. The height [m] of the external camera 300 from the road surface Rs, which is an imaging parameter, is set to H. The height [m] of the positioning point 10e of the vehicle 100 from the road surface Rs, which is a vehicle parameter, is set to h. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Equation (1) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0052] When the actual distance between the position of the external camera 300 and the position of the positioning point 10e of the vehicle 100 is defined as a first distance D, the first distance D can be expressed by the following formula (2). D=Do×(1-h / H) Equation (2) That is, the first distance D is determined by the observation distance Do, the height H of the external camera 300 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 100 as a vehicle parameter.

[0053] As shown in FIG. 11, if the distance between the reference position and the estimated position of the vehicle 100 is the estimated distance Dp, and the actual distance between the reference position and the vehicle 100 is the second distance Dt, the second distance Dt can be expressed by the following equation (3). Dt = Dp × (1-h / H) Equation (3)

[0054] Here, the estimated distance Dp can be calculated using the third distance Dc, which is the actual distance calculated from the fixed coordinate point Pf and the image coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the position acquisition unit 215 calculates the vehicle coordinate point Pv using the second distance Dt calculated by correcting the estimated distance Dp using the above formula (3) and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the positioning point 10e of the vehicle 100 in the global coordinate system GC, and therefore corresponds to the position of the vehicle 100 in the real space.

[0055] According to the first embodiment, a first learning model DM1 is prepared, which is trained in advance using a plurality of first initial learning data sets corresponding to the imaging regions of the plurality of external cameras 300, in which a training image including a background image and a vehicle image corresponding to the imaging region is associated with a region correct label. When the captured images Im1 to Im4 acquired by some of the plurality of external cameras 300 are input to the trained first learning model DM1, the information processing device 200 can execute the following process if the accuracy is less than a threshold. In this case, the information processing device 200 can retrain the first learning model DM1 using a first additional learning data set in which a training image including a background image and a vehicle image corresponding to the imaging region of the external camera 300 that acquired the captured images Im1 to Im4 whose accuracy is less than the threshold is associated with a region correct label. This makes it possible to suppress a decrease in detection accuracy when detecting the vehicle 100 from the captured images Im1-Im4 acquired by some of the multiple external cameras 300. Therefore, it is possible to suppress a decrease in accuracy of the first learning model DM1 when the appearance state in the captured area of ​​some of the multiple external cameras 300 has changed since the first learning model DM1 was learned.

[0056] According to the first embodiment, the information processing device 200 stores a plurality of first learning models DM1 prepared for each of the plurality of external cameras 300. In this way, when the captured images Im1 to Im4 captured by some of the plurality of external cameras 300 are input to the trained first learning model DM1, if the accuracy is less than the threshold, the information processing device 200 can execute the following process. In this case, the information processing device 200 can re-train only the first learning model DM1 corresponding to the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold. This can reduce the load required for re-training the first learning model DM1.

[0057] Furthermore, according to the first embodiment, when the re-learning process is completed, the information processing device 200 can acquire the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the re-learning process. This makes it possible to suppress a decrease in accuracy of the shape data Da generated using the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0058] According to the first embodiment, the information processing device 200 can re-learn the first learning model DM1 during the period when control of unmanned driving is being executed. Note that, in other embodiments, the information processing device 200 may re-learn the first learning model DM1 during the period when control of manned driving is being executed, for example, without executing control of unmanned driving.

[0059] According to the first embodiment, the vehicle 100 runs in an unmanned manner within the factory. In this way, the information processing device 200 can re-learn the first learning model DM1 during the process of manufacturing the vehicle 100.

[0060] B. Second embodiment: FIG. 12 is a block diagram showing a configuration of an information processing system 50a in the second embodiment. The information processing system 50a includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200a. In this embodiment, when captured images Im1 to Im4 acquired by some of the plurality of external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold, the information processing device 200a executes a transfer learning process for transferring the first learning model DM1 as additional learning. Transfer learning is an example of additional learning. Thus, the transfer learning process is an example of additional learning. The other configurations of the information processing system 50a are the same as those of the first embodiment unless otherwise described. The same reference numerals are attached to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0061] The information processing device 200 a is configured by a computer including a processor 201 a , a memory 202 a , an input / output interface 203 , and an internal bus 204 .

[0062] The memory 202a stores various information including a program PG2a that controls the operation of the information processing device 200a, a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300.

[0063] The processor 201a executes the program PG2a stored in the memory 202a to function as an accuracy acquisition unit 211, a learning data acquisition unit 212a, a learning unit 213a, a control information acquisition unit 214a, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217.

[0064] The learning data acquisition unit 212a executes the following process when the accuracy of the first learning model DM1 falls below a predetermined threshold when the captured images Im1-Im4 captured by some of the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning data acquisition unit 212a acquires a first additional learning data group used for transfer learning as additional learning of the first learning model DM1 whose accuracy falls below the threshold.

[0065] The learning unit 213a executes the following process when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning unit 213a executes a transfer learning process for transferring learning the first learning model DM1 whose accuracy is less than the threshold. The learning unit 213a executes, for example, the following process as the transfer learning process. Specifically, the learning unit 213a adds a layer of CNN constituting the first learning model DM1 whose accuracy is less than the threshold. Then, the learning unit 213a updates the parameters related to the added layer using the acquired first additional learning dataset without updating the trained parameters.

[0066] When the transfer learning process is completed, the control information acquisition unit 214a acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the transfer learning process.

[0067] Fig. 13 is a flowchart showing a learning processing method in the second embodiment. The flow shown in Fig. 13 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0068] In the accuracy acquisition step, the accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 previously trained using the first initial learning data group (step S221). When the captured images Im1 to Im4 acquired by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold (step S222: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212a acquires a first additional learning data group (step S223). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213a executes a transfer learning process to transfer learn the first learning model DM1 using the acquired first additional learning data group (step S224). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S222: No), the processor 201a ends the flow without executing the learning data acquisition process and the learning process.

[0069] Fig. 14 is a flowchart showing a method for acquiring shape data Da in the second embodiment. The flow shown in Fig. 14 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0070] In the control information acquisition step, the control information acquisition unit 214a executes the following process to acquire the shape data Da. Specifically, when the captured images Im1-Im4 captured by some of the multiple external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold (step S321: Yes) and the transfer learning process is completed (step S322: Yes), the control information acquisition unit 214a inputs the captured images Im1-Im4 to the first learning model DM1 after the transfer learning process to acquire the shape data Da (step S323). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S321: No), the control information acquisition unit 214a acquires shape data Da by inputting the captured images Im1 to Im4 into the first learning model DM1 on which transfer learning processing has not been performed (step S324).

[0071] According to the second embodiment, the information processing device 200a can execute the following process when the accuracy is less than a threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained first learning model DM1. In this case, the information processing device 200a can perform transfer learning of the first learning model DM1 using a first additional learning data set in which a training image including a background image corresponding to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a vehicle image, and an area correct label are associated with each other. In this way, it is possible to suppress a decrease in detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300. This makes it possible to suppress a decrease in accuracy of the shape data Da generated using the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0072] C. Third embodiment: FIG. 15 is a block diagram showing a configuration of an information processing system 50b in the third embodiment. The information processing system 50b includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200b. In this embodiment, when the re-learning process is completed and the accuracy of the first learning model DM1 after the re-learning process is equal to or higher than a threshold, the information processing device 200b inputs the captured images Im1 to Im4 into the first learning model DM1 after the re-learning process to acquire shape data Da. Unless otherwise described, the other configurations of the information processing system 50b are the same as those of the first embodiment. The same reference numerals are given to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0073] The information processing device 200 b is configured by a computer including a processor 201 b, a memory 202 b, an input / output interface 203, and an internal bus 204.

[0074] The memory 202b stores various information including a program PG2b that controls the operation of the information processing device 200b, a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300.

[0075] By executing program PG2b stored in memory 202b, processor 201b functions as an accuracy acquisition unit 211b, a learning data acquisition unit 212b, a learning unit 213b, a control information acquisition unit 214b, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217.

[0076] The accuracy acquisition unit 211b acquires the accuracy of the first learning model DM1 previously trained using the first initial learning data group. Furthermore, when the re-learning process is completed, the accuracy acquisition unit 211b acquires the accuracy of the first learning model DM1 after the re-learning process. Hereinafter, the accuracy of the first learning model DM1 previously trained using the first initial learning data group is also referred to as "initial accuracy". The accuracy of the first learning model DM1 after the re-learning process is also referred to as "post-learning accuracy". In this embodiment, the accuracy acquisition unit 211b acquires the post-learning accuracy using a plurality of captured images Im1 to Im4 acquired by capturing an image of a following vehicle 100 traveling behind the vehicle 100 in the training image included in the first initial learning data group by the external camera 300.

[0077] The learning data acquisition unit 212b acquires a first additional learning data group when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images Im1-Im4 acquired by some of the multiple external cameras 300 are input to the first learning model DM1. Furthermore, the learning data acquisition unit 212b reacquires the first additional learning data group when the re-learning process is completed and the post-learning accuracy is less than the threshold. At this time, the re-acquired first additional learning data group may include the same training images as the training images used in the re-learning process in which the post-learning accuracy became less than the threshold, or may include different training images.

[0078] The learning unit 213b executes a re-learning process to re-learn the first learning model DM1 when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images Im1-Im4 acquired by some of the multiple external cameras 300 are input to the first learning model DM1. Furthermore, when the re-learning process is completed and the post-learning accuracy is less than the threshold, the learning unit 213b executes a re-learning process to re-learn the first learning model DM1 using the reacquired first additional learning data group.

[0079] When the re-learning process is completed and the post-learning accuracy is equal to or greater than the threshold, the control information acquisition unit 214b inputs the captured images Im1-Im4 to the first learning model DM1 after the re-learning process to acquire shape data Da. At this time, when the ratio of the number of captured images Im1-Im4 whose post-learning accuracy is equal to or greater than the threshold to the total number of the captured images Im1-Im4 used to calculate the post-learning accuracy is equal to or greater than a predetermined reference value, the control information acquisition unit 214b inputs the captured images Im1-Im4 to the first learning model DM1 after the re-learning process to acquire shape data Da.

[0080] Fig. 16 is a flowchart showing a learning processing method in the third embodiment. The flow shown in Fig. 16 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling in an unmanned driving mode, for example.

[0081] In the accuracy acquisition step, the accuracy acquisition unit 211b acquires an initial accuracy (step S231). When the captured images Im1 to Im4 acquired by some of the multiple external cameras 300 are input to the first learning model DM1, if the initial accuracy is less than a predetermined threshold (step S232: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212a acquires a first additional learning data group (step S233). After the learning data acquisition step, the learning step is executed. In the first learning step, the learning unit 213a executes a re-learning process to re-learn the first learning model DM1 using the acquired first additional learning data group (step S234). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S232: No), the processor 201b ends the flow without executing the learning data acquisition process and the learning process.

[0082] When the re-learning process is completed (step S235: Yes), an accuracy confirmation step is executed. In the accuracy confirmation step, the accuracy acquisition unit 211b acquires the post-learning accuracy (step S236). When the ratio of the number of captured images Im1 to Im4 whose post-learning accuracy is equal to or greater than the threshold to the total number of captured images Im1 to Im4 used to calculate the post-learning accuracy is equal to or greater than a predetermined reference value (step S237: Yes), the flow ends. On the other hand, when the ratio of the number of captured images Im1 to Im4 whose post-learning accuracy is equal to or greater than the threshold to the total number of captured images Im1 to Im4 used to calculate the post-learning accuracy is less than the predetermined reference value (step S237: No), each step from step S233 to step S237 is repeatedly executed.

[0083] Fig. 17 is a flowchart showing a method for acquiring shape data Da in the third embodiment. The flow shown in Fig. 17 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0084] When the captured images Im1-Im4 captured by some of the external cameras 300 are input to the first learning model DM1, if the initial accuracy is less than a predetermined threshold (step S331: Yes), and the re-learning process is completed (step S332: Yes), the accuracy acquisition unit 211b acquires the post-learning accuracy (step S333). If the ratio of the number of captured images Im1-Im4 whose post-learning accuracy is equal to or greater than the threshold to the total number of captured images Im1-Im4 used to calculate the post-learning accuracy is equal to or greater than a predetermined reference value (step S334: Yes), the control information acquisition unit 214b acquires the shape data Da by inputting the captured images Im1-Im4 to the first learning model DM1 after the re-learning process (step S335). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the initial accuracy is greater than or equal to a predetermined threshold (step S331: No), the control information acquisition unit 214a acquires shape data Da by inputting the captured images Im1 to Im4 into the first learning model DM1 on which the re-learning process has not been performed (step S336).

[0085] According to the above third embodiment, when the re-learning process is completed and the post-learning accuracy is equal to or greater than a threshold, the information processing device 200b can input the captured images Im1 to Im4 to the first learning model DM1 after the re-learning process to obtain shape data Da.

[0086] Furthermore, according to the above third embodiment, the information processing device 200b can acquire post-learning accuracy using a plurality of captured images Im1 to Im4 acquired by capturing an image of a vehicle 100 traveling behind the vehicle 100 in a training image included in the first initial learning data group by an external camera 300.

[0087] Furthermore, according to the above third embodiment, the information processing device 200b can determine that the post-learning accuracy has reached a threshold value or greater when the ratio of the number of captured images Im1 to Im4 for which the post-learning accuracy has reached a threshold value or greater to the total number of captured images Im1 to Im4 used to calculate the post-learning accuracy is greater than or equal to a predetermined reference value.

[0088] D. Fourth embodiment: FIG. 18 is a block diagram showing a configuration of an information processing system 50c in the fourth embodiment. The information processing system 50c includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200c. In this embodiment, the information processing device 200c inputs captured images Im1 to Im4 into a second learning model DM2 as a machine learning model DM to obtain rectangular coordinate values ​​as parameters for signal generation. The other configurations of the information processing system 50c are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0089] 19 is a diagram for explaining the rectangular coordinate values ​​Cr1 to Cr4. The rectangular coordinate values ​​Cr1 to Cr4 are the coordinate values ​​of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc. The vehicle circumscribing rectangle Rc is a quadrangle that is set so as to surround the area occupied by the vehicle 100 when the vehicle 100 is projected onto the road surface Rs in the images Ti, Im1 to Im4. The rectangular coordinate values ​​Cr1 to Cr4 are used to calculate at least one of the position and orientation of the vehicle 100.

[0090] As shown in FIG. 18, the information processing device 200c is configured by a computer including a processor 201c, a memory 202c, an input / output interface 203, and an internal bus 204.

[0091] The memory 202c stores various information including a program PG2c that controls the operation of the information processing device 200c, a reference route RR, a plurality of second learning models DM2 prepared for each external camera 300, a rectangle database Db1, and camera parameters Pa.

[0092] The second learning model DM2 outputs rectangular coordinate values ​​Cr1 to Cr4 when the captured images Im1 to Im4 are input. The second learning model DM2 is previously trained by supervised learning using a second initial learning data group. The second initial learning data group includes a plurality of second initial learning data sets corresponding to the respective image capture areas of the plurality of external cameras 300. In each of the plurality of second initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the image capture area of ​​the external camera 300 is associated with a rectangular coordinate correct label. The rectangular coordinate correct label is a correct label indicating the coordinate values ​​Cr1 to Cr4 of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc set for the vehicle 100 included in the training image Ti. In the rectangular coordinate correct label, each of the coordinate values ​​Cr1 to Cr4 is associated with additional information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc the coordinate values ​​Cr1 to Cr4 are. In this embodiment, the rectangular coordinate values ​​Cr1 to Cr4 output from the second learning model DM2 are coordinate values ​​in the image coordinate system. Therefore, in the rectangular coordinate correct label, each of the coordinate values ​​Cr1 to Cr4 is represented in the image coordinate system.

[0093] The second learning model DM2 may be, for example, a CNN that realizes either semantic segmentation or instance segmentation. During learning of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result of the second learning model DM2 and the rectangular coordinate correct label.

[0094] The rectangle database Db1 is used to calculate the coordinate values ​​of the positioning point 10e of the vehicle 100 from the rectangle coordinate values ​​Cr1 to Cr4. The rectangle database Db1 is a database that indicates the relative positional relationship between each of the vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc and the positioning point 10e of the vehicle 100.

[0095] The camera parameters Pa are parameters related to the external camera 300. In this embodiment, the camera parameters Pa are used to convert the coordinate values ​​of the positioning point 10e of the vehicle 100 expressed in the image coordinate system into the global coordinate system GC. The camera parameters Pa are, for example, the installation position, installation attitude, and focal length of the external camera 300.

[0096] The processor 201c functions as an accuracy acquisition unit 211c, a learning data acquisition unit 212c, a learning unit 213c, a control information acquisition unit 214c, a position acquisition unit 215c, a signal generation unit 216, and a transmission unit 217 by executing the program PG2c stored in the memory 202c.

[0097] The accuracy acquisition unit 211c acquires the accuracy of the second learning model DM2 that has been trained in advance using the second initial training data group. In this embodiment, the second learning model DM2 is prepared for each external camera 300. Therefore, the accuracy acquisition unit 211c acquires the accuracy of each of the multiple second learning models DM2 corresponding to each external camera 300.

[0098] The learning data acquisition unit 212c executes the following process when the accuracy of the second learning model DM2 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the second learning model DM2. In this case, the learning data acquisition unit 212c acquires a second additional learning data group used for re-learning the second learning model DM2 as additional learning. The second additional learning data group includes one or more second additional learning data sets in which a training image Ti including a background image corresponding to an imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a vehicle image, and a rectangular coordinate correct label are associated with each other. In this embodiment, the second learning model DM2 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212c acquires a second additional learning data group for re-learning the second learning model DM2 whose accuracy is less than the threshold.

[0099] The learning unit 213c executes the following process when the accuracy of the second learning model DM2 is less than a predetermined threshold when the captured images Im1-Im4 captured by some of the multiple external cameras 300 are input to the second learning model DM2. In this case, the learning unit 213c executes a re-learning process to re-learn the second learning model DM2. In this embodiment, the second learning model DM2 is prepared for each external camera 300. Therefore, the learning unit 213c re-learns the second learning model DM2 corresponding to the external camera 300 that captured the captured images Im1-Im4 whose accuracy is less than the threshold, using the acquired second additional learning data group.

[0100] The control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2. In this embodiment, the second learning model DM2 is prepared for each external camera 300. Therefore, the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 corresponding to the external camera 300 that acquired the captured images Im1 to Im4. At this time, when the re-learning process is completed, the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 after the re-learning process.

[0101] The position acquisition unit 215c calculates the coordinate value of the positioning point 10e of the vehicle 100 from the rectangle coordinate values ​​Cr1 to Cr4 using the rectangle database Db1. Then, the position acquisition unit 215c converts the coordinate value of the positioning point 10e of the vehicle 100 expressed in the image coordinate system into a coordinate value in the global coordinate system GC using the camera parameter Pa. At this time, in order to acquire the camera parameter Pa, the position acquisition unit 215c may acquire a camera identifier for identifying the multiple external cameras 300 and indicating the external camera 300 that acquired the captured images Im1 to Im4.

[0102] The position acquisition unit 215c further calculates the orientation of the vehicle 100. The position acquisition unit 215c calculates the orientation of the vehicle 100 using, for example, the shape of the vehicle circumscribing rectangle Rc and supplementary information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc have coordinate values ​​Cr1 to Cr4.

[0103] Fig. 20 is a flowchart showing a learning processing method in the fourth embodiment. The flow shown in Fig. 20 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0104] In the accuracy acquisition step, the accuracy acquisition unit 211c acquires the accuracy of the second learning model DM2 previously trained using the second initial learning data group (step S241). When the captured images Im1 to Im4 acquired by some of the multiple external cameras 300 are input to the second learning model DM2, if the accuracy of the second learning model DM2 is less than a predetermined threshold (step S242: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212c acquires a second additional learning data group (step S243). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213c executes a re-learning process to re-train the second learning model DM2 using the acquired second additional learning data group (step S244). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the second learning model DM2, if the accuracy of the second learning model DM2 is equal to or greater than a predetermined threshold (step S242: No), the processor 201c ends the flow without executing the learning data acquisition process and the learning process.

[0105] Fig. 21 is a flowchart showing a method for acquiring the rectangular coordinate values ​​Cr1 to Cr4. The flow shown in Fig. 21 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0106] In the control information acquisition step, the control information acquisition unit 214c executes the following process to acquire the rectangular coordinate values ​​Cr1 to Cr4. Specifically, when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the second learning model DM2, if the accuracy of the second learning model DM2 is less than a predetermined threshold (step S341: Yes), and the re-learning process is completed (step S342: Yes), the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 after the re-learning process (step S343). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the second learning model DM2, if the accuracy of the second learning model DM2 is greater than or equal to a predetermined threshold (step S341: No), the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 into the second learning model DM2 on which the re-learning process has not been performed (step S344).

[0107] According to the fourth embodiment, the information processing device 200c can execute the following process when the accuracy is less than a threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained second learning model DM2. In this case, the information processing device 200c can retrain the second learning model DM2 using a second additional learning data set in which a training image Ti including a background image and a vehicle image according to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, and a rectangular coordinate correct label are associated with each other. In this way, when the appearance state in the imaging area of ​​some of the external cameras 300 among the multiple external cameras 300 has changed since the second learning model DM2 was trained, it is possible to suppress a decrease in the accuracy of the second learning model DM2.

[0108] Furthermore, according to the fourth embodiment, when the re-learning process is completed, the information processing device 200c can obtain the rectangular coordinate values ​​Cr1-Cr4 by inputting the captured images Im1-Im4 to the second learning model DM2 after the re-learning process. This makes it possible to prevent a decrease in accuracy of the rectangular coordinate values ​​Cr1-Cr4 calculated using the captured images Im1-Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0109] The rectangular coordinate values ​​Cr1 to Cr4 may be coordinate values ​​in the camera coordinate system or may be coordinate values ​​in the global coordinate system GC. When the rectangular coordinate values ​​Cr1 to Cr4 are coordinate values ​​expressed by either the camera coordinate system or the global coordinate system GC, the second learning model DM2 is learned, for example, as follows. In this case, the second learning model DM2 is learned by associating at least one of a camera identifier indicating the external camera 300 that acquired the training image Ti and a camera parameter Pa of the external camera 300 that acquired the training image Ti with the training image Ti. In this case, in the rectangular coordinate correct label, each coordinate value Cr1 to Cr4 is expressed, for example, by the same coordinate system as the coordinate system of the output rectangular coordinate values ​​Cr1 to Cr4.

[0110] Furthermore, the information processing device 200c may acquire the coordinate values ​​of the ground contact surface of the wheel as the signal generation parameters instead of the rectangular coordinate values ​​Cr1 to Cr4. Even in this form, the information processing device 200c can acquire the rectangular coordinate values ​​Cr1 to Cr4 and generate the driving control signal.

[0111] E. Fifth embodiment: FIG. 22 is a block diagram showing a configuration of an information processing system 50d in the fifth embodiment. The information processing system 50d includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200d. In this embodiment, the information processing device 200d inputs captured images Im1 to Im4 into a third learning model DM3 as a machine learning model DM to obtain three-dimensional coordinate values ​​as parameters for signal generation. The other configurations of the information processing system 50d are the same as those of the first embodiment unless otherwise described. The same reference numerals are attached to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0112] 23 is a diagram for explaining the three-dimensional coordinate values ​​Cb1 to Cb8. The three-dimensional coordinate values ​​Cb1 to Cb8 are the coordinate values ​​of eight vertices Vb1 to Vb8 of a vehicle circumscribing rectangular solid Rb. The vehicle circumscribing rectangular solid Rb is a rectangular solid set so as to surround the vehicle 100 in the images Ti, Im1 to Im4. The three-dimensional coordinate values ​​Cb1 to Cb8 are used to calculate at least one of the position and orientation of the vehicle 100.

[0113] As shown in FIG. 22, the information processing device 200d is configured by a computer including a processor 201d, a memory 202d, an input / output interface 203, and an internal bus 204.

[0114] The memory 202d stores various information including a program PG2d that controls the operation of the information processing device 200d, a reference path RR, a plurality of third learning models DM3 prepared for each external camera 300, a stereoscopic database Db2, and camera parameters Pa.

[0115] The third learning model DM3 outputs three-dimensional coordinate values ​​Cb1 to Cb8 when the captured images Im1 to Im4 are input. The third learning model DM3 is trained in advance by supervised learning using a third initial learning data group. The third initial learning data group includes a plurality of third initial learning data sets corresponding to the respective imaging areas of the plurality of external cameras 300. In each of the plurality of third initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the imaging area of ​​the external camera 300 is associated with a three-dimensional coordinate correct label. The three-dimensional coordinate correct label is a correct label indicating the coordinate values ​​Cb1 to Cb8 of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular solid Rb set for the vehicle 100 included in the training image Ti. In the stereoscopic coordinate answer label, each of the coordinate values ​​Cb1 to Cb8 is associated with additional information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb the coordinate values ​​Cb1 to Cb8 are. In this embodiment, the stereoscopic coordinate values ​​Cb1 to Cb8 output from the third learning model DM3 are coordinate values ​​in the camera coordinate system. Therefore, in the stereoscopic coordinate answer label, each of the coordinate values ​​Cb1 to Cb8 is expressed in the camera coordinate system. The third learning model DM3 is learned by associating at least one of the camera parameters Pa and the camera identifier used to acquire the camera parameters Pa with the training image Ti.

[0116] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the third learning model DM3. During learning of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the third learning model DM3 and the 3D coordinate correct label.

[0117] The three-dimensional database Db2 is used to calculate the coordinate values ​​of the positioning point 10e of the vehicle 100 from the three-dimensional coordinate values ​​Cb1 to Cr8. The three-dimensional database Db2 is a database that indicates the relative positional relationship between each of the vertices Vb1 to Vb8 of the vehicle circumscribed rectangular solid Rb and the positioning point 10e of the vehicle 100.

[0118] The processor 201d functions as an accuracy acquisition unit 211d, a learning data acquisition unit 212d, a learning unit 213d, a control information acquisition unit 214d, a position acquisition unit 215d, a signal generation unit 216, and a transmission unit 217 by executing the program PG2d stored in the memory 202d.

[0119] The accuracy acquisition unit 211d acquires the accuracy of the third learning model DM3 that has been trained in advance using the third initial training data group. In this embodiment, the third learning model DM3 is prepared for each external camera 300. Therefore, the accuracy acquisition unit 211d acquires the accuracy of each of the multiple third learning models DM3 corresponding to each external camera 300.

[0120] The learning data acquisition unit 212d executes the following process when the accuracy of the third learning model DM3 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the third learning model DM3. In this case, the learning data acquisition unit 212d acquires a third additional learning data group used for re-learning the third learning model DM3 as additional learning. The third additional learning data group includes one or more third additional learning data sets in which a training image Ti including a background image corresponding to an imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a vehicle image, and a stereoscopic coordinate correct label are associated with each other. In this embodiment, the third learning model DM3 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212d acquires a third additional learning data group for re-learning the third learning model DM3 whose accuracy is less than the threshold.

[0121] The learning unit 213d executes the following process when the accuracy of the third learning model DM3 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input to the third learning model DM3. In this case, the learning unit 213d executes a re-learning process to re-learn the third learning model DM3. In this embodiment, the third learning model DM3 is prepared for each external camera 300. Therefore, the learning unit 213d re-learns the third learning model DM3 corresponding to the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, using the acquired third additional learning data group.

[0122] The control information acquisition unit 214d acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3. In this embodiment, the third learning model DM3 is prepared for each external camera 300. Therefore, the control information acquisition unit 214d acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3 corresponding to the external camera 300 that acquired the captured images Im1 to Im4. At this time, when the re-learning process is completed, the control information acquisition unit 214d acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3 after the re-learning process.

[0123] The position acquisition unit 215d calculates the coordinate value of the positioning point 10e of the vehicle 100 from the three-dimensional coordinate values ​​Cb1 to Cb8 using the three-dimensional database Db2. Then, the position acquisition unit 215d converts the coordinate value of the positioning point 10e of the vehicle 100 expressed in the image coordinate system into a coordinate value in the camera coordinate system using the camera parameter Pa. At this time, the position acquisition unit 215d may acquire a camera identifier in order to acquire the camera parameter Pa.

[0124] The position acquisition unit 215d further calculates the orientation of the vehicle 100. The position acquisition unit 215d calculates the orientation of the vehicle 100 by using, for example, the shape of the vehicle circumscribed rectangular solid Rb and supplementary information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular solid Rb have coordinate values ​​Cb1 to Cb8.

[0125] Fig. 24 is a flowchart showing a learning processing method in the fifth embodiment. The flow shown in Fig. 24 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0126] In the accuracy acquisition step, the accuracy acquisition unit 211d acquires the accuracy of the third learning model DM3 previously trained using the third initial learning data group (step S251). When the captured images Im1 to Im4 acquired by some of the multiple external cameras 300 are input to the third learning model DM3, if the accuracy of the third learning model DM3 is less than a predetermined threshold (step S252: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212d acquires a third additional learning data group (step S253). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213d executes a re-learning process to re-train the third learning model DM3 using the acquired third additional learning data group (step S254). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the third learning model DM3, if the accuracy of the third learning model DM3 is greater than or equal to a predetermined threshold (step S252: No), the processor 201d ends the flow without executing the learning data acquisition process and the learning process.

[0127] Fig. 25 is a flowchart showing a method for acquiring the three-dimensional coordinate values ​​Cb1 to Cb8. The flow shown in Fig. 25 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0128] In the control information acquisition step, the control information acquisition unit 214d executes the following process to acquire the three-dimensional coordinate values ​​Cb1 to Cb8. Specifically, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input to the third learning model DM3, if the accuracy of the third learning model DM3 is less than a predetermined threshold (step S351: Yes), and the re-learning process is completed (step S352: Yes), the control information acquisition unit 214d inputs the captured images Im1 to Im4 to the third learning model DM3 after the re-learning process to acquire the three-dimensional coordinate values ​​Cb1 to Cb8 (step S353). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the third learning model DM3, if the accuracy of the third learning model DM3 is greater than or equal to a predetermined threshold (step S351: No), the control information acquisition unit 214d acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 into the third learning model DM3 on which the re-learning process has not been performed (step S354).

[0129] According to the fifth embodiment, the information processing device 200d can execute the following process when the accuracy is less than a threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained third learning model DM3. In this case, the information processing device 200d can retrain the third learning model DM3 using a third additional learning data set in which a training image Ti including a background image corresponding to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a vehicle image, and a stereoscopic coordinate correct label are associated with each other. In this way, when the appearance state in the imaging area of ​​some of the external cameras 300 among the multiple external cameras 300 has changed since the time when the third learning model DM3 was trained, it is possible to suppress a decrease in the accuracy of the third learning model DM3.

[0130] Furthermore, according to the fifth embodiment, when the information processing device 200d completes the re-learning process, it is possible to obtain the three-dimensional coordinate values ​​Cb1-Cb8 by inputting the captured images Im1-Im4 to the third learning model DM3 after the re-learning process. This makes it possible to prevent a decrease in accuracy of the three-dimensional coordinate values ​​Cb1-Cb8 calculated using the captured images Im1-Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0131] The stereoscopic coordinate values ​​Cb1 to Cb8 may be coordinate values ​​in an image coordinate system or may be coordinate values ​​in a global coordinate system GC.

[0132] F. Sixth embodiment: FIG. 26 is a block diagram showing a configuration of an information processing system 50e in the sixth embodiment. The information processing system 50e includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200e. In this embodiment, the information processing device 200e inputs captured images Im1 to Im4 to a fourth learning model DM4 as a machine learning model DM, thereby acquiring vehicle position information as a parameter for signal generation. The other configurations of the information processing system 50d are the same as those of the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0133] The information processing device 200 e is configured by a computer including a processor 201 e, a memory 202 e, an input / output interface 203, and an internal bus 204.

[0134] The memory 202e stores various information including a program PG2e for controlling the operation of the information processing device 200e, a reference route RR, a plurality of fourth learning models DM4 prepared for each external camera 300, and camera parameters Pa. In this embodiment, a program for generating a driving control signal is prepared for each external camera 300. Thus, a reference route RR is prepared for each external camera 300. A target position determined in the process of generating a driving control signal is represented by X, Y, and Z coordinates in the camera coordinate system.

[0135] The fourth learning model DM4 outputs vehicle position information when the captured images Im1 to Im4 are input. In this embodiment, the fourth learning model DM4 outputs the coordinate value of the positioning point 10e of the vehicle 100 and the orientation of the vehicle 100 when the captured images Im1 to Im4 are input. The fourth learning model DM4 is trained in advance by supervised learning using a fourth initial learning data group. The fourth initial learning data group includes a plurality of fourth initial learning data sets corresponding to the respective imaging areas of the plurality of external cameras 300. In each of the plurality of fourth initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the imaging area of ​​the external camera 300, a position coordinate correct answer label, and an orientation correct answer label are associated with each other. The orientation correct answer label is a correct answer label indicating the orientation of the vehicle 100 included in the training image Ti. The position coordinate correct answer label is a correct answer label indicating the coordinate value of the positioning point 10e of the vehicle 100 included in the training image Ti. In this embodiment, the coordinate value of the positioning point 10e of the vehicle 100 output from the fourth learning model DM4 is a coordinate value in the camera coordinate system. Therefore, in the position coordinate answer label, the coordinate value of the positioning point 10e of the vehicle 100 is expressed in the camera coordinate system. The fourth learning model DM4 is learned by associating at least one of the camera parameter Pa and the camera identifier used to acquire the camera parameter Pa with the training image Ti.

[0136] For the fourth learning model DM4, for example, a CNN that realizes either semantic segmentation or instance segmentation can be used. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result of the fourth learning model DM4 and the position coordinate correct label and the orientation correct label.

[0137] The processor 201e functions as an accuracy acquisition unit 211e, a learning data acquisition unit 212e, a learning unit 213e, a control information acquisition unit 214e, a signal generation unit 216, and a transmission unit 217 by executing a program PG2e stored in the memory 202e.

[0138] The accuracy acquisition unit 211e acquires the accuracy of the fourth learning model DM4 that has been trained in advance using the fourth initial training data group. In this embodiment, the fourth learning model DM4 is prepared for each external camera 300. Therefore, the accuracy acquisition unit 211e acquires the accuracy of each of the multiple fourth learning models DM4 corresponding to each external camera 300.

[0139] The learning data acquisition unit 212e executes the following process when the accuracy of the fourth learning model DM4 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the fourth learning model DM4. In this case, the learning data acquisition unit 212e acquires a fourth additional learning data group used for re-learning the fourth learning model DM4 as additional learning. The fourth additional learning data group includes one or more fourth additional learning data sets in which a training image Ti including a background image corresponding to an imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a vehicle image, a position coordinate correct answer label, and a direction correct answer label are associated with each other. In this embodiment, the fourth learning model DM4 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212e acquires a fourth additional learning data group for re-learning the fourth learning model DM4 whose accuracy is less than the threshold.

[0140] The learning unit 213e executes the following process when the accuracy of the fourth learning model DM4 is less than a predetermined threshold when the captured images Im1-Im4 captured by some of the multiple external cameras 300 are input to the fourth learning model DM4. In this case, the learning unit 213e executes a re-learning process to re-learn the fourth learning model DM4. In this embodiment, the fourth learning model DM4 is prepared for each external camera 300. Therefore, the learning unit 213e re-learns the fourth learning model DM4 corresponding to the external camera 300 that captured the captured images Im1-Im4 whose accuracy is less than the threshold, using the acquired fourth additional learning data group.

[0141] The control information acquisition unit 214e acquires vehicle position information by inputting the captured images Im1-Im4 to the fourth learning model DM4. In this embodiment, a fourth learning model DM4 is prepared for each external camera 300. Therefore, the control information acquisition unit 214e acquires vehicle position information by inputting the captured images Im1-Im4 to the fourth learning model DM4 corresponding to the external camera 300 that acquired the captured images Im1-Im4. At this time, when the re-learning process is completed, the control information acquisition unit 214e acquires vehicle position information by inputting the captured images Im1-Im4 to the fourth learning model DM4 after the re-learning process.

[0142] Fig. 27 is a flowchart showing a learning processing method in the sixth embodiment. The flow shown in Fig. 27 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0143] In the accuracy acquisition step, the accuracy acquisition unit 211e acquires the accuracy of the fourth learning model DM4 previously trained using the fourth initial learning data group (step S261). When the captured images Im1 to Im4 acquired by some of the multiple external cameras 300 are input to the fourth learning model DM4, if the accuracy of the fourth learning model DM4 is less than a predetermined threshold (step S262: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212e acquires a fourth additional learning data group (step S263). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213e executes a re-learning process to re-train the fourth learning model DM4 using the acquired fourth additional learning data group (step S264). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the fourth learning model DM4, if the accuracy of the fourth learning model DM4 is greater than or equal to a predetermined threshold (step S262: No), the processor 201e ends the flow without executing the learning data acquisition process and the learning process.

[0144] Fig. 28 is a flowchart showing a method for acquiring vehicle position information using the fourth learning model DM4. The flow shown in Fig. 28 is executed repeatedly at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0145] In the control information acquisition step, the control information acquisition unit 214e executes the following process to acquire vehicle position information. Specifically, when the captured images Im1-Im4 captured by some of the multiple external cameras 300 are input to the fourth learning model DM4, if the accuracy of the fourth learning model DM4 is less than a predetermined threshold (step S361: Yes), and the re-learning process is completed (step S362: Yes), the control information acquisition unit 214e inputs the captured images Im1-Im4 to the fourth learning model DM4 after the re-learning process to acquire vehicle position information (step S363). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the fourth learning model DM4, if the accuracy of the fourth learning model DM4 is equal to or greater than a predetermined threshold (step S361: No), the control information acquisition unit 214e acquires vehicle position information by inputting the captured images Im1 to Im4 into the fourth learning model DM4 on which the re-learning process has not been performed (step S364).

[0146] According to the sixth embodiment, the information processing device 200e can execute the following process when the accuracy is less than a threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained fourth learning model DM4. In this case, the information processing device 200e can retrain the fourth learning model DM4 using a fourth additional learning data set in which a training image Ti including a background image and a vehicle image according to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a position coordinate correct answer label, and a direction correct answer label are associated with each other. In this way, when the appearance state in the imaging area of ​​some of the external cameras 300 among the multiple external cameras 300 has changed since the fourth learning model DM4 was trained, it is possible to suppress a decrease in the accuracy of the fourth learning model DM4.

[0147] Furthermore, according to the sixth embodiment, when the re-learning process is completed, the information processing device 200e can acquire vehicle position information by inputting the captured images Im1-Im4 to the fourth learning model DM4 after the re-learning process. This makes it possible to suppress a decrease in the accuracy of the vehicle position information calculated using the captured images Im1-Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0148] G. Seventh embodiment: FIG. 29 is a block diagram showing a configuration of an information processing system 50f in the seventh embodiment. The information processing system 50f includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200f. In this embodiment, the information processing device 200f acquires a driving control signal by inputting captured images Im1 to Im4 and a camera identifier to a fifth learning model DM5 as a machine learning model DM. The other configurations of the information processing system 50f are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0149] The information processing device 200 f is configured by a computer including a processor 201 f, a memory 202 f, an input / output interface 203 , and an internal bus 204 .

[0150] The memory 202f stores various information including a program PG2f that controls the operation of the information processing device 200f, a reference route RR, and a plurality of fifth learning models DM5 prepared for each external camera 300.

[0151] The fifth learning model DM5 outputs a driving control signal when the captured images Im1 to Im4 and a camera identifier indicating the external camera 300 that captured the captured images Im1 to Im4 are input. The fifth learning model DM5 is pre-trained by supervised learning using a fifth initial learning data group. The fifth initial learning data group includes a plurality of fifth initial learning data sets corresponding to the respective imaging areas of the plurality of external cameras 300. In each of the plurality of fifth initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the imaging area of ​​the external camera 300 is associated with a signal correct answer label. The signal correct answer label is a correct answer label indicating a target value of each parameter included in the driving control signal. The signal correct answer label is generated based on a reference route RR and a target speed in the imaging area of ​​the external camera 300 that captured the training image Ti.

[0152] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the fifth learning model DM5. During learning of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result of the fifth learning model DM5 and the signal correct answer label.

[0153] The processor 201f functions as an accuracy acquisition unit 211f, a learning data acquisition unit 212f, a learning unit 213f, a control information acquisition unit 214f, and a transmission unit 217 by executing a program PG2f stored in the memory 202f.

[0154] The accuracy acquisition unit 211f acquires the accuracy of the fifth learning model DM5 that has been trained in advance using the fifth initial training data group. In this embodiment, the fifth learning model DM5 is prepared for each external camera 300. Therefore, the accuracy acquisition unit 211f acquires the accuracy of each of the multiple fifth learning models DM5 corresponding to each external camera 300.

[0155] The learning data acquisition unit 212f executes the following process when the accuracy of the fifth learning model DM5 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the fifth learning model DM5. In this case, the learning data acquisition unit 212f acquires a fifth additional learning data group used for re-learning the fifth learning model DM5 as additional learning. The fifth additional learning data group includes one or more fifth additional learning data sets in which a training image Ti including a background image corresponding to an imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a vehicle image, and a signal correct answer label are associated with each other. In this embodiment, the fifth learning model DM5 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212f acquires a fifth additional learning data group for re-learning the fifth learning model DM5 whose accuracy is less than the threshold.

[0156] The learning unit 213f executes the following process when the accuracy of the fifth learning model DM5 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input to the fifth learning model DM5. In this case, the learning unit 213f executes a re-learning process to re-learn the fifth learning model DM5. In this embodiment, the fifth learning model DM5 is prepared for each external camera 300. Therefore, the learning unit 213f re-learns the fifth learning model DM5 corresponding to the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, using the acquired fifth additional learning data group.

[0157] The control information acquisition unit 214f ​​acquires a driving control signal by inputting the captured images Im1-Im4 to the fifth learning model DM5. In this embodiment, the fifth learning model DM5 is prepared for each external camera 300. Therefore, the control information acquisition unit 214f ​​acquires a driving control signal by inputting the captured images Im1-Im4 to the fifth learning model DM5 corresponding to the external camera 300 that acquired the captured images Im1-Im4. At this time, when the re-learning process is completed, the control information acquisition unit 214f ​​acquires a driving control signal by inputting the captured images Im1-Im4 to the fifth learning model DM5 after the re-learning process.

[0158] Fig. 30 is a flowchart showing a learning processing method in the seventh embodiment. The flow shown in Fig. 30 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0159] In the accuracy acquisition step, the accuracy acquisition unit 211f acquires the accuracy of the fifth learning model DM5 previously trained using the fifth initial learning data group (step S271). When the captured images Im1 to Im4 acquired by some of the external cameras 300 among the multiple external cameras 300 are input to the fifth learning model DM5, if the accuracy of the fifth learning model DM5 is less than a predetermined threshold (step S272: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212f acquires a fifth additional learning data group (step S273). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213f executes a re-learning process to re-learn the fifth learning model DM5 using the acquired fifth additional learning data group (step S274). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the fifth learning model DM5, if the accuracy of the fifth learning model DM5 is greater than or equal to a predetermined threshold (step S272: No), the processor 201f ends the flow without executing the learning data acquisition process and the learning process.

[0160] Fig. 31 is a flowchart showing a method for acquiring a driving control signal using the fifth learning model DM5. The flow shown in Fig. 31 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts driving in an unmanned driving state, for example.

[0161] In the control information acquisition step, the control information acquisition unit 214f ​​executes the following process to acquire a driving control signal. Specifically, when the captured images Im1-Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the fifth learning model DM5, if the accuracy of the fifth learning model DM5 is less than a predetermined threshold (step S371: Yes), and the re-learning process is completed (step S372: Yes), the control information acquisition unit 214f ​​acquires a driving control signal by inputting the captured images Im1-Im4 to the fifth learning model DM5 after the re-learning process (step S373). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the fifth learning model DM5, if the accuracy of the fifth learning model DM5 is greater than or equal to a predetermined threshold (step S371: No), the control information acquisition unit 214f ​​acquires a driving control signal by inputting the captured images Im1 to Im4 into the fifth learning model DM5 on which the re-learning process has not been performed (step S374).

[0162] According to the seventh embodiment, the information processing device 200f can execute the following process when the accuracy is less than a threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained fifth learning model DM5. In this case, the information processing device 200f can retrain the fifth learning model DM5 using a fifth additional learning data set in which a training image Ti including a background image and a vehicle image according to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold is associated with a signal correct answer label. In this way, when the appearance state in the imaging area of ​​some of the external cameras 300 among the multiple external cameras 300 has changed since the fifth learning model DM5 was learned, it is possible to suppress a decrease in the accuracy of the fifth learning model DM5.

[0163] Furthermore, according to the seventh embodiment, when the re-learning process is completed, the information processing device 200f can acquire the driving control signal by inputting the captured images Im1-Im4 to the fifth learning model DM5 after the re-learning process. This makes it possible to suppress a decrease in accuracy of the driving control signal generated using the captured images Im1-Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0164] The information input to the fifth learning model DM5 is not limited to the above. The information processing device 200f may acquire a traveling control signal by inputting other information, such as speed information indicating the traveling speed of the vehicle 100, route information indicating the reference route RR of the vehicle 100, and a pinion angle, in addition to the captured images Im1 to Im4 and the camera identifier, to the fifth learning model DM5. In this manner, the information processing device 200f can acquire a more appropriate traveling control signal according to the traveling situation of the vehicle 100 and the reference route RR.

[0165] H. Eighth embodiment: FIG. 32 is a block diagram showing a configuration of an information processing system 50g in the eighth embodiment. The information processing system 50f includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200g. In this embodiment, the information processing device 200g acquires shape data Da by inputting captured images Im1 to Im4 to a sixth learning model DM6 as a machine learning model DM. The other configurations of the information processing system 50g are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0166] FIG. 33 is a diagram showing an example of a method for generating a training image Ti used in learning the sixth learning model DM6. In this embodiment, the training image Ti is a synthetic image Ci that reproduces a real space. The synthetic image Ci is a two-dimensional image generated by synthesizing a virtual vehicle image Vi with one of the background images Bi, which are a real background image Br and a virtual background image Bv. The real background image Br is a background image Bi that is acquired by actually capturing an image of the target area TA using the external camera 300. The virtual background image Bv is a background image Bi that virtually reproduces the target area TA. The virtual background image Bv is, for example, either two-dimensional CAD data or three-dimensional CAD data that represents the target area TA. The virtual vehicle image Vi is an image that virtually reproduces the vehicle 100. The virtual vehicle image Vi is either two-dimensional CAD data or three-dimensional CAD data that represents the vehicle 100.

[0167] In this embodiment, the composite image Ci is generated by compositing the virtual vehicle image Vi with the virtual background image Bv. Specifically, the composite image Ci is generated by, for example, superimposing the virtual vehicle image Vi on each target area TA of the production line of the factory FC reproduced in the virtual space. That is, in this embodiment, the training image Ti is generated by utilizing the digital twin.

[0168] In this embodiment, the composite image Ci is generated using a virtual vehicle image Vi that virtually reproduces the vehicle 100 according to the exterior state of the vehicle 100 to be detected. The exterior state of the vehicle 100 is classified, for example, by the body type of the vehicle 100, the exterior color of the vehicle 100, and the product name of the vehicle 100. The body type of the vehicle 100 is a group of vehicle types when a plurality of types of the vehicle 100 are classified according to the exterior shape of the vehicle 100. When the exterior state of the vehicle 100 is classified according to the body type of the vehicle 100, the vehicle 100 is classified into one body type according to the vehicle rank determined by the overall length, width, and height of the vehicle 100 and the exterior shape of the vehicle 100. The vehicle rank is also called the vehicle body or body size. In this embodiment, the body type is, for example, "SUV", "sedan", "station wagon", "minivan", "minibox", "compact car", and "light car". The external appearance of the vehicle 100 may be classified by a predetermined classification number appropriately given according to the model, specifications, and the like.

[0169] When the virtual background image Bv is three-dimensional data, the virtual background image Bv is converted from three-dimensional data to two-dimensional data in the process of generating the composite image Ci. When the virtual vehicle image Vi is three-dimensional data, the virtual vehicle image Vi is converted from three-dimensional data to two-dimensional data in the process of generating the composite image Ci. In addition, in the process of combining the virtual vehicle image Vi with the background image Bi, the position of the virtual vehicle image Vi with respect to the background image Bi may be adjusted, or the pixel values ​​of each of the images Bi, Ci, and Vi may be adjusted.

[0170] The information processing device 200 g is configured by a computer including a processor 201 g, a memory 202 g, an input / output interface 203, and an internal bus 204.

[0171] The memory 202g stores various information including a program PG2g that controls the operation of the information processing device 200g, a reference route RR, and a plurality of sixth learning models DM6 prepared for each external camera 300.

[0172] The sixth learning model DM6 outputs shape data Da when the captured images Im1 to Im4 are input. The sixth learning model DM6 is trained in advance by supervised learning using a sixth initial learning data group. The sixth initial learning data group includes a plurality of sixth initial learning data sets corresponding to the respective captured areas of the plurality of external cameras 300. In each of the sixth initial learning data sets, a training image Ti including a virtual background image Bv corresponding to the captured area of ​​the external camera 300 and a virtual vehicle image Vi virtually reproducing the vehicle 100 corresponding to the external appearance state of the vehicle 100 to be detected is associated with a region correct label Lr.

[0173] For the sixth learning model DM6, for example, a CNN that realizes either semantic segmentation or instance segmentation can be used. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the sixth learning model DM6 and the region correct label Lr.

[0174] By executing the program PG2g stored in the memory 202g, the processor 201g functions as an accuracy acquisition unit 211g, an appearance acquisition unit 218, a learning data acquisition unit 212g, a learning unit 213g, a control information acquisition unit 214g, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217.

[0175] The accuracy acquisition unit 211g acquires the accuracy of the sixth learning model DM6 that has been trained in advance using the sixth initial training data group. In this embodiment, the sixth learning model DM6 is prepared for each external camera 300. Therefore, the accuracy acquisition unit 211g acquires the accuracy of each of the multiple sixth learning models DM6 corresponding to each external camera 300.

[0176] The appearance acquisition unit 218 acquires appearance information. The appearance information is information indicating the appearance state of the vehicle 100 to be detected. The appearance information is generated by, for example, using management information to identify the appearance state of the vehicle 100 to be detected. The management information is information indicating the running order of the multiple vehicles 100 running in the imaging area of ​​the external camera 300. In the management information, a vehicle identifier and a camera identifier are associated with each other. The vehicle identifier is a unique identifier assigned to the vehicles 100 so as not to overlap with each other in order to identify the multiple vehicles 100. The management information is generated using, for example, vehicle position information, a transmission history of a running control signal to the vehicle 100, and an installation location of the external camera 300.

[0177] The learning data acquisition unit 212g executes the following process when the accuracy of the sixth learning model DM6 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the sixth learning model DM6. In this case, the learning data acquisition unit 212g acquires a sixth additional learning data group used for re-learning the sixth learning model DM6 as additional learning. The sixth additional learning data group includes one or more sixth additional learning data sets in which training images Ti including a virtual background image Bv corresponding to the captured area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, and a virtual vehicle image Vi that virtually reproduces the vehicle 100 in an appearance state specified by the appearance information, are associated with an area correct label Lr. In this embodiment, the sixth learning model DM6 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212g acquires a sixth group of additional learning data for re-learning the sixth learning model DM6 whose accuracy has fallen below the threshold.

[0178] The learning unit 213g executes the following process when the accuracy of the sixth learning model DM6 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the sixth learning model DM6. In this case, the learning unit 213g executes a re-learning process to re-learn the sixth learning model DM6. In this embodiment, the sixth learning model DM6 is prepared for each external camera 300. Therefore, the learning unit 213g re-learns the sixth learning model DM6 corresponding to the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, using the acquired sixth additional learning data group. In other words, the learning unit 213g re-learns the sixth learning model DM6 using a virtual vehicle image Vi that virtually reproduces the vehicle 100 in an appearance state specified by the appearance information.

[0179] The control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6. In this embodiment, the sixth learning model DM6 is prepared for each external camera 300. Therefore, the control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6 corresponding to the external camera 300 that acquired the captured images Im1 to Im4. At this time, when the re-learning process is completed, the control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6 after the re-learning process.

[0180] Fig. 34 is a flowchart showing a learning processing method in the eighth embodiment. The flow shown in Fig. 34 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0181] In the accuracy acquisition step, the accuracy acquisition unit 211g acquires the accuracy of the sixth learning model DM6 previously trained using the sixth initial learning data group (step S281). When the captured images Im1 to Im4 acquired by some of the external cameras 300 among the multiple external cameras 300 are input to the sixth learning model DM6, if the accuracy of the sixth learning model DM6 is less than a predetermined threshold (step S282: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212g acquires a sixth additional learning data group (step S283). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213g executes a re-learning process to re-learn the sixth learning model DM6 using the acquired sixth additional learning data group (step S284). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the sixth learning model DM6, if the accuracy of the sixth learning model DM6 is greater than or equal to a predetermined threshold (step S282: No), the processor 201g ends the flow without executing the learning data acquisition process and the learning process.

[0182] Fig. 35 is a flowchart showing a method for acquiring shape data Da in the eighth embodiment. The flow shown in Fig. 35 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0183] In the control information acquisition step, the control information acquisition unit 214g executes the following process to acquire the shape data Da. Specifically, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input to the sixth learning model DM6, if the accuracy of the sixth learning model DM6 is less than a predetermined threshold (step S381: Yes), and the re-learning process is completed (step S382: Yes), the control information acquisition unit 214g inputs the captured images Im1 to Im4 to the sixth learning model DM6 after the re-learning process to acquire the shape data Da (step S383). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the sixth learning model DM6, if the accuracy of the sixth learning model DM6 is greater than or equal to a predetermined threshold (step S381: No), the control information acquisition unit 214g acquires shape data Da by inputting the captured images Im1 to Im4 into the sixth learning model DM6 on which the re-learning process has not been performed (step S384).

[0184] According to the above-mentioned eighth embodiment, when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained sixth learning model DM6, the information processing device 200g can execute the following process if the accuracy is less than a threshold. In this case, the information processing device 200g can retrain the sixth learning model DM6 by using the composite image Ci generated by combining the virtual vehicle image Vi with the background image Bi as the training image Ti.

[0185] Furthermore, according to the eighth embodiment, the information processing device 200g can retrain the sixth learning model DM6 using a sixth additional learning data set including a virtual vehicle image Vi that virtually reproduces the vehicle 100 in an appearance state specified by the appearance information. In this way, when the sixth learning model DM6 is retrained, it is possible to learn a feature amount according to the appearance state of the vehicle 100 to be detected. This can improve the accuracy of the sixth learning model DM6.

[0186] According to the eighth embodiment, when the virtual vehicle image Vi is synthesized with the background image Bi, at least one of the position and the orientation of the vehicle 100 can be made different by adjusting the arrangement of the virtual vehicle image Vi with respect to the background image Bi. This makes it possible to easily generate a plurality of training images Ti in which at least one of the position and the orientation of the vehicle 100 with respect to the passage TR is different.

[0187] According to the eighth embodiment, in the process of synthesizing the virtual vehicle image Vi with the background image Bi, at least one of the pixel values ​​of the background image Bi, the virtual vehicle image Vi, and the synthetic image Ci can be changed from the initial value. This makes it possible to easily generate a plurality of training images Ti with different imaging conditions such as brightness. Therefore, it is possible to easily generate a plurality of training images Ti assuming a case where pixel values ​​such as brightness change depending on the weather, time of day, season, lighting environment, etc.

[0188] In addition, when the training image Ti is a synthetic image Ci as in this embodiment, the correct label of the sixth learning model DM6 may be a contour correct label Lc instead of the area correct label Lr. The contour correct label Lc is a correct label representing the contour of the vehicle 100 extracted from the virtual vehicle image Vi. In this form, by extracting the outer shape of the virtual vehicle image Vi, it is possible to easily identify whether each area constituting the training image Ti is an area representing the vehicle 100 or an area representing something other than the vehicle 100. This reduces the load required for preparation for learning the sixth learning model DM6.

[0189] I. Ninth embodiment: FIG. 36 is a block diagram showing a configuration of an information processing system 50h in the ninth embodiment. The information processing system 50h includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200h. In this embodiment, when the captured images Im1 to Im4 captured by some of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold, the information processing device 200h trains a seventh learning model DM7 as a machine learning model DM separately from the first learning model DM1. Then, the information processing device 200h inputs the captured images Im1 to Im4 to the seventh learning model DM7 to obtain shape data Da. The other configurations of the information processing system 50h are the same as those of the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0190] The information processing device 200 h is configured by a computer including a processor 201 h, a memory 202 h, an input / output interface 203, and an internal bus 204.

[0191] The memory 202h stores various information including a program PG2h that controls the operation of the information processing device 200h, a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300.

[0192] The processor 201h functions as an accuracy acquisition unit 211, a learning data acquisition unit 212h, a learning unit 213h, a control information acquisition unit 214h, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217 by executing the program PG2h stored in the memory 202h.

[0193] The learning data acquisition unit 212h executes the following process when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning data acquisition unit 212h discards the first learning model DM1 whose accuracy is less than the threshold, and acquires a new learning data set used to newly learn the seventh learning model DM7. The new learning data set includes one or more new learning data sets in which a training image Ti including a background image Bi and a vehicle image according to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, and a region correct answer label Lr are associated with each other. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Therefore, the learning data acquisition unit 212h discards the first learning model DM1 whose accuracy is below the threshold, and acquires a new group of learning data for training the seventh learning model DM7.

[0194] When the captured images Im1 to Im4 acquired by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, and the accuracy of the first learning model DM1 is less than a predetermined threshold, the learning unit 213h executes the following process. In this case, the learning unit 213h discards the first learning model DM1. Then, the learning unit 213h executes a new learning process to newly learn a seventh learning model DM7. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Therefore, the learning unit 213h discards the first learning model DM1 corresponding to the external camera 300 that acquired the captured images Im1 to Im4 whose accuracy is less than the threshold. Then, the learning unit 213h uses the acquired new learning data group to learn the seventh learning model DM7 corresponding to the external camera 300 that acquired the captured images Im1 to Im4 whose accuracy is less than the threshold. For example, a CNN that realizes either semantic segmentation or instance segmentation can be used as the seventh learning model DM7. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the seventh learning model DM7 and the region correct label Lr. The learning unit 213h stores the created seventh learning model DM7 in the memory 202h.

[0195] When the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, the control information acquisition unit 214g inputs the captured images Im1 to Im4 to the first learning model DM1 to acquire shape data Da if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold. On the other hand, when the captured images Im1 to Im4 captured by some of the external cameras 300 are input to the first learning model DM1, the control information acquisition unit 214g inputs the captured images Im1 to Im4 to the seventh learning model DM7 to acquire shape data Da if the accuracy of the first learning model DM1 is less than a predetermined threshold. In this embodiment, each learning model DM1, DM7 is prepared for each external camera 300. Therefore, the control information acquisition unit 214h acquires the shape data Da by inputting the captured images Im1 to Im4 to the learning models DM1, DM7 corresponding to the external cameras 300 that acquired the captured images Im1 to Im4.

[0196] Fig. 37 is a flowchart showing a learning processing method in the 9th embodiment. The flow shown in Fig. 37 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0197] In the accuracy acquisition step, the accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 previously trained using the first initial learning data group (step S291). When the captured images Im1 to Im4 acquired by some of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold (step S292: Yes), the learning data acquisition step is executed. In the learning data acquisition step, the learning data acquisition unit 212h acquires a new learning data group (step S293). After the learning data acquisition step, the learning step is executed. In the learning step, the learning unit 213h discards the first learning model DM1 stored in the memory 202h (step S294). Then, the learning unit 213h executes a new learning process to newly train the seventh learning model DM7 using the acquired new learning data group (step S295). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S292: No), the processor 201h ends the flow without executing the learning data acquisition process and the learning process.

[0198] Fig. 38 is a flowchart showing a method for acquiring shape data Da in the 9th embodiment. The flow shown in Fig. 38 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0199] In the control information acquisition step, the control information acquisition unit 214h executes the following process to acquire shape data Da. Specifically, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold (step S391: Yes), and the new learning process of the seventh learning model DM7 is completed (step S392: Yes), the control information acquisition unit 214g executes the following process. In this case, the control information acquisition unit 214g inputs the captured images Im1 to Im4 to the seventh learning model DM7 to acquire shape data Da (step S393). On the other hand, when the captured images Im1 to Im4 captured by some of the multiple external cameras 300 are input into the first learning model DM1, if the accuracy of the first learning model DM1 is equal to or greater than a predetermined threshold (step S391: No), the control information acquisition unit 214h acquires shape data Da by inputting the captured images Im1 to Im4 into the first learning model DM1 (step S394).

[0200] According to the ninth embodiment, the information processing device 200h can execute the following process when the accuracy is less than a threshold when the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300 are input to the trained first learning model DM1. In this case, the information processing device 200h can newly train the seventh learning model DM7 using a new learning data set in which a background image Bi corresponding to the imaging area of ​​the external camera 300 that captured the captured images Im1 to Im4 whose accuracy is less than the threshold, a training image Ti including a vehicle image, and an area correct answer label Lr are associated with each other. In this way, when the appearance state in the imaging area of ​​some of the external cameras 300 among the multiple external cameras 300 changes from when the first learning model DM1 was trained, it is possible to suppress a decrease in the accuracy of the machine learning models DM1 and DM7.

[0201] Furthermore, according to the ninth embodiment, when new learning of the seventh learning model DM7 is completed, the information processing device 200h can obtain the shape data Da by inputting the captured images Im1 to Im4 to the seventh learning model DM7. This makes it possible to suppress a decrease in accuracy of the shape data Da generated using the captured images Im1 to Im4 captured by some of the external cameras 300 among the multiple external cameras 300.

[0202] J. Tenth embodiment: FIG. 39 is a block diagram showing the configuration of an information processing system 50v in the tenth embodiment. The information processing system 50v includes one or more vehicles 100v, a plurality of external cameras 300, and an information processing device 200v. In this embodiment, a vehicle control device 110v mounted on the vehicle 100v acquires vehicle position information using the detection result output from the external camera 300, and generates a driving control signal. The other configurations of the information processing system 50v are the same as those of the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those of the first embodiment, and the description thereof will be omitted.

[0203] The information processing device 200 v is configured by a computer including a processor 201 v, a memory 202 v, an input / output interface 203, and an internal bus 204.

[0204] The memory 202v stores various information including a program PG2v that controls the operation of the information processing device 200v and a plurality of first learning models DM1 prepared for each external camera 300.

[0205] The processor 201v functions as an accuracy acquisition unit 211, a learning data acquisition unit 212, a learning unit 213, and an update unit 219 by executing a program PG2v stored in the memory 202v.

[0206] The update unit 219 updates the first learning model DM1 stored in the vehicle control device 110v each time the first learning model DM1 is re-learned so that the first learning model DM1 stored in the vehicle control device 110v is in its latest state.

[0207] The vehicle control device 110 v is configured by a computer including a processor 111 v, a memory 112 v, an input / output interface 113, and an internal bus 114.

[0208] The memory 112v stores various information including a program PG1v that controls the operation of the vehicle control device 110v, a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300.

[0209] The processor 111v executes a program PG1v stored in the memory 112v, thereby functioning as a control information acquisition unit 116, a position acquisition unit 117, a signal generation unit 118, and a vehicle control unit 115v.

[0210] The control information acquisition unit 116 acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Therefore, the control information acquisition unit 116 acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 corresponding to the external camera 300 that acquired the captured images Im1 to Im4. At this time, the first learning model DM1 stored in the memory 112v is updated every time the re-learning process is executed. Therefore, when the re-learning process is completed in the information processing device 200v, the control information acquisition unit 116 acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the re-learning process.

[0211] The position acquisition unit 117 acquires vehicle position information using the shape data Da. The signal generation unit 118 generates a driving control signal. The vehicle control unit 115v controls the actuator group 120 using the generated driving control signal to drive the vehicle 100.

[0212] Fig. 40 is a flowchart showing a processing procedure of the travel control of the vehicle 100v in the tenth embodiment. The flow shown in Fig. 40 is repeatedly executed at predetermined time intervals from the time when the vehicle 100 starts traveling by unmanned driving, for example.

[0213] In step S121, the position acquisition unit 117 acquires vehicle position information using the detection result output from the external camera 300. In step S122, the signal generation unit 118 determines a target position to which the vehicle 100v should next head. In step S123, the signal generation unit 118 generates a driving control signal for driving the vehicle 100v toward the determined target position. In step S124, the vehicle control unit 115v controls the actuator group 120 using the generated driving control signal to drive the vehicle 100v according to the parameters represented in the driving control signal. The processor 111v repeats the acquisition of vehicle position information, the determination of the target position, the generation of the driving control signal, and the control of the actuators at a predetermined cycle. According to the information processing system 50v in this embodiment, the vehicle 100v can be driven by the autonomous control of the vehicle 100v without remotely controlling the vehicle 100v by the information processing device 200v.

[0214] K. Other Embodiments: K-1. Other embodiment 1: At least some of the functions of the information processing devices 200, 200a to 200h, 200v may be one function of the vehicle control devices 110, 110v, or one function of the external camera 300. For example, when all the functions of the information processing devices 200, 200a to 200h, 200v are realized by the external camera 300, the information processing devices 200, 200a to 200h, 200v can acquire the coordinate values ​​Cr1 to Cr4, Cb1 to Cb8 in the camera coordinate system without using a camera identifier.

[0215] K-2. Other embodiment 2: The memories 202, 202a to 202h, 202v of the information processing devices 200, 200a to 200h, 200v may store one machine learning model DM that is commonly used for the captured images Im1 to Im4 acquired by each of the multiple external cameras 300. Even in this form, it is possible to prevent the accuracy of the machine learning model DM from decreasing when the appearance state of the imaging area of ​​some of the multiple external cameras 300 has changed since the time when the machine learning model DM was learned.

[0216] K-3. Other embodiment 3: When the learning process is completed and the accuracy is equal to or greater than the threshold when the captured images Im1 to Im4 captured by any of the multiple external cameras 300 are input to the machine learning model DM after the learning process, the control information acquisition units 116, 214, 214a to 214h may acquire the control information by inputting the captured images Im1 to Im4 to the machine learning model DM after the learning process. In other words, when the ratio of the number of the captured images Im1 to Im4 whose post-learning accuracy is equal to or greater than the threshold to the total number of the multiple captured images Im1 to Im4 used to calculate the post-learning accuracy is 100%, the control information acquisition units 116, 214, 214a to 214h may acquire the control information by inputting the captured images Im1 to Im4 to the machine learning model DM after the learning process. Even in this form, it is possible to suppress a decrease in the accuracy of the control information acquired using the captured images Im1 to Im4 captured by some of the multiple external cameras 300.

[0217] K-4. Other embodiment 4: The transmission unit 217 of the information processing device 200, 200a to 200h may transmit a driving control signal different from the reference control signal to the vehicle 100 when acquiring the captured images Im1 to Im4 used as the training images Ti in the learning process. The reference control signal is either a driving control signal acquired by the control information acquisition unit 214f ​​or a driving control signal generated from the signal generation parameters acquired by the control information acquisition units 214, 214a to 214e, 214g, and 214h. For example, when acquiring the captured images Im1 to Im4 used as the training images Ti in the learning process, in order to acquire clear captured images Im1 to Im4 with less blur, the transmission unit 217 may transmit the following driving control signal to the vehicle 100. In this case, the transmission unit 217 transmits to the vehicle 100, for example, a driving control signal for causing the vehicle 100 to travel at an acceleration smaller than the acceleration represented by the reference control signal. With this configuration, a clear training image Ti with less blur can be easily generated. Furthermore, in the case of acquiring the captured images Im1 to Im4 used as the training images Ti in the learning process, in order to increase the number of the captured images Im1 to Im4 that can be acquired within a certain time, the transmission unit 217 may transmit the following driving control signal to the vehicle 100. In this case, the transmission unit 217 transmits to the vehicle 100, for example, a driving control signal for causing the vehicle 100 to travel at an acceleration higher than the acceleration represented by the reference control signal. In this form, it is possible to acquire more captured images Im1 to Im4 within a certain time. In the case of acquiring the captured images Im1 to Im4 used as the training images Ti in the learning process, in order to acquire the captured images Im1 to Im4 including the vehicle 100 in a plurality of different orientations, the transmission unit 217 may transmit to the vehicle 100 the following driving control signal. In this case, the transmission unit 217 transmits to the vehicle 100, for example, a driving control signal for causing the vehicle 100 to travel at a steering angle different from the steering angle represented by the reference control signal. In this manner, a plurality of training images Ti in which the orientation of the vehicle 100 with respect to the passage TR is different can be easily generated. In addition, when there are two or more routes on which the vehicle 100 can travel, the transmission unit 217 may transmit the following travel control signal to the vehicle 100.In this case, the transmission unit 217 transmits to the vehicle 100 a traveling control signal generated using a reference route RR different from the reference route RR used to generate the reference control signal. In this manner, more vehicles 100 can be made to travel along the passages TR included in the imaging area that is the target of the learning process. As described above, by transmitting a traveling control signal different from the reference control signal to the vehicle 100, the information processing devices 200, 200a to 200h can reduce the load required for preparing the training image Ti and make it easier to generate a desired training image Ti.

[0218] K-5. Other embodiment 5: In each of the first to ninth embodiments, the information processing devices 200, 200a to 200h execute processes from acquisition of vehicle position information to generation of a driving control signal. In contrast, at least a part of the processes from acquisition of vehicle position information to generation of a driving control signal may be executed by the vehicle 100. For example, the following forms (1) to (3) may be used.

[0219] (1) The information processing devices 200, 200a to 200h may acquire vehicle position information, determine a target position to which the vehicle 100 should next head, and generate a route from the current location of the vehicle 100 represented in the acquired vehicle position information to the target position. The information processing devices 200, 200a to 200h may generate a route to a target position between the current location and the destination, or may generate a route to the destination. The information processing devices 200, 200a to 200h may transmit the generated route to the vehicle 100. The vehicle 100 may generate a driving control signal so that the vehicle 100 drives on the route received from the information processing devices 200, 200a to 200h, and control the actuator group 120 using the generated driving control signal.

[0220] (2) The information processing devices 200, 200a to 200h may acquire vehicle position information and transmit the acquired vehicle position information to the vehicle 100. The vehicle 100 may determine a target position to which the vehicle 100 should next head, generate a route from the current position of the vehicle 100 represented in the received vehicle position information to the target position, generate a driving control signal so that the vehicle 100 travels on the generated route, and control the actuator group 120 using the generated driving control signal.

[0221] (3) In the above-mentioned embodiments (1) and (2), an internal sensor may be mounted on the vehicle 100, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. The internal sensor is a sensor mounted on the vehicle 100. The internal sensor may include, for example, a sensor for detecting the motion state of the vehicle 100, a sensor for detecting the operating state of each part of the vehicle 100, and a sensor for detecting the environment around the vehicle 100. Specifically, the internal sensor may include, for example, a camera, LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyro sensor, and the like. For example, in the above-mentioned embodiment (1), the information processing device 200, 200a to 200h may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor on the route when generating a route. In the above-mentioned embodiment (1), the vehicle 100 may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor on the driving control signal when generating a driving control signal. In the above embodiment (2), the vehicle 100 may acquire the detection result of the internal sensor, and when generating a route, may reflect the detection result of the internal sensor in the route. In the above embodiment (2), the vehicle 100 may acquire the detection result of the internal sensor, and when generating a route, may reflect the detection result of the internal sensor in the route.

[0222] K-6. Other embodiment 6: In the above tenth embodiment, an internal sensor may be mounted on the vehicle 100v, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. For example, the vehicle 100v may acquire a detection result of the internal sensor, and when generating a route, may reflect the detection result of the internal sensor in the route. The vehicle 100v may acquire a detection result of the internal sensor, and when generating a driving control signal, may reflect the detection result of the internal sensor in the driving control signal.

[0223] K-7. Other embodiment 7: In the above tenth embodiment, the vehicle 100v acquires vehicle position information using the detection result of the external camera 300 as an external sensor that is a sensor located outside the vehicle 100v. In contrast, the vehicle 100v may be equipped with an internal sensor, and the vehicle 100v may acquire vehicle position information using the detection result of the internal sensor, determine a target position to which the vehicle 100v should next head, generate a route from the current location of the vehicle 100v represented in the acquired vehicle position information to the target position, generate a travel control signal for traveling along the generated route, and control the actuator group 120 using the generated travel control signal. In this case, the vehicle 100v can travel without using any detection result of the external sensor. The vehicle 100v may acquire a target arrival time or traffic congestion information from outside the vehicle 100v, and reflect the target arrival time or traffic congestion information in at least one of the route and the travel control signal.

[0224] K-8. Other embodiment 8: In each of the first to ninth embodiments, the information processing devices 200, 200a to 200h automatically generate a driving control signal to be transmitted to the vehicle 100. In contrast, the information processing devices 200, 200a to 200h may generate a driving control signal to be transmitted to the vehicle 100 in accordance with the operation of an external operator located outside the vehicle 100. For example, the external operator may operate a steering device including a display for displaying the captured images Im1 to Im4 output from an external sensor, a steering wheel for remotely operating the vehicle 100, an accelerator pedal, a brake pedal, and a communication device for communicating with the information processing devices 200, 200a to 200h by wired communication or wireless communication, and the information processing devices 200, 200a to 200h may generate a driving control signal in accordance with the operation applied to the steering device.

[0225] K-9. Other embodiment 9: In each of the above embodiments, the vehicle 100, 100v may have a configuration capable of moving by unmanned driving, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 100, 100v may have at least a vehicle control device 110 and an actuator group 120 in order to perform the three functions of "running", "turning", and "stopping" by unmanned driving. When the vehicle 100, 100v acquires information from the outside for unmanned driving, the vehicle 100, 100v may further have a communication device 130. That is, the vehicle 100, 100v capable of moving by unmanned driving may not be equipped with at least a part of interior parts such as a driver's seat and a dashboard, may not be equipped with at least a part of exterior parts such as a bumper and a fender, and may not be equipped with a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 100, 100v before the vehicle 100, 100v is shipped from the factory FC, or the remaining parts such as the body shell may be attached to the vehicle 100, 100v after the vehicle 100, 100v is shipped from the factory FC without the remaining parts such as the body shell being attached to the vehicle 100, 100v. Each part may be attached from any direction such as the upper side, lower side, front side, rear side, right side, or left side of the vehicle 100, 100v, and may be attached from the same direction or from different directions. Note that the position of the platform shape may be determined in the same manner as the vehicle 100, 100v in each of the above embodiments.

[0226] K-10. Other embodiment 10: The vehicle 100, 100v may be manufactured by combining a plurality of modules. The module means a unit composed of a plurality of parts grouped according to the parts and functions of the vehicle 100, 100v. For example, the platform of the vehicle 100, 100v may be manufactured by combining a front module constituting the front part of the platform, a central module constituting the central part of the platform, and a rear module constituting the rear part of the platform. The number of modules constituting the platform is not limited to three, and may be two or less or four or more. In addition to or instead of the parts constituting the platform, parts constituting parts of the vehicle 100, 100v other than the platform may be modularized. The various modules may include any exterior parts such as a bumper or a grille, or any interior parts such as a seat or a console. In addition to the vehicle 100, 100v, any type of moving body may be manufactured by combining a plurality of modules. Such a module may be manufactured, for example, by joining a plurality of parts by welding or a fastener, or may be manufactured by integrally molding at least a part of the parts constituting the module as one part by casting. The molding method for integrally molding a single component, particularly a relatively large component, is also called gigacast or megacast. For example, the front module, the center module, and the rear module may be manufactured using gigacast.

[0227] K-11. Other embodiment 11: Transporting vehicles 100, 100v using unmanned driving of vehicles 100, 100v is also called "self-propelled transport." A configuration for realizing self-propelled transport is also called a "vehicle remote-controlled autonomous traveling transport system." A production method for producing vehicles 100, 100v using self-propelled transport is also called "self-propelled production." In self-propelled production, for example, at factory FC where vehicles 100, 100v are manufactured, at least a portion of the transport of vehicles 100, 100v is realized by self-propelled transport.

[0228] K-12. Other embodiment 12: In each of the above embodiments, some or all of the functions and processes implemented by software may be implemented by hardware. Also, some or all of the functions and processes implemented by hardware may be implemented by software. As hardware for implementing the various functions in each of the above embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0229] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0230] 10e...positioning point, 50, 50a to 50h, 50v...information processing system, 100, 100v...vehicle, 110, 110v...vehicle control device, 111, 111v...vehicle control device processor, 112, 112v...vehicle control device memory, 113...vehicle control device input / output interface, 114...vehicle control device internal bus, 115, 115v...vehicle control unit, 116, 214, 214a to 214h...control information acquisition unit, 117, 215, 215c, 215d...position acquisition unit, 118, 216...signal generation unit, 120...actuator group, 130...vehicle Communication device, 200, 200a to 200h, 200v...information processing device, 201, 201a to 201h, 201v...processor of information processing device, 202, 202a to 202h, 202v...memory of information processing device, 203...input / output interface of information processing device, 204...internal bus of information processing device, 205...communication device of information processing device, 211, 211b to 211g...accuracy acquisition unit, 212, 212a to 212h...learning data acquisition unit, 213, 213a to 213h...learning unit, 217...transmission unit, 218...appearance acquisition unit, 219...update unit, 300...external Camera, A1... necessary area, A2... unnecessary area, Bi... background image, Br... real background image, Bv... virtual background image, C... center of gravity, Cb1~Cb8... 3D coordinate values, Ci... synthetic image, Cr1~Cr4... rectangular coordinate values, D... first distance, ΔD... observation error, Da... shape data, Db1... rectangular database, Db2... 3D database, Dc... third distance, DM... machine learning model, DM1... first learning model, DM2... second learning model, DM3... third learning model, DM4... fourth learning model, DM5... fifth learning model, DM6... sixth learning model, DM7... seventh learning model ,Do...observation distance, Dp...estimated distance, Dt...second distance, FC...factory, GC...global coordinate system, h...height of vehicle positioning point from road surface, H...height of external camera from road surface, Im1...original image, Im2...corrected image, Im3...rotated image, Im4...processed image, Im1~Im4...captured image, Im5...first mask image, Im6...second mask image, Lc...contour correct answer label, Lr...area correct answer label, Ms...mask area, P0...base coordinate point, P1...first coordinate point, P2...second coordinate point, P3...image coordinate point, PG1,PG1v,PG2,PG2a~PG2h,PG2v...program, Pa...camera parameters, Pc...imaging coordinate point, Pf...fixed coordinate point, PL1...first location, PL2...second location, Pv...vehicle coordinate point, R0...base circumscribing rectangle, R1...first circumscribing rectangle, R2...second circumscribing rectangle, RR...reference route, Rb...vehicle circumscribing rectangle, Rc...vehicle circumscribing rectangle, Rs...road surface, SA...surrounding area, TA...target area, TR...passage, Ti...training image, V...movement vector, Vb1~Vb8...vertices of vehicle circumscribing rectangle, Vc1~Vc4...vertices of vehicle circumscribing rectangle, Vi...virtual vehicle image,

Claims

1. An information processing device used to move a movable body that can be moved by unmanned operation, an accuracy acquisition unit that acquires the accuracy of a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body to move the moving body by unmanned operation and a parameter used when generating the control signal when an image acquired by imaging the moving body by an external camera installed at a location different from the moving body is input; a training data acquisition unit that acquires a group of training data used for training the machine learning model when the accuracy is less than a predetermined threshold; A learning unit that executes a learning process to train the machine learning model using the acquired group of learning data, The information processing device includes one or more learning data sets in which the learning data group corresponds to training images including a background image representing a target area including the passageway along which the moving body moves and the surrounding area of ​​the passageway, a moving body image representing the moving body, and a correct answer label.

2. 2. The information processing device according to claim 1, The machine learning model is trained in advance using a group of initial training data, the learning data acquisition unit acquires an additional learning data group as the learning data group used for additional learning of the machine learning model when the accuracy is less than the threshold value when the captured images acquired by some of the external cameras among the plurality of external cameras are input to the machine learning model; The learning unit executes an additional learning process for additionally learning the machine learning model using the acquired additional learning data group; The initial learning data group includes a plurality of initial learning data sets corresponding to the imaging areas of the plurality of external cameras, the initial learning data sets including the background image corresponding to the imaging area, the moving object image, and a training image and a correct answer label associated with each other; The additional learning data group includes one or more additional learning datasets that correspond to the training image including the background image corresponding to the imaging area of ​​the external camera that acquired the captured image whose accuracy is below the threshold, the moving object image, and the correct label.

3. 2. The information processing device according to claim 1, The machine learning model is trained in advance using a group of initial training data, the learning data acquisition unit acquires a new learning data group as the learning data group used to newly train a machine learning model different from the machine learning model when the accuracy is less than the threshold value when the captured images acquired by some of the multiple external cameras are input to the machine learning model; The learning unit executes a new learning process to train a new machine learning model using the acquired new learning data group, The initial learning data group includes a plurality of initial learning data sets corresponding to the imaging areas of the plurality of external cameras, the initial learning data sets including the background image corresponding to the imaging area, the moving object image, and a training image and a correct answer label associated with each other; The new learning data group includes one or more new learning datasets that correspond to the training image including the background image corresponding to the imaging area of ​​the external camera that acquired the captured image whose accuracy is below the threshold, the moving object image, and the correct label.

4. 2. The information processing device according to claim 1, The training image is at least one of the captured image and a composite image, The information processing device generates the composite image by combining a virtual moving body image that virtually reproduces the moving body with either a real background image obtained by capturing an image of the target area with the external camera or a virtual background image that virtually reproduces the target area.

5. The information processing device according to claim 4, further comprising: an appearance acquisition unit that acquires appearance information indicating an appearance state of the moving object, The learning unit executes the learning process using the virtual moving object image that virtually reproduces the moving object in the appearance state specified by the appearance information.

6. 2. The information processing device according to claim 1, The learning unit executes the learning process during a period in which control of the unmanned driving is being performed.

7. The information processing device according to claim 1 , further comprising: a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image to the machine learning model; When the learning process is completed, the control information acquisition unit acquires the control information by inputting the captured image to the machine learning model after the learning process.

8. The information processing device according to claim 1 , further comprising: a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image to the machine learning model; The accuracy acquisition unit further acquires accuracy of the machine learning model after the learning process, When the learning process is completed and the accuracy of the machine learning model after the learning process is equal to or greater than the threshold, the control information acquisition unit acquires the control information by inputting the captured image to the machine learning model after the learning process.

9. 9. The information processing device according to claim 8, The accuracy acquisition unit acquires the accuracy of the machine learning model after the learning process by using a plurality of captured images acquired by capturing an image of a moving object moving behind the moving object included in the training image with the external camera, An information processing device in which, when the ratio of the number of captured images for which the accuracy of the machine learning model after the learning process is equal to or greater than the threshold to the total number of the captured images used to calculate the accuracy of the machine learning model after the learning process is equal to or greater than a reference value, the control information acquisition unit acquires the control information by inputting the captured images to the machine learning model after the learning process.

10. The information processing device according to claim 1 , further comprising: a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image to the machine learning model; a transmitter that transmits the control signal to the moving object, The training image is an image obtained by imaging the moving body that is performing the control of the unmanned driving by the external camera, An information processing device in which, when acquiring the captured image to be used as the training image in the learning process, the transmission unit transmits to the moving body a control signal that is different from the control signal acquired by the control information acquisition unit and the control signal generated from the parameters acquired by the control information acquisition unit.

11. An information processing system used to move a movable body that can be moved by unmanned operation, A moving body, A plurality of external cameras installed at locations different from the moving object; An information processing system comprising: the information processing device according to any one of claims 1 to 10.

12. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: an accuracy acquisition process for acquiring the accuracy of a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body to move the moving body by unmanned operation and a parameter used when generating the control signal when an image acquired by imaging the moving body with an external camera installed at a location different from the moving body is input; a training data acquisition step of acquiring a group of training data used for training the machine learning model when the accuracy is less than a predetermined threshold; A learning process for executing a learning process for learning the machine learning model using the acquired group of learning data, An information processing method, wherein the learning data group includes one or more learning datasets in which training images including a background image representing a target area including the passageway along which the moving body moves and the surrounding area of ​​the passageway, a moving body image representing the moving body, and a correct answer label are associated with each other.

Images