Information processing apparatus, information processing system, and information processing method

Abstract

Provided is a technique capable of suppressing a decrease in accuracy of a machine learning model when an appearance state in an imaging region of a camera changes from the time of learning by the machine learning model. An information processing device for moving a moving body that can be moved by unmanned driving includes: an accuracy acquisition unit that acquires accuracy of a machine learning model; a learning data acquisition unit that acquires a learning data group for learning the machine learning model when the accuracy is less than a predetermined threshold value; and a learning unit that executes learning processing for learning the machine learning model using the learning data set.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. Background Art

[0002] Conventionally, there is known a technology for automatically driving a vehicle by remote control by monitoring the driving of the vehicle using a camera outside the vehicle (Patent Document 1). Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application No. 2017-538619 Summary of the invention Problems to be solved by the invention

[0004] In order to make a moving object such as a vehicle move without human control, an image of the moving object captured by a camera is input into a machine learning model to obtain at least one of a control signal that specifies the movement of the moving object and a parameter for generating the control signal. However, if the appearance of the shooting area of ​​the camera has changed since the machine learning model was learned, the accuracy of the machine learning model may decrease. Means for solving problems

[0005] The present disclosure can be implemented as the following aspects.

[0006] (1) According to a first aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body that can be moved by unmanned driving comprises: an accuracy acquisition unit that acquires the accuracy of a machine learning model, the machine learning model outputting at least one of a control signal that specifies the action of the mobile body and a parameter used when generating the control signal when an image captured by capturing the mobile body using an external camera installed at a location different from the mobile body is input; a learning data acquisition unit that acquires a learning data set for learning the machine learning model when the accuracy is less than a predetermined threshold; and a learning unit that performs a learning process for learning the machine learning model using the acquired learning data set, the learning data set including one or more learning data sets in which training images including a background image representing a target area and a mobile image representing the mobile body and a correct answer label are associated with each other. According to this aspect, when the accuracy of the machine learning model is less than the threshold, the information processing device can use the acquired learning data set to learn the machine learning model. In this way, when the appearance state of the shooting area of ​​the external camera has changed since the time of learning the machine learning model, the decrease in the accuracy of the machine learning model can be suppressed. (2) In the above method, it may be that the machine learning model is pre-learned using an initial learning data set, and the learning data acquisition unit obtains an additional learning data set as the learning data set for additionally learning the machine learning model when the accuracy of the captured image obtained by a part of the external cameras among the plurality of external cameras is less than the threshold when the captured image is input into the machine learning model, and the learning unit performs additional learning processing for additionally learning the machine learning model using the acquired additional learning data set, the initial learning data set includes a plurality of initial learning data sets corresponding to each capturing area of ​​the plurality of external cameras, the plurality of initial learning data sets establish correspondence between training images including the background image and the moving object image corresponding to the capturing area and correct labels, and the additional learning data set includes one or more additional learning data sets that establish correspondence between the training images including the background image and the moving object image corresponding to the capturing area of ​​the external camera that obtained the captured image with the accuracy less than the threshold and the correct labels. According to this method, a machine learning model pre-learned using a plurality of initial learning data sets corresponding to each shooting area of ​​a plurality of external cameras is prepared, wherein the plurality of initial learning data sets establish correspondence between training images including background images and moving body images corresponding to the shooting areas and correct labels. When the accuracy of a shot image obtained by a part of the plurality of external cameras is less than a threshold when it is input to the learned machine learning model, the information processing device can perform the following processing. In this case, the information processing device can additionally learn the machine learning model using a data set for additional learning that establishes correspondence between training images including background images and moving body images corresponding to the shooting areas of the external cameras that obtained the shot images with an accuracy less than the threshold and correct labels. In this way, when the appearance state in the shooting areas of a part of the plurality of external cameras has changed since the time when the machine learning model was learned, the accuracy of the machine learning model can be suppressed from decreasing. (3) In the above method, it may be that the machine learning model is pre-learned using an initial learning data set, and the learning data acquisition unit obtains a new learning data set as the learning data set for newly learning a machine learning model different from the machine learning model when the captured image obtained by a part of the external cameras among the plurality of external cameras is input into the machine learning model and the accuracy is less than the threshold value, and the learning unit performs a new learning process for learning the new machine learning model using the acquired new learning data set, the initial learning data set includes a plurality of initial learning data sets corresponding to each capturing area of ​​the plurality of external cameras, the plurality of initial learning data sets establish correspondence between training images including the background image and the moving object image corresponding to the capturing area and correct labels, and the new learning data set includes one or more new learning data sets that establish correspondence between training images including the background image and the moving object image corresponding to the capturing area of ​​the external camera that obtained the captured image with the accuracy less than the threshold value and correct labels. According to this method, a machine learning model that is pre-learned using a plurality of initial learning data sets corresponding to each shooting area of ​​a plurality of external cameras is prepared, wherein the plurality of initial learning data sets establish correspondence between training images including background images and moving body images corresponding to the shooting areas and correct labels. When the accuracy of a shot image obtained by a part of the plurality of external cameras is less than a threshold when it is input to the learned machine learning model, the information processing device can perform the following processing. In this case, the information processing device can use a new learning data set that establishes correspondence between training images including background images and moving body images corresponding to the shooting area of ​​the external camera that obtained the shot image with an accuracy less than the threshold and correct labels to newly learn the machine learning model. In this way, when the appearance state in the shooting area of ​​a part of the plurality of external cameras has changed since the time when the machine learning model was learned, the accuracy of the machine learning model can be suppressed from decreasing. (4) In the above method, the training image may be at least one of the captured image and the synthesized image, and the synthesized image is generated by synthesizing the background image of either an actual background image obtained by capturing the object area using the external camera and a virtual background image virtually reproducing the object area into a virtual moving body image that virtually reproduces the moving body. According to this method, the information processing device can use the synthesized image generated by synthesizing the virtual moving body image into the background image of either the actual background image or the virtual background image as a training image used in learning the machine learning model. (5) In the above method, it may be that there is also an appearance acquisition unit that acquires appearance information representing the appearance state of the moving body, and the learning unit performs the learning process using the virtual moving body image of the moving body that virtually reproduces the appearance state determined by the appearance information. According to this method, the information processing device can use the training image including the virtual moving body image of the moving body that virtually reproduces the appearance state determined by the appearance information to make the machine learning model learn. In this way, when the machine learning model is learned, the feature value corresponding to the appearance state of the moving body of the object can be learned. In this way, the accuracy of the machine learning model can be improved. (6) In the above aspect, the learning unit may execute the learning process while the unmanned driving control is being executed. According to this aspect, the information processing device can execute the learning process of causing the machine learning model to learn while the mobile body is executing the unmanned driving control. (7) In the above-mentioned method, it is possible that there is also a control information acquisition unit that obtains control information of at least one of the control signal and the parameter by inputting the captured image to the machine learning model, and when the learning process is completed, the control information acquisition unit obtains the control information by inputting the captured image to the machine learning model after the learning process. According to this method, when the learning process is completed, the information processing device can obtain control information by inputting the captured image to the machine learning model after the learning process. Thereby, it is possible to suppress the decrease in the accuracy of the control information obtained using the captured images obtained by some of the external cameras among the multiple external cameras. (8) In the above-mentioned method, it is possible that there is also a control information acquisition unit that obtains control information of at least one of the control signal and the parameter by inputting the captured image into the machine learning model, and the accuracy acquisition unit also obtains the accuracy of the machine learning model after the learning process. When the learning process is completed and the accuracy of the machine learning model after the learning process is greater than the threshold, the control information acquisition unit obtains the control information by inputting the captured image into the machine learning model after the learning process. According to this method, when the learning process is completed and the accuracy of the machine learning model after the learning process is greater than the threshold, the information processing device can obtain control information by inputting the captured image into the machine learning model after the learning process. As a result, it is possible to further suppress the decrease in the accuracy of the control information obtained using the captured images obtained by a part of the external cameras among the multiple external cameras. (9) In the above-mentioned aspect, the accuracy acquisition unit may acquire the accuracy of the machine learning model after the learning process using a plurality of captured images acquired by capturing the moving body moving behind the moving body included in the training image using the external camera, and when the ratio of the number of captured images for which the accuracy of the machine learning model after the learning process becomes equal to or greater than the threshold to the total number of the plurality of captured images used to calculate the accuracy of the machine learning model after the learning process is equal to or greater than a reference value, the control information acquisition unit may acquire the control information by inputting the captured images to the machine learning model after the learning process. According to this aspect, the information processing device may acquire the control information by inputting the captured images to the machine learning model after the learning process when the ratio of the number of captured images for which the accuracy of the machine learning model after the learning process becomes equal to or greater than the threshold to the total number of the plurality of captured images used to calculate the accuracy of the machine learning model after the learning process is equal to or greater than a reference value. Thus, it is possible to further suppress a decrease in the accuracy of the control information acquired using the captured images acquired by a part of the plurality of external cameras. (10) In the above-mentioned method, it may be that the information processing device further comprises: a control information acquisition unit that acquires control information of at least one of the control signal and the parameter by inputting the captured image into the machine learning model; and a transmission unit that transmits the control signal to the mobile body, wherein the training image is the captured image acquired by using the external camera to capture the mobile body that is performing the unmanned driving control, and when the captured image used as the training image in the learning process is acquired, the transmission unit transmits the control signal different from the control signal acquired by the control information acquisition unit and the control signal generated based on the parameter acquired by the control information acquisition unit to the mobile body. According to this method, the information processing device can transmit the control signal different from the control signal acquired using the machine learning model and the control signal generated based on the parameter acquired using the machine learning model to the mobile body. As a result, it is possible to reduce the load required for preparing the training image or to make it easier to generate the desired training image. (11) According to the second embodiment of the present disclosure, an information processing system is provided. The information processing system for moving a mobile body capable of moving by unmanned driving comprises: a mobile body; a plurality of external cameras, which are arranged at a location different from the mobile body; and an information processing device as described in the above embodiment. According to this embodiment, when the accuracy of the machine learning model is less than a threshold value, the information processing system can use the acquired learning data set to learn the machine learning model. In this way, when the appearance state of the shooting area of ​​the external camera has changed since the time when the machine learning model was learned, the decrease in the accuracy of the machine learning model can be suppressed. (12) According to a third aspect of the present disclosure, an information processing method is provided. The information processing method for moving a mobile body that can be moved by unmanned driving comprises: an accuracy acquisition step of acquiring the accuracy of a machine learning model, the machine learning model outputting at least one of a control signal that specifies the action of the mobile body and a parameter used when generating the control signal when an image captured by capturing the mobile body using an external camera installed at a location different from the mobile body is input; a learning data acquisition step of acquiring a learning data set for learning the machine learning model when the accuracy is less than a predetermined threshold; and a learning step of performing a learning process for learning the machine learning model using the acquired learning data set, the learning data set including one or more learning data sets in which a training image including a background image representing a target area and a mobile image representing the mobile body and a correct answer label are associated with each other. The target area includes a path for the mobile body to move and a peripheral area of ​​the path. According to this aspect, when the accuracy of the machine learning model is less than the threshold, the machine learning model can be learned using the acquired learning data set. In this way, when the appearance state of the shooting area of ​​the external camera has changed since the time of learning the machine learning model, the decrease in the accuracy of the machine learning model can be suppressed. The present disclosure can be implemented in various forms other than the above-mentioned information processing device, information processing system, and information processing method. For example, it can be implemented in the form of an information processing device, an information processing system, and a manufacturing method of a mobile body, a control method of an information processing device, an information processing system, and a mobile body, a computer program for implementing the control method, a non-transitory recording medium recording the computer program, and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a conceptual diagram showing the configuration of the information processing system in the first embodiment. Figure 2 It is a block diagram showing the configuration of the information processing system in the first embodiment. Figure 3This is a flowchart showing a processing procedure of the travel control of the vehicle in the first embodiment. Figure 4 : is a flowchart showing the learning processing method in the first embodiment. Figure 5 This is a flowchart showing the method of acquiring shape data in the first embodiment. Figure 6 This is a flowchart showing an example of a method for obtaining the position of a vehicle using shape data. Figure 7 It shows that the Figure 6 Schematic diagram of various image examples of the method shown. Figure 8 This is a diagram for explaining the details of the coordinate point calculation process. Fig. 9 This is a diagram for explaining the method of calculating basic coordinate points. Fig.10 This is the first diagram for explaining the details of the position conversion process. Fig.11 This is a second diagram for explaining the details of the position conversion process. Fig.12 It is a block diagram showing the configuration of an information processing system in the second embodiment. Fig.13 : is a flowchart showing the learning processing method in the second embodiment. Fig.14 This is a flowchart showing a method for acquiring shape data in the second embodiment. Fig.15 It is a block diagram showing the configuration of an information processing system in the third embodiment. Fig.16 : is a flowchart showing the learning processing method in the third embodiment. Fig.17 This is a flowchart showing a method for acquiring shape data in the third embodiment. Fig.18 It is a block diagram showing the configuration of an information processing system in the fourth embodiment. Fig.19 It is a diagram for explaining the coordinate values ​​of a rectangle. Fig. 20 : is a flowchart showing the learning processing method in the fourth embodiment. Fig.21 This is a flowchart showing a method for obtaining rectangle coordinate values. Fig. 22 It is a block diagram showing the configuration of an information processing system in the fifth embodiment. Fig.23 It is a diagram for explaining the three-dimensional coordinate values. Fig.24 : is a flowchart showing the learning processing method in the fifth embodiment. Fig.25 This is a flowchart showing a method for obtaining three-dimensional coordinate values. Fig.26 It is a block diagram showing the configuration of an information processing system in the sixth embodiment. Fig. 27 : is a flowchart showing the learning processing method in the sixth embodiment. Fig.28 : is a flowchart showing a method of acquiring vehicle position information using the fourth learning model. Fig.29 It is a block diagram showing the configuration of an information processing system in the seventh embodiment. Fig.30 : is a flowchart showing the learning processing method in the seventh embodiment. Fig.31 This is a flowchart showing a method of acquiring a travel control signal using the fifth learning model. Fig.32 It is a block diagram showing the configuration of an information processing system in the eighth embodiment. Fig.33 This is a diagram showing an example of a method for generating a training image used for learning the sixth learning model. Fig.34 : is a flowchart showing the learning processing method in the eighth embodiment. Fig.35 This is a flowchart showing a method for acquiring shape data in the eighth embodiment. Fig.36 It is a block diagram showing the structure of the information processing system in the ninth embodiment. Fig.37 : is a flowchart showing the learning processing method in the ninth embodiment. Fig.38 This is a flowchart showing a method for acquiring shape data in the ninth embodiment. Fig.39 It is a block diagram showing the configuration of an information processing system in the tenth embodiment. Fig.40 1 is a flowchart showing a processing procedure of driving control of a vehicle in the tenth embodiment. DETAILED DESCRIPTION

[0008] A. First Implementation Method: A-1. Structure of information processing system: Figure 11 is a conceptual diagram showing the structure of the information processing system 50 in the first embodiment. The information processing system 50 is a system for moving a moving body without relying on the driving operation of a passenger on the moving body. The information processing system 50 includes one or more moving bodies, a plurality of external cameras 300 installed in a location different from the moving body, and an information processing device 200.

[0009] In the present disclosure, "mobile body" means an object that can move, such as a vehicle 100, an electric vertical take-off and landing aircraft (so-called flying car). In the present embodiment, the mobile body is a vehicle 100. The vehicle 100 may be a vehicle 100 that travels on wheels, or a vehicle 100 that travels on an infinite track, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a chariot, a construction vehicle, etc. The vehicle 100 includes electric vehicles (BEV: Battery Electric Vehicle), gasoline vehicles, hybrid vehicles, and fuel cell vehicles. In the case where the mobile body is other than the vehicle 100, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "travel" can be appropriately replaced with "movement".

[0010] The vehicle 100 is configured to be able to travel by unmanned driving. "Unmanned driving" means driving that does not rely on the driving operation of the passengers. The driving operation means an operation related to at least any one of "driving", "turning" and "stopping" of the vehicle 100. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 100 or by autonomous control of the vehicle 100. In the vehicle 100 that travels by unmanned driving, there may also be passengers who do not perform driving operations. Passengers who do not perform driving operations include, for example, people who simply sit in the seats of the vehicle 100, and people who perform operations different from driving operations such as assembly, inspection, and switch operations while riding in the vehicle 100. It should be noted that driving that relies on the driving operations of the passengers is sometimes referred to as "manned driving".

[0011] In this specification, “remote control” includes “full remote control” in which all actions of the vehicle 100 are completely determined from outside the vehicle 100, and “partial remote control” in which part of the actions of the vehicle 100 are determined from outside the vehicle 100. In addition, “autonomous control” includes “full autonomous control” in which the vehicle 100 autonomously controls its own actions without receiving any information from a device outside the vehicle 100, and “partial autonomous control” in which the vehicle 100 autonomously controls its own actions using information received from a device outside the vehicle 100.

[0012] In the present embodiment, the information processing system 50 is used in a factory FC for manufacturing a vehicle 100. The reference coordinate system of the factory FC is the global coordinate system GC. That is, any position in the factory FC is represented by the coordinates of X, Y, and Z in the global coordinate system GC. The factory FC includes a first place PL1 and a second place PL2. The first place PL1 and the second place PL2 are connected by a passage TR on which the vehicle 100 can travel. In the factory FC, a plurality of external cameras 300 are arranged along the passage TR. The position of each external camera 300 in the factory FC is adjusted in advance. The vehicle 100 moves from the first place PL1 to the second place PL2 through the passage TR by unmanned driving.

[0013] Figure 2 2 is a block diagram showing the structure of the information processing system 50. The vehicle 100 includes a vehicle control device 110 for controlling various parts of the vehicle 100, an actuator group 120 including one or more actuators driven under the control of the vehicle control device 110, and a communication device 130 for communicating with an external device such as the information processing device 200 by wireless communication. The actuator group 120 includes an actuator of a driving device for accelerating the vehicle 100, an actuator of a steering device for changing the traveling direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100.

[0014] The vehicle control device 110 is composed of a computer including a processor 111, a memory 112, an input / output interface 113, and an internal bus 114. The processor 111, the memory 112, and the input / output interface 113 are connected via the internal bus 114 in a manner capable of bidirectional communication. The input / output interface 113 is connected to an actuator group 120 and a communication device 130. The processor 111 functions as a vehicle control unit 115 by executing a program PG1 stored in the memory 112.

[0015] The vehicle control unit 115 drives the vehicle 100 by controlling the actuator group 120. The vehicle control unit 115 controls the actuator group 120 using the driving control signal received from the information processing device 200, thereby enabling the vehicle 100 to drive. The driving control signal is a control signal for driving the vehicle 100. The driving control signal specifies the action of the vehicle 100 in order to move the vehicle 100 by unmanned driving. In the present embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 100 as parameters. In other embodiments, the driving control signal may include the speed of the vehicle 100 as a parameter instead of or in addition to the acceleration of the vehicle 100.

[0016] The information processing device 200 is used to drive the vehicle 100 by unmanned driving. The information processing device 200 is, for example, a server installed in a different place from the vehicle 100. The information processing device 200 causes the vehicle 100 to drive by unmanned driving by learning the machine learning model DM or obtaining and using the learned machine learning model DM. When the machine learning model DM is input with a captured image, it outputs at least one of a driving control signal and a signal generation parameter. The captured image is captured by using an external camera 300 to capture the image including Figure 1 The passage TR on which the vehicle 100 shown in FIG. 1 travels, the surrounding area SA of the passage TR, and the photographed area of ​​the vehicle 100 traveling on the passage TR are obtained. The photographed image is any one of the original image as the detection result output from the external camera 300 and the processed image after processing the original image. The signal generation parameter is a parameter used when generating the driving control signal. The signal generation parameter is, for example, any one of the vehicle position information, the coordinate value for obtaining at least one of the position and orientation of the vehicle 100, and the shape data for obtaining at least one of the position and orientation of the vehicle 100. The vehicle position information is the position information that becomes the basis for generating the driving control signal. In the present embodiment, the vehicle position information includes the position and orientation of the vehicle 100 in the global coordinate system GC of the factory FC. The shape data is data indicating the appearance shape of the vehicle 100. The shape data is generated by detecting the vehicle 100 from the photographed image. The shape data is, for example, a mask image in which the mask area is added to the photographed image by masking the area indicating the vehicle 100 in each area constituting the photographed image. Each area constituting the photographed image is, for example, one pixel constituting the photographed image.

[0017] like Figure 2 As shown, in the present embodiment, in order to make the vehicle 100 travel by unmanned driving, a first learning model DM1 as a machine learning model DM is used. When a captured image is input, the first learning model DM1 outputs shape data as a parameter for signal generation. Thus, the information processing device 200 obtains shape data by inputting the captured image into the first learning model DM1. In addition, the information processing device 200 obtains vehicle position information using the shape data. In addition, the information processing device 200 generates a travel control signal using the vehicle position information, etc., and sends it to the vehicle 100. Thus, the information processing device 200 causes the vehicle 100 to travel by remote control.

[0018] Here, sometimes the appearance state in the shooting area of ​​a part of the external cameras 300 among the multiple external cameras 300 changes from the time of learning the first learning model DM1. In this case, the detection accuracy when detecting the vehicle 100 from the captured images obtained by the external cameras 300 among the multiple external cameras 300 may decrease. Therefore, when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images obtained by the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, the information processing device 200 performs the following processing. In this case, the information processing device 200 performs a re-learning process of re-learning the first learning model DM1 that has been learned using a training image corresponding to the shooting area of ​​the external camera 300 that obtained the captured image with an accuracy less than the threshold. Relearning is an example of additional learning. Therefore, the re-learning process is an example of additional learning process.

[0019] It should be noted that the factors that affect the change of the appearance state in the shooting area are, for example, dirt generated in the object area TA. Figure 1 As shown, the target area TA is an area including the passage TR on which the vehicle 100 travels and the surrounding area SA of the passage TR. The element that affects the change of the appearance state in the shooting area may also be the concavity and convexity generated in the target area TA. The concavity and convexity generated in the target area TA may be, for example, the depression generated in the passage TR due to the thinning of the road surface Rs due to the increase in the number of times the vehicle 100 travels. The concavity and convexity generated in the target area TA may also be the protrusion generated in the passage TR due to the bulging of the road surface Rs. In addition, the element that affects the change of the appearance state in the shooting area may also be an object different from the vehicle 100 that is arranged in a movable manner in the target area TA. The object that is arranged in a movable manner in the target area TA may be, for example, a manufacturing device, a road cone, a billboard, and other tools that can be appropriately changed in configuration. The object that is arranged in a movable manner in the target area TA may also be a flying object such as a fallen leaf. In addition, the element that affects the change of the appearance state in the shooting area may also be the road surface color of the passage TR. The road surface color of the passage TR may be, for example, the color of a demarcation line such as a white line, the paint color of the ground including the passage TR, or the color of the road surface Rs after paving. In addition, the factor affecting the change of the appearance state in the shooting area may also be the illumination condition of the lighting illuminating the passage TR. The illumination condition changes due to, for example, the difference in illumination according to the accumulated lighting time from the start time of the use of the lighting, the decrease in illumination due to the stop of power supply caused by a power outage, etc.

[0020] The information processing device 200 is composed of a computer including a processor 201, a memory 202, an input / output interface 203, and an internal bus 204. The processor 201, the memory 202, and the input / output interface 203 are connected via the internal bus 204 in a manner capable of bidirectional communication. A communication device 205 for communicating with various devices outside the information processing device 200 is connected to the input / output interface 203. The communication device 205 can communicate with the vehicle 100 by wireless communication, and can communicate with each external camera 300 by wired communication or wireless communication.

[0021] The memory 202 stores various information including a program PG2 for controlling the operation of the information processing device 200, a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300. The memory 202 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0022] The first learning model DM1 is a machine learning model DM that has been learned using artificial intelligence by detecting the appearance shape of the vehicle 100 from the captured image and outputting shape data. The first learning model DM1 is pre-learned by supervised learning using the first initial learning data set. The first initial learning data set includes a plurality of first initial learning data sets corresponding to each of the captured areas of the plurality of external cameras 300. In the plurality of first initial learning data sets, a training image including a background image and a vehicle image corresponding to the captured area of ​​the external camera 300 and a region correct answer label are respectively associated. The background image is an image representing the object area TA. The background image does not include the vehicle 100, but includes the object area TA. The vehicle image is an image representing the vehicle 100. In the present embodiment, the training image is a captured image obtained by capturing the vehicle 100 that is performing unmanned driving control using the external camera 300. The region correct answer label indicates whether each region in the training image is a region representing the vehicle 100 or a region representing other than the vehicle 100. The region correct answer label is a correct answer label corresponding to the shape data output from the first learning model DM1. For the first learning model DM1, for example, a convolutional neural network (hereinafter referred to as CNN) that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, back propagation (error back propagation method) is used to update the parameters of the CNN in a manner that reduces the error between the output result of the first learning model DM1 and the correct label of the region.

[0023] The processor 201 executes the program PG2 stored in the memory 202 to function as an accuracy acquisition unit 211 , a learning data acquisition unit 212 , a learning unit 213 , a control information acquisition unit 214 , a position acquisition unit 215 , a signal generation unit 216 , and a transmission unit 217 .

[0024] The accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 pre-learned using the first initial learning data set. The accuracy of the first learning model DM1 is acquired by comparing the output result of the first learning model DM1 with the correct label of the region. The accuracy of the first learning model DM1 is, for example, any one of the accuracy, precision, recall, and F value of the first learning model DM1. In the present embodiment, the first learning model DM1 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211 acquires the accuracy of each of the multiple first learning models DM1 corresponding to each external camera 300.

[0025] The learning data acquisition unit 212 performs the following processing when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images acquired by a part of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning data acquisition unit 212 acquires a first additional learning data set for relearning the first learning model DM1 as additional learning. The first additional learning data set includes one or more first additional learning data sets that establish corresponding training images and regional correct answer labels including background images and vehicle images corresponding to the captured areas of the external cameras 300 that acquired the captured images with an accuracy less than the threshold value. In the present embodiment, the first learning model DM1 is prepared for each external camera 300. Thus, the learning data acquisition unit 212 acquires the first additional learning data set for relearning the first learning model DM1 with an accuracy less than the threshold value.

[0026] The learning unit 213 performs the following processing when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured image obtained by a part of the external cameras 300 among the multiple external cameras 300 is input to the first learning model DM1. In this case, the learning unit 213 performs a relearning process of relearning the first learning model DM1. As a relearning process, the learning unit 213 updates, for example, the trained parameters of at least a part of the first learning model DM1. As a relearning process, the learning unit 213 may also perform a process of adding a layer of the CNN constituting the first learning model DM1. In the present embodiment, the first learning model DM1 is prepared for each external camera 300. Thus, the learning unit 213 relearns the first learning model DM1 corresponding to the external camera 300 that has obtained the captured image with an accuracy less than the threshold value using the first additional learning data group obtained. At this time, the learning unit 213 performs the relearning process during the execution of the unmanned driving control.

[0027] The control information acquisition unit 214 acquires control information of at least one of the driving control signal and the signal generation parameter by inputting the captured image to the first learning model DM1. In the present embodiment, the first learning model DM1 is prepared for each external camera 300. In addition, the first learning model DM1 is a model that outputs shape data as a signal generation parameter. Thus, the control information acquisition unit 214 acquires shape data by inputting the captured image to the first learning model DM1 corresponding to the external camera 300 that acquired the captured image. At this time, when the relearning process has been completed, the control information acquisition unit 214 acquires shape data by inputting the captured image to the first learning model DM1 after the relearning process.

[0028] The position acquisition unit 215 acquires the vehicle position information. In the present embodiment, the position acquisition unit 215 acquires the vehicle position information using shape data. In the present embodiment, the position of the vehicle 100 is the position of a measuring point preset for a specific part of the vehicle 100. The orientation of the vehicle 100 is represented by a direction indicated by a vector extending from the rear side to the front side of the vehicle 100 along the front-rear axis passing through the center of gravity of the vehicle 100.

[0029] The signal generating unit 216 generates a travel control signal for controlling the actuator group 120 of the vehicle 100. The transmitting unit 217 transmits the travel control signal to the vehicle 100. Thus, the information processing device 200 causes the vehicle 100 to travel by remote control. In addition to the travel control signal, the processor 201 of the information processing device 200 may generate and output a control signal for controlling an actuator that operates various equipment such as various auxiliary machines, wipers, power windows, and lights provided in the vehicle 100. That is, the processor 201 of the information processing device 200 may also cause such various equipment and various auxiliary machines to operate by remote control.

[0030] A-2. Vehicle driving control method: Figure 3 1 is a flowchart showing a processing procedure of the travel control of the vehicle 100 in the first embodiment. Figure 3 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0031] In step S111, the position acquisition unit 215 acquires the vehicle position information of the vehicle 100 using the detection result output from the external camera 300. Specifically, in step S111, the position acquisition unit 215 acquires the vehicle position information using the captured image acquired from the camera serving as the external camera 300.

[0032] Specifically, in step S111, the position acquisition unit 215 calculates the coordinates of the measuring point of the vehicle 100 in the coordinate system of the captured image, that is, the local coordinate system, using the shape data obtained by inputting the captured image to the first learning model DM1, and transforms the calculated coordinates into coordinates in the global coordinate system GC, thereby acquiring the position of the vehicle 100. In addition, the position acquisition unit 215 calculates the orientation of the vehicle 100 based on the orientation of the movement vector of the vehicle 100 calculated from the position change of the feature point of the vehicle 100 between the frames of the captured image, for example, using the optical flow method, thereby acquiring the orientation of the vehicle 100.

[0033] In step S112, the signal generating unit 216 determines the target position to which the vehicle 100 should go next. In the present embodiment, the target position is represented by the coordinates of X, Y, and Z in the global coordinate system GC. The reference path RR, which is the path that the vehicle 100 should travel, is pre-stored in the memory 202 of the information processing device 200. The path is represented by a node representing a departure point, a node representing a passing point, a node representing a destination, and a road section connecting each node. The signal generating unit 216 uses the vehicle position information and the reference path RR to determine the target position to which the vehicle 100 should go next. The signal generating unit 216 determines the target position on the reference path RR that is ahead of the current position of the vehicle 100.

[0034] In step S113, the signal generating unit 216 generates a driving control signal for driving the vehicle 100 toward the determined target position. The signal generating unit 216 calculates the driving speed of the vehicle 100 according to the change in the position of the vehicle 100, and compares the calculated driving speed with the target speed. As a whole, the signal generating unit 216 determines the acceleration in a manner to accelerate the vehicle 100 when the driving speed is lower than the target speed, and determines the acceleration in a manner to decelerate the vehicle 100 when the driving speed is higher than the target speed. In addition, when the vehicle 100 is located on the reference path RR, the signal generating unit 216 determines the steering angle and the acceleration in a manner to prevent the vehicle 100 from deviating from the reference path RR, and when the vehicle 100 is not located on the reference path RR, in other words, when the vehicle 100 deviates from the reference path RR, the signal generating unit 216 determines the steering angle and the acceleration in a manner to return the vehicle 100 to the reference path RR.

[0035] In step S114, the transmission unit 217 transmits the generated travel control signal to the vehicle 100. The processor 201 repeatedly acquires the position of the vehicle 100, determines the target position, generates the travel control signal, transmits the travel control signal, and the like in a predetermined cycle.

[0036] In step S115, the vehicle control unit 115 of the vehicle 100 receives the travel control signal transmitted from the information processing device 200. In step S116, the vehicle control unit 115 controls the actuator group 120 using the received travel control signal to drive the vehicle 100 at the acceleration and steering angle indicated by the travel control signal. The processor 111 of the vehicle 100 repeatedly receives the travel control signal and controls the actuator group 120 in a predetermined cycle. According to the information processing system 50 in this embodiment, the vehicle 100 can be driven by remote control, and the vehicle 100 can be moved without using a transport device such as a crane or a conveyor.

[0037] A-3. Information processing methods: Figure 4 : is a flowchart showing the learning processing method in the first embodiment. Figure 4 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0038] In the accuracy acquisition process, the accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 pre-learned using the first initial learning data group (step S211). When the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured image acquired by the external camera 300 of a part of the multiple external cameras 300 is input into the first learning model DM1 (step S212: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212 acquires the first additional learning data group (step S213). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213 performs a re-learning process of re-learning the first learning model DM1 using the acquired first additional learning data group (step S214). On the other hand, when the accuracy of the first learning model DM1 becomes above a predetermined threshold when the captured images obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1 (step S212: No), the processor 201 does not execute the learning data acquisition process and the learning process, and ends the process.

[0039] Figure 5 This is a flowchart showing the method of acquiring shape data in the first embodiment. Figure 5 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0040] In the control information acquisition step, the control information acquisition unit 214 acquires shape data by performing the following processing. Specifically, when the precision of the first learning model DM1 is less than a predetermined threshold value when the captured image acquired by a part of the external cameras 300 among the plurality of external cameras 300 is input to the first learning model DM1 (step S311: Yes), and when the relearning process is completed (step S312: Yes), the control information acquisition unit 214 acquires shape data by inputting the captured image to the first learning model DM1 after the relearning process (step S313). On the other hand, when the precision of the first learning model DM1 becomes greater than a predetermined threshold value when the captured image acquired by a part of the external cameras 300 among the plurality of external cameras 300 is input to the first learning model DM1 (step S311: No), the control information acquisition unit 214 acquires shape data by inputting the captured image to the first learning model DM1 that has not been relearned (step S314).

[0041] A-4. How to obtain the vehicle's position: Figure 6 This is a flowchart showing an example of a method for generating shape data and a method for acquiring the position of the vehicle 100 using the shape data. Figure 7 It shows that the Figure 6 Schematic diagram of examples of various images Im1 to Im6 when using the method shown. Figure 7 In the Figure 6 In the present embodiment, the measuring point 10e of the vehicle 100 is the rear end of the left side surface of the vehicle 100. It should be noted that the measuring point 10e of the vehicle 100 may be other than the rear end of the left side surface of the vehicle 100.

[0042] In the image acquisition process (step S410), the control information acquisition unit 214 acquires the original image Im1 output from the external camera 300. In the distortion correction process (step S420), the control information acquisition unit 214 generates a corrected image Im2 as a processed image by correcting the distortion of the original image Im1. In the rotation process process (step S430), the control information acquisition unit 214 rotates the corrected image Im2. Thus, the control information acquisition unit 214 generates a rotated image Im3 as a processed image. In the cropping process process (step S440), the control information acquisition unit 214 deletes the unnecessary area A2 other than the necessary area A1 composed of the vehicle 100 and the surrounding area of ​​the vehicle 100 from the rotated image Im3 among the areas constituting the rotated image Im3. Thus, the control information acquisition unit 214 generates a processed image Im4 as a processed image. Each of the steps S420 to S440 is a pre-processing step for improving the detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4. Therefore, at least any one of the steps S420 to S440 may be omitted.

[0043] In the detection process (step S450), the control information acquisition unit 214 inputs the processed image Im4 to the first learning model DM1. Thus, the control information acquisition unit 214 detects the vehicle 100 from the processed image Im4 and acquires the first mask image Im5 as the shape data Da. The first mask image Im5 is an image in which the mask region Ms is added to the processed image Im4 by masking the region representing the vehicle 100 among the regions constituting the processed image Im4.

[0044] In the perspective transformation process (step S460), the position acquisition unit 215 generates the second mask image Im6 by performing perspective transformation on the first mask image Im5. The position acquisition unit 215 uses, for example, predetermined perspective transformation parameters to perspective transform the first mask image Im5 into a bird's-eye view image when viewed from a viewpoint above the vehicle 100 that is approximately perpendicular to the road surface Rs. The perspective transformation parameters are, for example, parameters related to the position information and internal parameters of the external camera 300 obtained by calibration. Thus, the position acquisition unit 215 generates the second mask image Im6 represented by the image coordinate system based on the first mask image Im5 represented by the camera coordinate system. The camera coordinate system is a coordinate system having the focus of the external camera 300 as the origin and having coordinate axes represented by the Xc axis and the Yc axis orthogonal to the Xc axis. The image coordinate system is a coordinate system having a point on the image plane as the origin and having coordinate axes represented by the Xi axis and the Yi axis orthogonal to the Xi axis.

[0045] In the coordinate point calculation step (step S470 ), the position acquisition unit 215 calculates an image coordinate point indicating the position of the vehicle 100 in the image coordinate system. Figure 8 2 is a diagram for explaining the details of the coordinate point calculation process. To calculate the image coordinate point, the position acquisition unit 215 first calculates the base coordinate point P0 based on the first circumscribed rectangle R1 set for the mask region Ms in the first mask image Im5 which is the image before perspective transformation. Fig. 9 : is a diagram for explaining a method of calculating the basic coordinate point P0. In order to calculate the basic coordinate point P0, the position acquisition unit 215 sets a basic circumscribed rectangle R0 for the mask area Ms in the first mask image Im5. Next, the position acquisition unit 215 rotates the first mask image Im5 by a required amount of rotation with the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 100 corresponding to the mask area Ms in the first mask image Im5 is oriented in a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device on which the first mask image Im5 is displayed. Next, the position acquisition unit 215 sets a first circumscribed rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the movement vector V. Next, the position acquisition unit 215 reversely rotates the first mask image Im5 to which the first circumscribed rectangle R1 is attached by the above-mentioned amount of rotation with the center of gravity C of the mask area Ms as the rotation center. Thus, the position acquisition unit 215 uses the coordinate point of one vertex having the coordinates closest to the position measurement point 10 e of the vehicle 100 among the four vertices of the first circumscribed rectangle R1 as the base coordinate point P0 .

[0046] Then, if Figure 8 As shown, the position acquisition unit 215 performs perspective transformation on the first mask image Im5 after reverse rotation, that is, the first mask image Im5 after calculating the basic coordinate point P0. Thus, the position acquisition unit 215 uses the coordinate point corresponding to the basic coordinate point P0 as the first coordinate point P1 in the first circumscribed rectangle R1 deformed by the perspective transformation.

[0047] Next, the position acquisition unit 215 sets a second circumscribed rectangle R2 for the mask region Ms in the second mask image Im6 after perspective transformation of the first mask image Im5. Then, the position acquisition unit 215 sets the vertex of the second circumscribed rectangle R2 that indicates the same position as the first coordinate point P1 as the second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points that indicate the same position, and therefore have a correlation relationship with each other.

[0048] Next, the position acquisition unit 215 corrects and replaces the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 according to the size relationship between the coordinate values ​​of the first coordinate point P1 and the coordinate values ​​of the second coordinate point P2. When the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is larger than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the position acquisition unit 215 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 in the Yi direction of the first coordinate point P1 is larger than the coordinate value Yi2 in the Yi direction of the second coordinate point P2 (Yi1>Yi2), the position acquisition unit 215 replaces the coordinate value Yi1 in the Yi direction of the first coordinate point P1 with the coordinate value Yi2 in the Yi direction of the second coordinate point P2. Figure 8 In the example shown, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. In addition, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. As a result, the image coordinate point P3 has the coordinates (Xi2, Yi1). In this way, the position acquisition unit 215 calculates the image coordinate point P3 indicating the position of the vehicle 100 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2.

[0049] In the position conversion process (step S480), the position acquisition unit 215 calculates the vehicle coordinate point representing the position of the measuring point 10e of the vehicle 100 in the global coordinate system GC by converting the image coordinate point P3 into the vehicle coordinate point. The position acquisition unit 215 converts the image coordinate point P3 into the vehicle coordinate point using the relational expressions (1) to (3) described later, which include the vehicle coordinate point as the target variable and include the image coordinate point P3, the shooting parameter, and the vehicle parameter as the explanatory variables. The shooting parameter is a parameter related to the distance of the external camera 300 from a predetermined reference point. In the present embodiment, the shooting parameter is the height H (described later) of the external camera 300 from the road surface Rs. Fig.10 The vehicle parameter is a parameter related to the distance between the measuring point 10e of the vehicle 100 and the reference point. In the present embodiment, the vehicle parameter is the height h (described later) of the measuring point 10e of the vehicle 100 from the road surface Rs. Fig.10 ).

[0050] Fig.10 This is the first figure for explaining the details of the position conversion process. Fig.10 , the vehicle 100 is shown as viewed from the left side. Fig.11 FIG. 2 is a second diagram for explaining the details of the position conversion process. Fig.11 , the vehicle 100 is shown as viewed from the roof side. Fig.10 and Fig.11 The global coordinate system GC shown is a coordinate system having a fixed coordinate point Pf representing an arbitrary reference position on the road surface Rs as an origin and having coordinate axes represented by an Xg axis and a Yg axis orthogonal to the Xg axis. The photographing coordinate point Pc is a coordinate point representing the position of the external camera 300 that outputs the original image Im1 used in calculating the image coordinate point P3 in the global coordinate system GC. The fixed coordinate point Pf and the photographing coordinate point Pc are pre-stored in the memory 202 of the information processing device 200, for example.

[0051] like Fig.10 As shown in FIG. 1 , the observation distance between the position of the external camera 300 and the position of the vehicle 100 (image coordinate point P3) on the XgYg plane is set to Do. The observation error is set to ΔD. The height [m] of the external camera 300 from the road surface Rs as a shooting parameter is set to H. The height [m] of the measuring point 10e of the vehicle 100 from the road surface Rs as a vehicle parameter is set to h. In this case, the observation error ΔD can be expressed by the following formula (1). ΔD=h / H×Do Formula (1) That is, the larger the observation distance Do is, the larger the observation error ΔD is.

[0052] When the actual distance between the position of the external camera 300 and the position of the measuring point 10 e of the vehicle 100 is defined as a first distance D, the first distance D can be represented by the following equation (2). D=Do×(1-h / H) Formula (2) That is, the first distance D is determined by the observation distance Do, the height H of the external camera 300 as an imaging parameter, and the height h of the measurement point 10 e of the vehicle 100 as a vehicle parameter.

[0053] like Fig.11 As shown, when the distance between the reference position and the estimated position of the vehicle 100 is set as the estimated distance Dp and the actual distance between the reference position and the vehicle 100 is set as the second distance Dt, the second distance Dt can be expressed by the following formula (3). Dt=Dp×(1-h / H) Formula (3)

[0054] Here, the estimated distance Dp can be calculated using the third distance Dc, the image coordinate point P3, and the fixed coordinate point Pf. The third distance Dc is an actual distance obtained from the fixed coordinate point Pf and the photographed coordinate point Pc. Therefore, the position acquisition unit 215 calculates the vehicle coordinate point Pv using the second distance Dt obtained by correcting the estimated distance Dp using the above formula (3) and the fixed coordinate point Pf. At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the measuring point 10e of the vehicle 100 in the global coordinate system GC, and therefore corresponds to the position of the vehicle 100 in the actual space.

[0055] According to the first embodiment, a first learning model DM1 pre-learned using a plurality of first initial learning data sets corresponding to each of the photographing areas of the plurality of external cameras 300 is prepared, wherein the plurality of first initial learning data sets associate training images including background images and vehicle images corresponding to the photographing areas with regional correct labels. When the precision of the photographed images Im1 to Im4 acquired by a part of the plurality of external cameras 300 is less than a threshold when input to the first learning model DM1 that has been learned, the information processing device 200 can perform the following processing. In this case, the information processing device 200 can re-learn the first learning model DM1 using a first additional learning data set associated with training images including background images and vehicle images corresponding to the photographing areas of the external cameras 300 that acquired the photographed images Im1 to Im4 with a precision less than the threshold, and regional correct labels. In this way, it is possible to suppress a decrease in the detection precision when detecting the vehicle 100 from the photographed images Im1 to Im4 acquired by a part of the plurality of external cameras 300. Thus, when the appearance state in the imaging area of ​​some of the external cameras 300 among the plurality of external cameras 300 has changed since the time of learning the first learning model DM1 , it is possible to suppress a decrease in the accuracy of the first learning model DM1 .

[0056] In addition, according to the first embodiment, the information processing device 200 stores a plurality of first learning models DM1 prepared for the plurality of external cameras 300. In this way, when the accuracy of the captured images Im1 to Im4 obtained by a part of the plurality of external cameras 300 is less than a threshold when input to the first learning model DM1 that has been learned, the information processing device 200 can perform the following processing. In this case, the information processing device 200 can relearn only the first learning model DM1 corresponding to the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold. In this way, the load required for relearning the first learning model DM1 can be reduced.

[0057] In addition, according to the first embodiment, when the relearning process is completed, the information processing device 200 can obtain the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the relearning process. This can suppress the decrease in the accuracy of the shape data Da generated using the captured images Im1 to Im4 acquired by the external cameras 300 of a part of the plurality of external cameras 300.

[0058] In addition, according to the first embodiment, the information processing device 200 can relearn the first learning model DM1 during the execution of unmanned driving control. It should be noted that in other embodiments, the information processing device 200 may not execute unmanned driving control, for example, relearn the first learning model DM1 during the execution of manned driving control.

[0059] According to the first embodiment, the vehicle 100 is driven unmanned in the factory. In this way, the information processing device 200 can relearn the first learning model DM1 during the manufacturing process of the vehicle 100.

[0060] B. Second Implementation Method: Fig.12 : is a block diagram showing the structure of the information processing system 50a in the second embodiment. The information processing system 50a includes one or more vehicles 100, multiple external cameras 300, and an information processing device 200a. In the present embodiment, when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, the information processing device 200a performs a transfer learning process of causing the first learning model DM1 to perform transfer learning as additional learning. Transfer learning is an example of additional learning. Therefore, the transfer learning process is an example of additional learning processing. The other structures of the information processing system 50a are the same as those of the first embodiment unless otherwise specified. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0061] The information processing device 200 a is constituted by a computer including a processor 201 a , a memory 202 a , an input / output interface 203 , and an internal bus 204 .

[0062] The memory 202 a stores various information including a program PG2 a for controlling the operation of the information processing device 200 a , a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300 .

[0063] The processor 201a functions as an accuracy acquisition unit 211, a learning data acquisition unit 212a, a learning unit 213a, a control information acquisition unit 214a, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217 by executing a program PG2a stored in the memory 202a.

[0064] The learning data acquisition unit 212a performs the following processing when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1. In this case, the learning data acquisition unit 212a acquires a first additional learning data group for performing transfer learning as additional learning on the first learning model DM1 whose accuracy is less than the threshold value.

[0065] The learning unit 213a performs the following processing when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning unit 213a performs a transfer learning process of transfer learning the first learning model DM1 whose accuracy is less than the threshold value. As a transfer learning process, the learning unit 213a performs the following processing, for example. Specifically, the learning unit 213a adds a layer of the CNN that constitutes the first learning model DM1 whose accuracy is less than the threshold value. In addition, the learning unit 213a uses the obtained first additional learning data set to update the parameters related to the added layer without updating the trained parameters.

[0066] When the transfer learning process is completed, the control information acquisition unit 214 a acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the transfer learning process.

[0067] Fig.13 : is a flowchart showing the learning processing method in the second embodiment. Fig.13 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0068] In the accuracy acquisition process, the accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 pre-learned using the first initial learning data group (step S221). When the captured images Im1~Im4 acquired by the external cameras 300 of a part of the multiple external cameras 300 are input into the first learning model DM1, when the accuracy of the first learning model DM1 is less than a predetermined threshold value (step S222: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212a acquires the first additional learning data group (step S223). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213a performs transfer learning processing for transfer learning of the first learning model DM1 using the acquired first additional learning data group (step S224). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1 and the accuracy of the first learning model DM1 becomes above a predetermined threshold (step S222: No), the processor 201a does not execute the learning data acquisition process and the learning process, and ends the process.

[0069] Fig.14 This is a flowchart showing a method of acquiring shape data Da in the second embodiment. Fig.14 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0070] In the control information acquisition step, the control information acquisition unit 214a acquires the shape data Da by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1, if the accuracy of the first learning model DM1 is less than a predetermined threshold value (step S321: Yes), and when the transfer learning process is completed (step S322: Yes), the control information acquisition unit 214a acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the transfer learning process (step S323). On the other hand, when the accuracy of the first learning model DM1 becomes above a predetermined threshold when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1 (step S321: No), the control information acquisition unit 214a acquires the shape data Da (step S324) by inputting the captured images Im1 to Im4 into the first learning model DM1 on which the transfer learning process has not been performed.

[0071] According to the second embodiment, the information processing device 200a can perform the following processing when the accuracy of the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the plurality of external cameras 300 is less than a threshold when input to the first learning model DM1 that has been learned. In this case, the information processing device 200a can transfer learning of the first learning model DM1 using a training image and a first additional learning data set in which the training image and the region correct answer label are associated, which include a background image and a vehicle image corresponding to the captured area of ​​the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold. In this way, it is possible to suppress a decrease in the detection accuracy when detecting the vehicle 100 from the captured images Im1 to Im4 obtained by a part of the external cameras 300. Thus, it is possible to suppress a decrease in the accuracy of the shape data Da generated using the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the plurality of external cameras 300.

[0072] C. Third Implementation Method: Fig.15 : is a block diagram showing the structure of the information processing system 50b in the third embodiment. The information processing system 50b includes one or more vehicles 100, multiple external cameras 300, and an information processing device 200b. In this embodiment, when the relearning process is completed and the accuracy of the first learning model DM1 after the relearning process is above the threshold, the information processing device 200b inputs the captured images Im1~Im4 to the first learning model DM1 after the relearning process to obtain the shape data Da. The other structures of the information processing system 50b are the same as those of the first embodiment unless otherwise specified. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0073] The information processing device 200 b is constituted by a computer including a processor 201 b , a memory 202 b , an input / output interface 203 , and an internal bus 204 .

[0074] The memory 202 b stores various information including a program PG2 b for controlling the operation of the information processing device 200 b , a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300 .

[0075] The processor 201b functions as an accuracy acquisition unit 211b, a learning data acquisition unit 212b, a learning unit 213b, a control information acquisition unit 214b, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217 by executing a program PG2b stored in the memory 202b.

[0076] The accuracy acquisition unit 211b acquires the accuracy of the first learning model DM1 pre-learned using the first initial learning data set. Moreover, the accuracy acquisition unit 211b acquires the accuracy of the first learning model DM1 after the re-learning process when the re-learning process has been completed. Hereinafter, the accuracy of the first learning model DM1 pre-learned using the first initial learning data set is also referred to as "initial accuracy". The accuracy of the first learning model DM1 after the re-learning process is also referred to as "post-learning accuracy". In this embodiment, the accuracy acquisition unit 211b uses a plurality of captured images Im1 to Im4 obtained by using an external camera 300 to capture a subsequent vehicle 100 traveling behind the vehicle 100 in the training image included in the first initial learning data set to acquire the post-learning accuracy.

[0077] The learning data acquisition unit 212b acquires the first additional learning data set when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1. In addition, the learning data acquisition unit 212b acquires the first additional learning data set again when the relearning process is completed and the accuracy after learning is less than the threshold value. At this time, the reacquired first additional learning data set may include the same training images as the training images used in the relearning process in which the accuracy after learning is less than the threshold value, or may include different training images.

[0078] The learning unit 213b performs a relearning process for relearning the first learning model DM1 when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1. Furthermore, when the relearning process is completed and the accuracy after learning is less than the threshold value, the learning unit 213b performs a relearning process for relearning the first learning model DM1 using the reacquired first additional learning data set.

[0079] When the relearning process is completed and the accuracy after learning is greater than the threshold, the control information acquisition unit 214b inputs the captured images Im1 to Im4 to the first learning model DM1 after the relearning process to acquire the shape data Da. At this time, when the ratio of the number of captured images Im1 to Im4 whose accuracy after learning is greater than the threshold to the total number of the plurality of captured images Im1 to Im4 used to calculate the accuracy after learning is greater than a predetermined reference value, the control information acquisition unit 214b acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the relearning process.

[0080] Fig.16: is a flowchart showing the learning processing method in the third embodiment. Fig.16 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0081] In the accuracy acquisition process, the accuracy acquisition unit 211b acquires the initial accuracy (step S231). When the initial accuracy of the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 is input into the first learning model DM1, it is less than a predetermined threshold value (step S232: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212b acquires the first additional learning data group (step S233). Following the learning data acquisition process, the learning process is executed. In the first learning process, the learning unit 213b performs a re-learning process of re-learning the first learning model DM1 using the acquired first additional learning data group (step S234). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1 and the accuracy of the first learning model DM1 becomes above a predetermined threshold (step S232: No), the processor 201b does not execute the learning data acquisition process and the learning process, and ends the process.

[0082] When the relearning process is completed (step S235: Yes), the accuracy confirmation process is performed. In the accuracy confirmation process, the accuracy acquisition unit 211b acquires the post-learning accuracy (step S236). When the ratio of the number of captured images Im1 to Im4 whose post-learning accuracy is above the threshold to the total number of the plurality of captured images Im1 to Im4 used to calculate the post-learning accuracy is above a predetermined reference value (step S237: Yes), the process is terminated. On the other hand, when the ratio of the number of captured images Im1 to Im4 whose post-learning accuracy is above the threshold to the total number of the plurality of captured images Im1 to Im4 used to calculate the post-learning accuracy is less than a predetermined reference value (step S237: No), each step from step S233 to step S237 is repeatedly performed.

[0083] Fig.17 This is a flowchart showing a method of acquiring shape data Da in the third embodiment. Fig.17 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0084] When the initial accuracy of the captured images Im1 to Im4 acquired by a part of the external cameras 300 is input to the first learning model DM1 and is less than a predetermined threshold value (step S331: Yes), and when the relearning process is completed (step S332: Yes), the accuracy acquisition unit 211b acquires the post-learning accuracy (step S333). When the ratio of the number of captured images Im1 to Im4 whose post-learning accuracy is greater than the threshold value relative to the total number of the plurality of captured images Im1 to Im4 used to calculate the post-learning accuracy is greater than a predetermined reference value (step S334: Yes), the control information acquisition unit 214b acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the relearning process (step S335). On the other hand, when the initial accuracy of captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 becomes above a predetermined threshold when being input into the first learning model DM1 (step S331: No), the control information acquisition unit 214b acquires shape data Da (step S336) by inputting captured images Im1 to Im4 into the first learning model DM1 on which no re-learning processing is performed.

[0085] According to the third embodiment, when the relearning process is completed and the post-learning accuracy is equal to or greater than the threshold, the information processing device 200b can input the captured images Im1 to Im4 to the first learning model DM1 after the relearning process to acquire the shape data Da.

[0086] According to the third embodiment, the information processing device 200b can obtain the post-learning accuracy using a plurality of captured images Im1 to Im4 obtained by using the external camera 300 to capture the vehicle 100 traveling behind the vehicle 100 in the training image included in the first initial learning data set.

[0087] In addition, according to the above-mentioned third embodiment, the information processing device 200b can determine that the accuracy after learning has become above the threshold when the ratio of the number of captured images Im1~Im4 whose accuracy after learning has become above the threshold to the total number of multiple captured images Im1~Im4 used to calculate the accuracy after learning is above a predetermined reference value.

[0088] D. Fourth Implementation Method: Fig.18: is a block diagram showing the structure of the information processing system 50c in the fourth embodiment. The information processing system 50c includes one or more vehicles 100, multiple external cameras 300, and an information processing device 200c. In this embodiment, the information processing device 200c obtains rectangular coordinate values ​​as parameters for signal generation by inputting captured images Im1 to Im4 to the second learning model DM2 as the machine learning model DM. The other structures of the information processing system 50c are the same as those of the first embodiment unless otherwise specified. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0089] Fig.19 This is a diagram for explaining the rectangular coordinate values ​​Cr1 to Cr4. The rectangular coordinate values ​​Cr1 to Cr4 are the coordinate values ​​of the four vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc. The vehicle circumscribed rectangle Rc is a quadrilateral set in the image Ti, Im1 to Im4 so as to surround the area occupied by the vehicle 100 when the vehicle 100 is projected onto the road surface Rs. The rectangular coordinate values ​​Cr1 to Cr4 are used to calculate at least one of the position and orientation of the vehicle 100.

[0090] like Fig.18 As shown, the information processing device 200 c is composed of a computer including a processor 201 c , a memory 202 c , an input / output interface 203 , and an internal bus 204 .

[0091] The memory 202 c stores various information including a program PG2 c for controlling the operation of the information processing device 200 c , a reference path RR, a plurality of second learning models DM2 prepared for each external camera 300 , a rectangle database Db1 , and camera parameters Pa.

[0092] The second learning model DM2 outputs rectangular coordinate values ​​Cr1 to Cr4 when the captured images Im1 to Im4 are input. The second learning model DM2 is pre-learned by supervised learning using the second initial learning data set. The second initial learning data set includes a plurality of second initial learning data sets corresponding to each of the captured areas of the plurality of external cameras 300. In the plurality of second initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the captured area of ​​the external camera 300 and a rectangular coordinate correct answer label are respectively associated. The rectangular coordinate correct answer label is a correct answer label indicating the coordinate values ​​Cr1 to Cr4 of the four vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc set for the vehicle 100 included in the training image Ti. In the rectangular coordinate correct answer label, for each coordinate value Cr1 to Cr4, supplementary information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc the coordinate value Cr1 to Cr4 is associated. In the present embodiment, the rectangular coordinate values ​​Cr1 to Cr4 outputted from the second learning model DM2 are coordinate values ​​in the image coordinate system. Therefore, in the rectangular coordinate correct answer label, each coordinate value Cr1 to Cr4 is represented by the image coordinate system.

[0093] For the second learning model DM2, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, by back propagation (error back propagation method), the parameters of the CNN are updated in a manner that reduces the error between the output result of the second learning model DM2 and the correct label of the rectangular coordinates.

[0094] The rectangle database Db1 is used to calculate the coordinate values ​​of the measuring point 10e of the vehicle 100 from the rectangle coordinate values ​​Cr1 to Cr4. The rectangle database Db1 is a database showing the relative positional relationship between each vertex Vc1 to Vc4 of the vehicle circumscribed rectangle Rc and the measuring point 10e of the vehicle 100.

[0095] The camera parameter Pa is a parameter related to the external camera 300. In the present embodiment, the camera parameter Pa is used to transform the coordinate value of the measuring point 10e of the vehicle 100 represented by the image coordinate system into the global coordinate system GC. The camera parameter Pa is, for example, the installation position, installation posture, and focal length of the external camera 300.

[0096] The processor 201c functions as an accuracy acquisition unit 211c, a learning data acquisition unit 212c, a learning unit 213c, a control information acquisition unit 214c, a position acquisition unit 215c, a signal generation unit 216, and a transmission unit 217 by executing a program PG2c stored in the memory 202c.

[0097] The accuracy acquisition unit 211c acquires the accuracy of the second learning model DM2 pre-learned using the second initial learning data set. In the present embodiment, the second learning model DM2 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211c acquires the accuracy of each of the plurality of second learning models DM2 corresponding to each external camera 300.

[0098] The learning data acquisition unit 212c performs the following processing when the accuracy of the second learning model DM2 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input to the second learning model DM2. In this case, the learning data acquisition unit 212c acquires a second additional learning data set for relearning the second learning model DM2 as additional learning. The second additional learning data set includes one or more second additional learning data sets in which the training images Ti including the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 that acquired the captured images Im1 to Im4 with an accuracy less than the threshold value and the rectangular coordinate correct answer label are established in correspondence. In the present embodiment, the second learning model DM2 is prepared for each external camera 300. Thus, the learning data acquisition unit 212c acquires the second additional learning data set for relearning the second learning model DM2 with an accuracy less than the threshold value.

[0099] The learning unit 213c performs the following processing when the accuracy of the second learning model DM2 is less than a predetermined threshold value when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the second learning model DM2. In this case, the learning unit 213c performs a relearning process for relearning the second learning model DM2. In the present embodiment, the second learning model DM2 is prepared for each external camera 300. Thus, the learning unit 213c relearns the second learning model DM2 corresponding to the external camera 300 that has obtained the captured images Im1 to Im4 with an accuracy less than the threshold value using the acquired second additional learning data group.

[0100] The control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2. In the present embodiment, the second learning model DM2 is prepared for each external camera 300. Thus, the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 corresponding to the external camera 300 that has acquired the captured images Im1 to Im4. At this time, when the relearning process is completed, the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 after the relearning process.

[0101] The position acquisition unit 215c uses the rectangular database Db1 to calculate the coordinate value of the measuring point 10e of the vehicle 100 from the rectangular coordinate values ​​Cr1 to Cr4. Furthermore, the position acquisition unit 215c transforms the coordinate value of the measuring point 10e of the vehicle 100 represented by the image coordinate system into the coordinate value in the global coordinate system GC using the camera parameter Pa. At this time, in order to acquire the camera parameter Pa, the position acquisition unit 215c may acquire the camera identifier indicating the external camera 300 that has acquired the captured images Im1 to Im4 among the camera identifiers for identifying the plurality of external cameras 300.

[0102] The position acquisition unit 215c also calculates the orientation of the vehicle 100. The position acquisition unit 215c calculates the orientation of the vehicle 100 using, for example, the shape of the vehicle circumscribed rectangle Rc and the accompanying information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc the coordinate values ​​Cr1 to Cr4 are.

[0103] Fig. 20 : is a flowchart showing the learning processing method in the fourth embodiment. Fig. 20 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0104] In the accuracy acquisition process, the accuracy acquisition unit 211c acquires the accuracy of the second learning model DM2 pre-learned using the second initial learning data group (step S241). When the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input into the second learning model DM2, and the accuracy of the second learning model DM2 is less than a predetermined threshold value (step S242: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212c acquires the second additional learning data group (step S243). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213c performs a re-learning process of re-learning the second learning model DM2 using the acquired second additional learning data group (step S244). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the second learning model DM2 and the accuracy of the second learning model DM2 becomes above a predetermined threshold (step S242: No), the processor 201c does not execute the learning data acquisition process and the learning process, and ends the process.

[0105] Fig.21 This is a flowchart showing a method of acquiring rectangle coordinate values ​​Cr1 to Cr4. Fig.21The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0106] In the control information acquisition process, the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the second learning model DM2, if the accuracy of the second learning model DM2 is less than a predetermined threshold value (step S341: yes), and when the relearning process is completed (step S342: yes), the control information acquisition unit 214c acquires the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 after the relearning process (step S343). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the second learning model DM2 and the accuracy of the second learning model DM2 becomes above a predetermined threshold (step S341: No), the control information acquisition unit 214c obtains the rectangular coordinate values ​​Cr1~Cr4 by inputting the captured images Im1~Im4 into the second learning model DM2 that has not performed the re-learning processing (step S344).

[0107] According to the fourth embodiment, the information processing device 200c can perform the following processing when the accuracy of the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 is less than a threshold when it is input to the learned second learning model DM2. In this case, the information processing device 200c can relearn the second learning model DM2 using a second additional learning data set that establishes a corresponding training image Ti containing background images and vehicle images corresponding to the shooting area of ​​the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold and rectangular coordinate correct answer labels. In this way, when the appearance state in the shooting area of ​​the external camera 300 among the multiple external cameras 300 has changed since the learning of the second learning model DM2, the accuracy of the second learning model DM2 can be suppressed from decreasing.

[0108] In addition, according to the fourth embodiment, when the relearning process is completed, the information processing device 200c can obtain the rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the second learning model DM2 after the relearning process. This can suppress the decrease in the accuracy of the rectangular coordinate values ​​Cr1 to Cr4 calculated using the captured images Im1 to Im4 obtained by the external cameras 300 of a part of the plurality of external cameras 300.

[0109] It should be noted that the rectangular coordinate values ​​Cr1 to Cr4 can be coordinate values ​​in the camera coordinate system or in the global coordinate system GC. In the case where the rectangular coordinate values ​​Cr1 to Cr4 are coordinate values ​​represented by either the camera coordinate system or the global coordinate system GC, the second learning model DM2 is learned, for example, as follows. In this case, the second learning model DM2 learns, for example, by establishing a correspondence between the camera identifier of the external camera 300 that acquired the training image Ti and the camera parameter Pa of the external camera 300 that acquired the training image Ti and the training image Ti. In addition, in this case, in the rectangular coordinate correct answer label, each coordinate value Cr1 to Cr4 is represented, for example, by the same coordinate system as the coordinate system of the output rectangular coordinate values ​​Cr1 to Cr4.

[0110] The information processing device 200c may also obtain the coordinate values ​​of the contact surface of the wheel as the signal generation parameter instead of the rectangular coordinate values ​​Cr1 to Cr4. Even in this embodiment, the information processing device 200c can obtain the rectangular coordinate values ​​Cr1 to Cr4 and generate the travel control signal.

[0111] E. Fifth Implementation Method: Fig. 22 : is a block diagram showing the structure of the information processing system 50d in the fifth embodiment. The information processing system 50d includes one or more vehicles 100, multiple external cameras 300, and an information processing device 200d. In this embodiment, the information processing device 200d obtains the stereo coordinate values ​​as parameters for signal generation by inputting the captured images Im1~Im4 to the third learning model DM3 as the machine learning model DM. The other structures of the information processing system 50d are the same as those of the first embodiment unless otherwise specified. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0112] Fig.23 This is a diagram for explaining the three-dimensional coordinate values ​​Cb1 to Cb8. The three-dimensional coordinate values ​​Cb1 to Cb8 are the coordinate values ​​of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb. The vehicle circumscribed rectangular parallelepiped Rb is a rectangular parallelepiped set to surround the vehicle 100 in the images Ti, Im1 to Im4. The three-dimensional coordinate values ​​Cb1 to Cb8 are used to calculate at least one of the position and orientation of the vehicle 100.

[0113] like Fig. 22 As shown, the information processing device 200 d is constituted by a computer including a processor 201 d , a memory 202 d , an input / output interface 203 , and an internal bus 204 .

[0114] The memory 202 d stores various information including a program PG2 d for controlling the operation of the information processing device 200 d , a reference path RR, a plurality of third learning models DM3 prepared for each external camera 300 , a stereo database Db2 , and camera parameters Pa.

[0115] The third learning model DM3 outputs stereo coordinate values ​​Cb1 to Cb8 when the captured images Im1 to Im4 are input. The third learning model DM3 is pre-learned by supervised learning using the third initial learning data set. The third initial learning data set includes a plurality of third initial learning data sets corresponding to each of the capturing areas of the plurality of external cameras 300. In the plurality of third initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the capturing area of ​​the external camera 300 and a stereo coordinate correct answer label are respectively associated. The stereo coordinate correct answer label is a correct answer label indicating the coordinate values ​​Cb1 to Cb8 of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb set for the vehicle 100 included in the training image Ti. In the stereo coordinate correct answer label, for each coordinate value Cb1 to Cb8, supplementary information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb the coordinate value Cb1 to Cb8 is associated. In this embodiment, the three-dimensional coordinate values ​​Cb1 to Cb8 output from the third learning model DM3 are coordinate values ​​in the camera coordinate system. Therefore, in the three-dimensional coordinate correct answer label, each coordinate value Cb1 to Cb8 is represented by the camera coordinate system. In addition, the third learning model DM3 learns at least one of the camera parameter Pa and the camera identifier used to obtain the camera parameter Pa in correspondence with the training image Ti.

[0116] For the third learning model DM3, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, back propagation (error back propagation method) is used to update the parameters of the CNN in a manner that reduces the error between the output result of the third learning model DM3 and the stereo coordinate correct solution label.

[0117] The three-dimensional database Db2 is used to calculate the coordinate values ​​of the measuring point 10e of the vehicle 100 based on the three-dimensional coordinate values ​​Cb1 to Cr8. The three-dimensional database Db2 is a database showing the relative positional relationship between each vertex Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb and the measuring point 10e of the vehicle 100.

[0118] The processor 201d functions as an accuracy acquisition unit 211d, a learning data acquisition unit 212d, a learning unit 213d, a control information acquisition unit 214d, a position acquisition unit 215d, a signal generation unit 216, and a transmission unit 217 by executing a program PG2d stored in the memory 202d.

[0119] The accuracy acquisition unit 211d acquires the accuracy of the third learning model DM3 pre-learned using the third initial learning data set. In the present embodiment, the third learning model DM3 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211d acquires the accuracy of each of the plurality of third learning models DM3 corresponding to each external camera 300.

[0120] The learning data acquisition unit 212d performs the following processing when the accuracy of the third learning model DM3 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input to the third learning model DM3. In this case, the learning data acquisition unit 212d acquires a third additional learning data set for relearning the third learning model DM3 as additional learning. The third additional learning data set includes one or more third additional learning data sets in which the training images Ti including the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 that acquired the captured images Im1 to Im4 with an accuracy less than the threshold value and the stereo coordinate correct answer label are established in correspondence. In the present embodiment, the third learning model DM3 is prepared for each external camera 300. Thus, the learning data acquisition unit 212d acquires the third additional learning data set for relearning the third learning model DM3 with an accuracy less than the threshold value.

[0121] The learning unit 213d performs the following processing when the accuracy of the third learning model DM3 is less than a predetermined threshold value when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the third learning model DM3. In this case, the learning unit 213d performs a relearning process for relearning the third learning model DM3. In the present embodiment, the third learning model DM3 is prepared for each external camera 300. Thus, the learning unit 213d relearns the third learning model DM3 corresponding to the external camera 300 that has obtained the captured images Im1 to Im4 with an accuracy less than the threshold value using the acquired third additional learning data group.

[0122] The control information acquisition unit 214d acquires the stereo coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3. In the present embodiment, the third learning model DM3 is prepared for each external camera 300. Thus, the control information acquisition unit 214d acquires the stereo coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3 corresponding to the external camera 300 that has acquired the captured images Im1 to Im4. At this time, when the relearning process is completed, the control information acquisition unit 214d acquires the stereo coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3 after the relearning process.

[0123] The position acquisition unit 215d uses the stereo database Db2 to calculate the coordinate value of the measuring point 10e of the vehicle 100 from the stereo coordinate values ​​Cb1 to Cb8. Furthermore, the position acquisition unit 215d converts the coordinate value of the measuring point 10e of the vehicle 100 represented by the image coordinate system into the coordinate value in the camera coordinate system using the camera parameter Pa. At this time, the position acquisition unit 215d may acquire the camera identifier in order to acquire the camera parameter Pa.

[0124] The position acquisition unit 215d also calculates the orientation of the vehicle 100. The position acquisition unit 215d calculates the orientation of the vehicle 100 using, for example, the shape of the vehicle circumscribed rectangular parallelepiped Rb and the accompanying information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb the coordinate values ​​Cb1 to Cb8 are.

[0125] Fig.24 : is a flowchart showing the learning processing method in the fifth embodiment. Fig.24 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0126] In the accuracy acquisition process, the accuracy acquisition unit 211d acquires the accuracy of the third learning model DM3 pre-learned using the third initial learning data group (step S251). When the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input into the third learning model DM3, if the accuracy of the third learning model DM3 is less than a predetermined threshold value (step S252: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212d acquires the third additional learning data group (step S253). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213d performs a re-learning process of re-learning the third learning model DM3 using the acquired third additional learning data group (step S254). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the third learning model DM3 and the accuracy of the third learning model DM3 becomes above a predetermined threshold (step S252: No), the processor 201d does not execute the learning data acquisition process and the learning process, and ends the process.

[0127] Fig.25 This is a flowchart showing a method of acquiring the three-dimensional coordinate values ​​Cb1 to Cb8. Fig.25 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0128] In the control information acquisition process, the control information acquisition unit 214d acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the third learning model DM3, if the accuracy of the third learning model DM3 is less than a predetermined threshold value (step S351: yes), and when the relearning process is completed (step S352: yes), the control information acquisition unit 214d acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3 after the relearning process (step S353). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the third learning model DM3 and the accuracy of the third learning model DM3 becomes above a predetermined threshold (step S351: No), the control information acquisition unit 214d obtains the stereo coordinate values ​​Cb1~Cb8 by inputting the captured images Im1~Im4 into the third learning model DM3 that has not performed the re-learning process (step S354).

[0129] According to the fifth embodiment, the information processing device 200d can perform the following processing when the accuracy of the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 is less than a threshold when it is input to the learned third learning model DM3. In this case, the information processing device 200d can relearn the third learning model DM3 using the third additional learning data set that establishes a corresponding relationship between the training image Ti containing the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold and the stereo coordinate correct answer label. In this way, when the appearance state in the shooting area of ​​the external camera 300 among the multiple external cameras 300 has changed since the learning of the third learning model DM3, the accuracy of the third learning model DM3 can be suppressed from decreasing.

[0130] In addition, according to the fifth embodiment, when the relearning process is completed, the information processing device 200d can obtain the stereo coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the third learning model DM3 after the relearning process. As a result, it is possible to suppress the decrease in the accuracy of the stereo coordinate values ​​Cb1 to Cb8 calculated using the captured images Im1 to Im4 obtained by the external cameras 300 of a part of the plurality of external cameras 300.

[0131] It should be noted that the three-dimensional coordinate values ​​Cb1 to Cb8 may be coordinate values ​​in the image coordinate system or coordinate values ​​in the global coordinate system GC.

[0132] F. Sixth Implementation Method: Fig.26 : is a block diagram showing the structure of the information processing system 50e in the sixth embodiment. The information processing system 50e includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200e. In the present embodiment, the information processing device 200e obtains vehicle position information as a parameter for signal generation by inputting the captured images Im1 to Im4 to the fourth learning model DM4 as the machine learning model DM. The other structures of the information processing system 50e are the same as those of the first embodiment unless otherwise specified. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0133] The information processing device 200 e is constituted by a computer including a processor 201 e , a memory 202 e , an input / output interface 203 , and an internal bus 204 .

[0134] The memory 202e stores various information including a program PG2e for controlling the operation of the information processing device 200e, a reference path RR, a plurality of fourth learning models DM4 prepared for each external camera 300, and a camera parameter Pa. In the present embodiment, a program for generating a travel control signal is prepared for each external camera 300. Thus, a reference path RR is prepared for each external camera 300. Furthermore, the target position determined in the process of generating the travel control signal is represented by the coordinates of X, Y, and Z in the camera coordinate system.

[0135] The fourth learning model DM4 outputs vehicle position information when the captured images Im1 to Im4 are input. In the present embodiment, the fourth learning model DM4 outputs the coordinate value of the measuring point 10e of the vehicle 100 and the orientation of the vehicle 100 when the captured images Im1 to Im4 are input. The fourth learning model DM4 is pre-learned by supervised learning using the fourth initial learning data set. The fourth initial learning data set includes a plurality of fourth initial learning data sets corresponding to each of the imaging regions of the plurality of external cameras 300. In the plurality of fourth initial learning data sets, a training image Ti including a background image and a vehicle image corresponding to the imaging region of the external camera 300, a position coordinate correct answer label, and an orientation correct answer label are associated with each other. The orientation correct answer label is a correct answer label indicating the orientation of the vehicle 100 included in the training image Ti. The position coordinate correct answer label is a correct answer label indicating the coordinate value of the measuring point 10e of the vehicle 100 included in the training image Ti. In this embodiment, the coordinate value of the measuring point 10e of the vehicle 100 output from the fourth learning model DM4 is a coordinate value in the camera coordinate system. Therefore, in the position coordinate correct answer label, the coordinate value of the measuring point 10e of the vehicle 100 is represented by the camera coordinate system. In addition, the fourth learning model DM4 learns at least one of the camera parameter Pa and the camera identifier for obtaining the camera parameter Pa in correspondence with the training image Ti.

[0136] For the fourth learning model DM4, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, by back propagation (error back propagation method), the parameters of the CNN are updated in a manner that reduces the error between the output result of the fourth learning model DM4 and the correct label of the position coordinate and the error toward the correct label.

[0137] The processor 201e functions as an accuracy acquisition unit 211e, a learning data acquisition unit 212e, a learning unit 213e, a control information acquisition unit 214e, a signal generation unit 216, and a transmission unit 217 by executing a program PG2e stored in the memory 202e.

[0138] The accuracy acquisition unit 211e acquires the accuracy of the fourth learning model DM4 pre-learned using the fourth initial learning data set. In this embodiment, the fourth learning model DM4 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211e acquires the accuracy of each of the plurality of fourth learning models DM4 corresponding to each external camera 300.

[0139] The learning data acquisition unit 212e performs the following processing when the accuracy of the fourth learning model DM4 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the fourth learning model DM4. In this case, the learning data acquisition unit 212e acquires a fourth additional learning data set for relearning the fourth learning model DM4 as additional learning. The fourth additional learning data set includes one or more fourth additional learning data sets corresponding to the training images Ti of the background image and the vehicle image corresponding to the photographing area of ​​the external camera 300 that acquired the captured images Im1 to Im4 with an accuracy less than the threshold value, the position coordinate correct answer label, and the orientation correct answer label. In the present embodiment, the fourth learning model DM4 is prepared for each external camera 300. Thus, the learning data acquisition unit 212e acquires the fourth additional learning data set for relearning the fourth learning model DM4 with an accuracy less than the threshold value.

[0140] The learning unit 213e performs the following processing when the accuracy of the fourth learning model DM4 is less than a predetermined threshold value when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the fourth learning model DM4. In this case, the learning unit 213e performs a relearning process for relearning the fourth learning model DM4. In the present embodiment, the fourth learning model DM4 is prepared for each external camera 300. Thus, the learning unit 213e relearns the fourth learning model DM4 corresponding to the external camera 300 that has obtained the captured images Im1 to Im4 with an accuracy less than the threshold value using the acquired fourth additional learning data group.

[0141] The control information acquisition unit 214e acquires the vehicle position information by inputting the captured images Im1 to Im4 to the fourth learning model DM4. In the present embodiment, the fourth learning model DM4 is prepared for each external camera 300. Thus, the control information acquisition unit 214e acquires the vehicle position information by inputting the captured images Im1 to Im4 to the fourth learning model DM4 corresponding to the external camera 300 that has acquired the captured images Im1 to Im4. At this time, when the relearning process is completed, the control information acquisition unit 214e acquires the vehicle position information by inputting the captured images Im1 to Im4 to the fourth learning model DM4 after the relearning process.

[0142] Fig. 27 : is a flowchart showing the learning processing method in the sixth embodiment. Fig. 27 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0143] In the accuracy acquisition process, the accuracy acquisition unit 211e acquires the accuracy of the fourth learning model DM4 pre-learned using the fourth initial learning data group (step S261). When the captured images Im1~Im4 acquired by the external cameras 300 of a part of the multiple external cameras 300 are input into the fourth learning model DM4, and the accuracy of the fourth learning model DM4 is less than a predetermined threshold value (step S262: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212e acquires the fourth additional learning data group (step S263). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213e performs a re-learning process of re-learning the fourth learning model DM4 using the acquired fourth additional learning data group (step S264). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the fourth learning model DM4 and the accuracy of the fourth learning model DM4 becomes above a predetermined threshold (step S262: No), the processor 201e does not execute the learning data acquisition process and the learning process, and ends the process.

[0144] Fig.28 : is a flowchart showing a method of acquiring vehicle position information using the fourth learning model DM4. Fig.28 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0145] In the control information acquisition step, the control information acquisition unit 214e acquires the vehicle position information by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the fourth learning model DM4, if the accuracy of the fourth learning model DM4 is less than a predetermined threshold value (step S361: yes), and when the relearning process is completed (step S362: yes), the control information acquisition unit 214e acquires the vehicle position information by inputting the captured images Im1 to Im4 to the fourth learning model DM4 after the relearning process (step S363). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the fourth learning model DM4 and the accuracy of the fourth learning model DM4 becomes above a predetermined threshold (step S361: No), the control information acquisition unit 214e obtains the vehicle position information by inputting the captured images Im1~Im4 into the fourth learning model DM4 that has not performed the re-learning process (step S364).

[0146] According to the sixth embodiment, the information processing device 200e can perform the following processing when the accuracy of the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 is less than the threshold when input into the learned fourth learning model DM4. In this case, the information processing device 200e can relearn the fourth learning model DM4 using the training image Ti containing the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold, the position coordinate correct answer label, and the orientation correct answer label. In this way, when the appearance state in the shooting area of ​​the external camera 300 among the multiple external cameras 300 has changed since the learning of the fourth learning model DM4, the accuracy of the fourth learning model DM4 can be suppressed from decreasing.

[0147] In addition, according to the sixth embodiment, when the relearning process is completed, the information processing device 200e can obtain the vehicle position information by inputting the captured images Im1 to Im4 to the fourth learning model DM4 after the relearning process. As a result, it is possible to suppress the decrease in the accuracy of the vehicle position information calculated using the captured images Im1 to Im4 obtained by the external cameras 300 of a part of the multiple external cameras 300.

[0148] G. Seventh Implementation Method: Fig.29: is a block diagram showing the structure of the information processing system 50f in the seventh embodiment. The information processing system 50f includes one or more vehicles 100, multiple external cameras 300, and an information processing device 200f. In this embodiment, the information processing device 200f obtains a driving control signal by inputting the captured images Im1~Im4 and the camera identifier to the fifth learning model DM5 as the machine learning model DM. The other structures of the information processing system 50f are the same as those of the first embodiment unless otherwise specified. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0149] The information processing device 200 f is constituted by a computer including a processor 201 f , a memory 202 f , an input / output interface 203 , and an internal bus 204 .

[0150] The memory 202 f stores various information including a program PG2 f for controlling the operation of the information processing device 200 f , a reference route RR, and a plurality of fifth learning models DM5 prepared for each external camera 300 .

[0151] The fifth learning model DM5 outputs a driving control signal when the captured images Im1 to Im4 and the camera identifier representing the external camera 300 that acquired the captured images Im1 to Im4 are input. The fifth learning model DM5 is pre-learned by supervised learning using the fifth initial learning data set. The fifth initial learning data set includes a plurality of fifth initial learning data sets corresponding to each of the shooting areas of the plurality of external cameras 300. In the plurality of fifth initial learning data sets, a correspondence is established between the training image Ti including the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 and the signal correct answer label. The signal correct answer label is a correct answer label representing the target value of each parameter included in the driving control signal. The signal correct answer label is generated based on the reference path RR and the target speed in the shooting area of ​​the external camera 300 that acquired the training image Ti.

[0152] For the fifth learning model DM5, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, back propagation (error back propagation method) is used to update the parameters of the CNN in a manner that reduces the error between the output result of the fifth learning model DM5 and the correct signal label.

[0153] The processor 201f functions as an accuracy acquisition unit 211f, a learning data acquisition unit 212f, a learning unit 213f, a control information acquisition unit 214f, and a transmission unit 217 by executing a program PG2f stored in the memory 202f.

[0154] The accuracy acquisition unit 211f acquires the accuracy of the fifth learning model DM5 pre-learned using the fifth initial learning data set. In the present embodiment, the fifth learning model DM5 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211f acquires the accuracy of each of the plurality of fifth learning models DM5 corresponding to each external camera 300.

[0155] The learning data acquisition unit 212f performs the following processing when the accuracy of the fifth learning model DM5 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input to the fifth learning model DM5. In this case, the learning data acquisition unit 212f acquires the fifth additional learning data set for relearning the fifth learning model DM5 as additional learning. The fifth additional learning data set includes one or more fifth additional learning data sets in which the training images Ti including the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 that acquired the captured images Im1 to Im4 with an accuracy less than the threshold value and the signal correct answer label are established in correspondence. In the present embodiment, the fifth learning model DM5 is prepared for each external camera 300. Thus, the learning data acquisition unit 212f acquires the fifth additional learning data set for relearning the fifth learning model DM5 with an accuracy less than the threshold value.

[0156] The learning unit 213f performs the following processing when the accuracy of the fifth learning model DM5 is less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input to the fifth learning model DM5. In this case, the learning unit 213f performs a relearning process for relearning the fifth learning model DM5. In the present embodiment, the fifth learning model DM5 is prepared for each external camera 300. Thus, the learning unit 213f relearns the fifth learning model DM5 corresponding to the external camera 300 that acquired the captured images Im1 to Im4 with an accuracy less than the threshold value using the acquired fifth additional learning data group.

[0157] The control information acquisition unit 214f ​​acquires the travel control signal by inputting the captured images Im1 to Im4 to the fifth learning model DM5. In the present embodiment, the fifth learning model DM5 is prepared for each external camera 300. Thus, the control information acquisition unit 214f ​​acquires the travel control signal by inputting the captured images Im1 to Im4 to the fifth learning model DM5 corresponding to the external camera 300 that has acquired the captured images Im1 to Im4. At this time, when the relearning process is completed, the control information acquisition unit 214f ​​acquires the travel control signal by inputting the captured images Im1 to Im4 to the fifth learning model DM5 after the relearning process.

[0158] Fig.30 : is a flowchart showing the learning processing method in the seventh embodiment. Fig.30 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0159] In the accuracy acquisition process, the accuracy acquisition unit 211f acquires the accuracy of the fifth learning model DM5 pre-learned using the fifth initial learning data group (step S271). When the captured images Im1~Im4 acquired by the external cameras 300 of a part of the multiple external cameras 300 are input into the fifth learning model DM5, when the accuracy of the fifth learning model DM5 is less than a predetermined threshold value (step S272: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212f acquires the fifth additional learning data group (step S273). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213f performs a re-learning process of re-learning the fifth learning model DM5 using the acquired fifth additional learning data group (step S274). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the fifth learning model DM5 and the accuracy of the fifth learning model DM5 becomes above a predetermined threshold (step S272: No), the processor 201f does not execute the learning data acquisition process and the learning process, and ends the process.

[0160] Fig.31 This is a flowchart showing a method of acquiring a travel control signal using the fifth learning model DM5. Fig.31 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0161] In the control information acquisition step, the control information acquisition unit 214f ​​acquires the travel control signal by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the fifth learning model DM5, if the accuracy of the fifth learning model DM5 is less than a predetermined threshold value (step S371: yes), and when the relearning process is completed (step S372: yes), the control information acquisition unit 214f ​​acquires the travel control signal by inputting the captured images Im1 to Im4 to the fifth learning model DM5 after the relearning process (step S373). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the fifth learning model DM5 and the accuracy of the fifth learning model DM5 becomes above a predetermined threshold (step S371: No), the control information acquisition unit 214f ​​obtains the driving control signal (step S374) by inputting the captured images Im1~Im4 into the fifth learning model DM5 that has not performed the re-learning process.

[0162] According to the seventh embodiment, the information processing device 200f can perform the following processing when the accuracy of the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 is less than the threshold when it is input to the fifth learning model DM5 that has been learned. In this case, the information processing device 200f can relearn the fifth learning model DM5 using the fifth additional learning data set that establishes a corresponding relationship between the training image Ti containing the background image and the vehicle image corresponding to the shooting area of ​​the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold and the signal correct answer label. In this way, when the appearance state in the shooting area of ​​the external camera 300 among the multiple external cameras 300 has changed since the learning of the fifth learning model DM5, the accuracy of the fifth learning model DM5 can be suppressed from decreasing.

[0163] In addition, according to the seventh embodiment, the information processing device 200f can obtain the driving control signal by inputting the captured images Im1 to Im4 to the fifth learning model DM5 after the relearning process when the relearning process is completed. This can suppress the decrease in the accuracy of the driving control signal generated using the captured images Im1 to Im4 obtained by the external cameras 300 of a part of the plurality of external cameras 300.

[0164] It should be noted that the information input to the fifth learning model DM5 is not limited to the above. The information processing device 200f can also obtain the driving control signal by inputting other information such as speed information indicating the driving speed of the vehicle 100, path information indicating the reference path RR of the vehicle 100, and the pinion angle to the fifth learning model DM5 in addition to the captured images Im1 to Im4 and the camera identifier. In this way, the information processing device 200f can obtain a more appropriate driving control signal corresponding to the driving condition of the vehicle 100 and the reference path RR.

[0165] H. Eighth Implementation Method: Fig.32 : is a block diagram showing the structure of the information processing system 50g in the eighth embodiment. The information processing system 50g includes one or more vehicles 100, a plurality of external cameras 300, and an information processing device 200g. In the present embodiment, the information processing device 200g obtains shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6 as the machine learning model DM. The other structures of the information processing system 50g are the same as those of the first embodiment unless otherwise specified. The same figure marks are given to the same structures as those of the first embodiment and the description is omitted.

[0166] Fig.33 : is a diagram showing an example of a method for generating a training image Ti used in learning the sixth learning model DM6. In the present embodiment, the training image Ti is a synthetic image Ci that reproduces the real space. The synthetic image Ci is a two-dimensional image generated by synthesizing a virtual vehicle image Vi with a background image Bi of either an actual background image Br or a virtual background image Bv. The actual background image Br is a background image Bi obtained by actually photographing the object area TA using an external camera 300. The virtual background image Bv is a background image Bi that virtually reproduces the object area TA. The virtual background image Bv is, for example, any one of two-dimensional CAD data and three-dimensional CAD data representing the object area TA. The virtual vehicle image Vi is an image that virtually reproduces the vehicle 100. The virtual vehicle image Vi is any one of two-dimensional CAD data and three-dimensional CAD data representing the vehicle 100.

[0167] In the present embodiment, the synthetic image Ci is generated by synthesizing the virtual vehicle image Vi with the virtual background image Bv. Specifically, the synthetic image Ci is generated by, for example, synthesizing the virtual vehicle image Vi in such a way that it overlaps each object area TA of the manufacturing line of the factory FC reproduced in the virtual space. That is, in the present embodiment, the training image Ti is generated by utilizing the digital twin.

[0168] In addition, in the present embodiment, the synthetic image Ci is generated using a virtual vehicle image Vi of the vehicle 100 corresponding to the appearance state of the vehicle 100 to be detected. The appearance state of the vehicle 100 is classified, for example, by the body type of the vehicle 100, the appearance color of the vehicle 100, and the product name of the vehicle 100. The body type of the vehicle 100 is a group of vehicle types when the vehicle 100 is classified in a manner of aggregating a plurality of categories according to the appearance shape of the vehicle 100. In the case where the appearance state of the vehicle 100 is classified by the body type of the vehicle 100, the vehicle 100 is classified into one body type by the vehicle specification determined by the full length, width and height of the vehicle 100 and the appearance shape of the vehicle 100. The vehicle specification is also called the vehicle shape or body size. In the present embodiment, the body type is, for example, "SUV", "sedan", "station wagon", "minivan", "van", "compact car", and "small car". It should be noted that the appearance state of the vehicle 100 can also be classified by a prescribed classification number appropriately assigned according to the type and specification.

[0169] It should be noted that, when the virtual background image Bv is three-dimensional data, the virtual background image Bv is transformed from three-dimensional data to two-dimensional data in the process of generating the synthetic image Ci. When the virtual vehicle image Vi is three-dimensional data, the virtual vehicle image Vi is transformed from three-dimensional data to two-dimensional data in the process of generating the synthetic image Ci. In addition, in the process of synthesizing the virtual vehicle image Vi with the background image Bi, the configuration of the virtual vehicle image Vi relative to the background image Bi or the pixel values ​​of each image Bi, Ci, Vi can also be adjusted.

[0170] The information processing device 200 g is constituted by a computer including a processor 201 g , a memory 202 g , an input / output interface 203 , and an internal bus 204 .

[0171] The memory 202 g stores various information including a program PG2 g for controlling the operation of the information processing device 200 g , a reference route RR, and a plurality of sixth learning models DM6 prepared for each external camera 300 .

[0172] The sixth learning model DM6 outputs shape data Da when the captured images Im1 to Im4 are input. The sixth learning model DM6 is pre-learned by supervised learning using the sixth initial learning data set. The sixth initial learning data set includes a plurality of sixth initial learning data sets corresponding to each of the shooting areas of the plurality of external cameras 300. In the plurality of sixth initial learning data sets, a correspondence is established between the training image Ti including the virtual background image Bv corresponding to the shooting area of ​​the external camera 300 and the virtual vehicle image Vi of the vehicle 100 virtually reproducing the appearance state of the vehicle 100 corresponding to the detection object, and the regional correct answer label Lr.

[0173] For the sixth learning model DM6, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, by back propagation (error back propagation method), the parameters of the CNN are updated in a manner that reduces the error between the output result of the sixth learning model DM6 and the correct label Lr of the region.

[0174] The processor 201g functions as an accuracy acquisition unit 211g, an appearance acquisition unit 218, a learning data acquisition unit 212g, a learning unit 213g, a control information acquisition unit 214g, a position acquisition unit 215, a signal generation unit 216 and a transmission unit 217 by executing a program PG2g stored in the memory 202g.

[0175] The accuracy acquisition unit 211g acquires the accuracy of the sixth learning model DM6 pre-learned using the sixth initial learning data set. In the present embodiment, the sixth learning model DM6 is prepared for each external camera 300. Thus, the accuracy acquisition unit 211g acquires the accuracy of each of the plurality of sixth learning models DM6 corresponding to each external camera 300.

[0176] The appearance acquisition unit 218 acquires appearance information. The appearance information is information indicating the appearance state of the vehicle 100 to be detected. The appearance information is generated, for example, by determining the appearance state of the vehicle 100 to be detected using management information. The management information is information indicating the driving order of multiple vehicles 100 driving in the shooting area of ​​the external camera 300. In the management information, a vehicle identifier and a camera identifier are associated. The vehicle identifier is a unique identifier assigned in a non-repeating manner between vehicles 100 in order to identify multiple vehicles 100. The management information is generated, for example, using vehicle position information, a transmission history of a driving control signal to the vehicle 100, and a location where the external camera 300 is installed.

[0177] The learning data acquisition unit 212g performs the following processing when the accuracy of the sixth learning model DM6 is less than a predetermined threshold value when the captured images Im1 to Im4 captured by a part of the external cameras 300 among the plurality of external cameras 300 are input to the sixth learning model DM6. In this case, the learning data acquisition unit 212g acquires a sixth additional learning data set for relearning the sixth learning model DM6 as additional learning. The sixth additional learning data set includes one or more sixth additional learning data sets in which a virtual background image Bv corresponding to the captured area of ​​the external camera 300 that has captured the captured images Im1 to Im4 with an accuracy less than the threshold value and a training image Ti of a virtual vehicle image Vi of the vehicle 100 that virtually reproduces the appearance state determined by the appearance information and the regional correct answer label Lr are associated. In the present embodiment, the sixth learning model DM6 is prepared for each external camera 300. Thus, the learning data acquisition unit 212g acquires the sixth additional learning data set for relearning the sixth learning model DM6 with an accuracy less than the threshold value.

[0178] The learning unit 213g performs the following processing when the accuracy of the sixth learning model DM6 is less than a predetermined threshold value when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the sixth learning model DM6. In this case, the learning unit 213g performs a relearning process to relearn the sixth learning model DM6. In the present embodiment, the sixth learning model DM6 is prepared for each external camera 300. Thus, the learning unit 213g relearns the sixth learning model DM6 corresponding to the external camera 300 that has obtained the captured images Im1 to Im4 with an accuracy less than the threshold value using the acquired sixth additional learning data group. That is, the learning unit 213g relearns the sixth learning model DM6 using the virtual vehicle image Vi of the vehicle 100 that virtually reproduces the appearance state determined by the appearance information.

[0179] The control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6. In the present embodiment, the sixth learning model DM6 is prepared for each external camera 300. Thus, the control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6 corresponding to the external camera 300 that has acquired the captured images Im1 to Im4. At this time, when the relearning process is completed, the control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6 after the relearning process.

[0180] Fig.34 : is a flowchart showing the learning processing method in the eighth embodiment. Fig.34 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0181] In the accuracy acquisition process, the accuracy acquisition unit 211g acquires the accuracy of the sixth learning model DM6 pre-learned using the sixth initial learning data group (step S281). When the captured images Im1~Im4 acquired by the external cameras 300 of a part of the multiple external cameras 300 are input into the sixth learning model DM6, when the accuracy of the sixth learning model DM6 is less than a predetermined threshold value (step S282: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212g acquires the sixth additional learning data group (step S283). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213g performs a re-learning process of re-learning the sixth learning model DM6 using the acquired sixth additional learning data group (step S284). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the sixth learning model DM6 and the accuracy of the sixth learning model DM6 becomes above a predetermined threshold (step S282: No), the processor 201g does not execute the learning data acquisition process and the learning process, and ends the process.

[0182] Fig.35 This is a flowchart showing a method of acquiring shape data Da in the eighth embodiment. Fig.35 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0183] In the control information acquisition step, the control information acquisition unit 214g acquires the shape data Da by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the sixth learning model DM6, if the accuracy of the sixth learning model DM6 is less than a predetermined threshold value (step S381: yes), and when the relearning process is completed (step S382: yes), the control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the sixth learning model DM6 after the relearning process (step S383). On the other hand, when the accuracy of the sixth learning model DM6 becomes above a predetermined threshold when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input into the sixth learning model DM6 (step S381: No), the control information acquisition unit 214g acquires the shape data Da (step S384) by inputting the captured images Im1 to Im4 into the sixth learning model DM6 on which the re-learning process has not been performed.

[0184] According to the eighth embodiment, the information processing device 200g can perform the following processing when the accuracy of the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 is less than the threshold value when input to the learned sixth learning model DM6. In this case, the information processing device 200g can relearn the sixth learning model DM6 by using the synthesized image Ci generated by synthesizing the virtual vehicle image Vi with the background image Bi as the training image Ti.

[0185] In addition, according to the eighth embodiment, the information processing device 200g can relearn the sixth learning model DM6 using the sixth additional learning data set including the virtual vehicle image Vi of the vehicle 100 that virtually reproduces the appearance state determined by the appearance information. In this way, when the sixth learning model DM6 is relearned, the feature value corresponding to the appearance state of the detection target vehicle 100 can be learned. As a result, the accuracy of the sixth learning model DM6 can be improved.

[0186] Furthermore, according to the eighth embodiment, when synthesizing the virtual vehicle image Vi with respect to the background image Bi, by adjusting the arrangement of the virtual vehicle image Vi relative to the background image Bi, it is possible to make at least one of the position and orientation of the vehicle 100 different. Thus, it is possible to easily generate a plurality of training images Ti in which at least one of the position and orientation of the vehicle 100 relative to the passage TR is different.

[0187] In addition, according to the eighth embodiment, in the process of synthesizing the virtual vehicle image Vi with the background image Bi, the pixel value of at least any one of the background image Bi, the virtual vehicle image Vi and the synthesized image Ci can be changed from the initial value. Thus, a plurality of training images Ti with different shooting conditions such as brightness can be easily generated. Thus, a plurality of training images Ti assuming that the pixel value of brightness and the like changes according to weather, time period, season, lighting environment, etc. can be easily generated.

[0188] It should be noted that, in the case where the training image Ti is a synthetic image Ci as in the present embodiment, the correct answer label of the sixth learning model DM6 may be a contour correct answer label Lc instead of the region correct answer label Lr. The contour correct answer label Lc is a correct answer label representing the contour of the vehicle 100 extracted from the virtual vehicle image Vi. In this way, by extracting the outer shape of the virtual vehicle image Vi, it is possible to easily determine whether each region constituting the training image Ti is a region representing the vehicle 100 or a region representing a region other than the vehicle 100. As a result, the load required for preparing the sixth learning model DM6 for learning can be reduced.

[0189] I. Ninth embodiment: Fig.36 : is a block diagram showing the structure of the information processing system 50h in the ninth embodiment. The information processing system 50h includes one or more vehicles 100, multiple external cameras 300, and an information processing device 200h. In this embodiment, when the accuracy of the first learning model DM1 is less than a predetermined threshold when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1, the information processing device 200h learns the seventh learning model DM7 as the machine learning model DM independently of the first learning model DM1. In addition, the information processing device 200h obtains the shape data Da by inputting the captured images Im1~Im4 to the seventh learning model DM7. As for other structures of the information processing system 50h, unless otherwise specified, they are the same as those of the first embodiment. For the same structures as the first embodiment, the same figure marks are marked and the description is omitted.

[0190] The information processing device 200 h is constituted by a computer including a processor 201 h , a memory 202 h , an input / output interface 203 , and an internal bus 204 .

[0191] The memory 202 h stores various information including a program PG2 h for controlling the operation of the information processing device 200 h , a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300 .

[0192] The processor 201h functions as an accuracy acquisition unit 211, a learning data acquisition unit 212h, a learning unit 213h, a control information acquisition unit 214h, a position acquisition unit 215, a signal generation unit 216, and a transmission unit 217 by executing a program PG2h stored in the memory 202h.

[0193] The learning data acquisition unit 212h performs the following processing when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1 to Im4 captured by a part of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning data acquisition unit 212h discards the first learning model DM1 whose accuracy is less than the threshold value, and obtains a new learning data set for newly learning the seventh learning model DM7. The new learning data set includes one or more new learning data sets that establish correspondence between the training images Ti of the background image Bi and the vehicle image corresponding to the shooting area of ​​the external camera 300 that has acquired the captured images Im1 to Im4 whose accuracy is less than the threshold value and the regional correct answer label Lr. In this embodiment, the first learning model DM1 is prepared for each external camera 300. As a result, the learning data acquisition unit 212h discards the first learning model DM1 whose accuracy is less than the threshold value, and obtains a new learning data set for newly learning the seventh learning model DM7.

[0194] The learning unit 213h performs the following processing when the accuracy of the first learning model DM1 is less than a predetermined threshold value when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input to the first learning model DM1. In this case, the learning unit 213h discards the first learning model DM1. In addition, the learning unit 213h performs a new learning process to newly learn the seventh learning model DM7. In this embodiment, the first learning model DM1 is prepared for each external camera 300. Thus, the learning unit 213h discards the first learning model DM1 corresponding to the external camera 300 that has obtained the captured images Im1 to Im4 with an accuracy less than the threshold value. In addition, the learning unit 213h uses the obtained new learning data group to learn the seventh learning model DM7 corresponding to the external camera 300 that has obtained the captured images Im1 to Im4 with an accuracy less than the threshold value. For the seventh learning model DM7, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, back propagation (error back propagation method) is used to update the parameters of the CNN so as to reduce the error between the output result of the seventh learning model DM7 and the correct region label Lr. The learning unit 213h stores the created seventh learning model DM7 in the memory 202h.

[0195] The control information acquisition unit 214h acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 when the accuracy of the first learning model DM1 becomes equal to or greater than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1. On the other hand, the control information acquisition unit 214h acquires shape data Da by inputting the captured images Im1 to Im4 to the seventh learning model DM7 when the accuracy of the first learning model DM1 becomes less than a predetermined threshold value when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1. In the present embodiment, each learning model DM1 and DM7 is prepared for each external camera 300. Thus, the control information acquisition unit 214h acquires shape data Da by inputting the captured images Im1 to Im4 to the learning models DM1 and DM7 corresponding to the external cameras 300 that acquired the captured images Im1 to Im4.

[0196] Fig.37 : is a flowchart showing the learning processing method in the ninth embodiment. Fig.37 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0197] In the accuracy acquisition process, the accuracy acquisition unit 211 acquires the accuracy of the first learning model DM1 pre-learned using the first initial learning data group (step S291). When the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1, and the accuracy of the first learning model DM1 is less than a predetermined threshold value (step S292: yes), the learning data acquisition process is executed. In the learning data acquisition process, the learning data acquisition unit 212h acquires a new learning data group (step S293). Following the learning data acquisition process, the learning process is executed. In the learning process, the learning unit 213h discards the first learning model DM1 stored in the memory 202h (step S294). Then, the learning unit 213h performs a new learning process of newly learning the seventh learning model DM7 using the acquired new learning data group (step S295). On the other hand, when the captured images Im1~Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1 and the accuracy of the first learning model DM1 becomes above a predetermined threshold (step S292: No), the processor 201h does not execute the learning data acquisition process and the learning process, and ends the process.

[0198] Fig.38This is a flowchart showing a method of acquiring shape data Da in the ninth embodiment. Fig.38 The illustrated flow is repeatedly executed at predetermined time intervals, for example, from the time when vehicle 100 starts traveling without human control.

[0199] In the control information acquisition step, the control information acquisition unit 214h acquires the shape data Da by performing the following processing. Specifically, when the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300 are input to the first learning model DM1, and the accuracy of the first learning model DM1 is less than a predetermined threshold value (step S391: Yes), and when the new learning process of the seventh learning model DM7 is completed (step S392: Yes), the control information acquisition unit 214g performs the following processing. In this case, the control information acquisition unit 214g acquires the shape data Da by inputting the captured images Im1 to Im4 to the seventh learning model DM7 (step S393). On the other hand, when the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 are input into the first learning model DM1, and the accuracy of the first learning model DM1 becomes above a predetermined threshold (step S391: No), the control information acquisition unit 214h obtains the shape data Da by inputting the captured images Im1 to Im4 into the first learning model DM1 (step S394).

[0200] According to the ninth embodiment, the information processing device 200h can perform the following processing when the accuracy of the captured images Im1 to Im4 obtained by a part of the external cameras 300 among the multiple external cameras 300 is less than the threshold when it is input to the first learning model DM1 that has been learned. In this case, the information processing device 200h can use the training image Ti and the area correct answer label Lr that establish a corresponding new learning data set including the background image Bi and the vehicle image corresponding to the shooting area of ​​the external camera 300 that obtained the captured images Im1 to Im4 with an accuracy less than the threshold to newly learn the seventh learning model DM7. In this way, when the appearance state in the shooting area of ​​the external camera 300 among the multiple external cameras 300 has changed since the learning of the first learning model DM1, the accuracy of the machine learning models DM1 and DM7 can be suppressed from decreasing.

[0201] Furthermore, according to the ninth embodiment, when the new learning of the seventh learning model DM7 is completed, the information processing device 200h can obtain the shape data Da by inputting the captured images Im1 to Im4 to the seventh learning model DM7. Thus, it is possible to suppress the decrease in the accuracy of the shape data Da generated using the captured images Im1 to Im4 acquired by a part of the external cameras 300 among the plurality of external cameras 300.

[0202] J. Tenth Implementation Method: Fig.39 1 is a block diagram showing the structure of an information processing system 50v in the tenth embodiment. The information processing system 50v includes one or more vehicles 100v, a plurality of external cameras 300, and an information processing device 200v. In the present embodiment, a vehicle control device 110v mounted on the vehicle 100v obtains vehicle position information using the detection result output from the external camera 300 and generates a driving control signal. The other structures of the information processing system 50v are the same as those of the first embodiment unless otherwise specified. The same reference numerals are used for the same structures as those of the first embodiment and the description thereof is omitted.

[0203] The information processing device 200 v is constituted by a computer including a processor 201 v , a memory 202 v , an input / output interface 203 , and an internal bus 204 .

[0204] The memory 202 v stores various information including a program PG2 v for controlling the operation of the information processing device 200 v and a plurality of first learning models DM1 prepared for each external camera 300 .

[0205] The processor 201 v functions as an accuracy acquisition unit 211 , a learning data acquisition unit 212 , a learning unit 213 , and an updating unit 219 by executing a program PG2 v stored in the memory 202 v .

[0206] The updating unit 219 updates the first learning model DM1 stored in the vehicle control device 110v every time the first learning model DM1 is relearned so that the first learning model DM1 stored in the vehicle control device 110v is in the latest state.

[0207] The vehicle control device 110 v is constituted by a computer including a processor 111 v , a memory 112 v , an input / output interface 113 , and an internal bus 114 .

[0208] The memory 112 v stores various information including a program PG1 v for controlling the operation of the vehicle control device 110 v , a reference route RR, and a plurality of first learning models DM1 prepared for each external camera 300 .

[0209] The processor 111 v executes the program PG1 v stored in the memory 112 v to function as a control information acquisition unit 116 , a position acquisition unit 117 , a signal generation unit 118 , and a vehicle control unit 115 v .

[0210] The control information acquisition unit 116 acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1. In the present embodiment, the first learning model DM1 is prepared for each external camera 300. Thus, the control information acquisition unit 116 acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 corresponding to the external camera 300 that has acquired the captured images Im1 to Im4. At this time, the first learning model DM1 stored in the memory 112v is updated each time the relearning process is performed. Therefore, when the relearning process is completed in the information processing device 200v, the control information acquisition unit 116 acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model DM1 after the relearning process.

[0211] The position acquisition unit 117 acquires the vehicle position information using the shape data Da. The signal generation unit 118 generates a travel control signal. The vehicle control unit 115v controls the actuator group 120 using the generated travel control signal to cause the vehicle 100v to travel.

[0212] Fig.40 100v is a flowchart showing a processing procedure of the travel control of the vehicle 100v in the tenth embodiment. Fig.40 The illustrated flow is repeatedly executed at predetermined time intervals starting from the time when the vehicle 100v starts traveling without human control, for example.

[0213] In step S121, the position acquisition unit 117 acquires the vehicle position information using the detection result output from the external camera 300. In step S122, the signal generation unit 118 determines the target position to which the vehicle 100v should go next. In step S123, the signal generation unit 118 generates a travel control signal for causing the vehicle 100v to travel toward the determined target position. In step S124, the vehicle control unit 115v controls the actuator group 120 using the generated travel control signal and causes the vehicle 100v to travel according to the parameters represented by the travel control signal. The processor 111v repeatedly acquires the vehicle position information, determines the target position, generates the travel control signal, and controls the actuator in a predetermined cycle. According to the information processing system 50v in this embodiment, even if the vehicle 100v is not remotely controlled by the information processing device 200v, the vehicle 100v can be driven by the autonomous control of the vehicle 100v.

[0214] K. Other implementation methods: K-1. Other implementation methods 1: At least a part of the function of the information processing device 200, 200a to 200h, 200v may be a function of the vehicle control device 110, 110v, or a function of the external camera 300. For example, when all the functions of the information processing device 200, 200a to 200h, 200v are implemented by the external camera 300, the information processing device 200, 200a to 200h, 200v can obtain the coordinate values ​​Cr1 to Cr4 and Cb1 to Cb8 in the camera coordinate system without using the camera identifier.

[0215] K-2. Other implementation methods 2: In the memories 202, 202a to 202h, 202v of the information processing devices 200, 200a to 200h, 200v, one machine learning model DM commonly used for the captured images Im1 to Im4 respectively acquired by the plurality of external cameras 300 may be stored. Even in such a manner, when the appearance state in the captured area of ​​a part of the plurality of external cameras 300 has changed since the time when the machine learning model DM was learned, it is possible to suppress a decrease in the accuracy of the machine learning model DM.

[0216] K-3. Other implementation methods 3: The control information acquisition units 116, 214, 214a to 214h may also acquire control information by inputting the captured images Im1 to Im4 to the machine learning model DM after the learning process when the learning process is completed and the accuracy is above the threshold value regardless of which captured images Im1 to Im4 acquired by the plurality of external cameras 300 is input to the machine learning model DM after the learning process. That is, the control information acquisition units 116, 214, 214a to 214h may also acquire control information by inputting the captured images Im1 to Im4 to the machine learning model DM after the learning process when the ratio of the number of captured images Im1 to Im4 whose accuracy is above the threshold value after learning to the total number of the plurality of captured images Im1 to Im4 used to calculate the accuracy after learning is 100%. Even in such a manner, it is possible to suppress a decrease in the accuracy of the control information acquired using the captured images Im1 to Im4 acquired by a part of the plurality of external cameras 300.

[0217] K-4. Other implementation methods 4: When the transmitting unit 217 of the information processing device 200, 200a to 200h acquires the captured images Im1 to Im4 used as the training images Ti in the learning process, it may transmit a travel control signal different from the reference control signal to the vehicle 100. The reference control signal is any one of the travel control signal acquired by the control information acquisition unit 214f ​​and the travel control signal generated based on the signal generation parameters acquired by the control information acquisition units 214, 214a to 214e, 214g, and 214h. For example, when the captured images Im1 to Im4 used as the training images Ti in the learning process are acquired, the transmitting unit 217 may transmit the following travel control signal to the vehicle 100 in order to acquire the captured images Im1 to Im4 with less jitter and clarity. In this case, the transmitting unit 217 transmits the travel control signal for driving the vehicle 100 at an acceleration smaller than the acceleration indicated by the reference control signal to the vehicle 100. With this method, it is possible to easily generate a training image Ti with less jitter and clarity. In addition, when the captured images Im1 to Im4 used as training images Ti in the learning process are obtained, in order to further increase the number of captured images Im1 to Im4 that can be obtained within a certain period of time, the transmission unit 217 may also transmit the following travel control signal to the vehicle 100. In this case, the transmission unit 217 transmits the travel control signal for causing the vehicle 100 to travel at an acceleration greater than the acceleration indicated by the reference control signal to the vehicle 100. In this manner, more captured images Im1 to Im4 can be obtained within a certain period of time. In addition, when the captured images Im1 to Im4 used as training images Ti in the learning process are obtained, in order to obtain captured images Im1 to Im4 of the vehicle 100 including a plurality of different orientations, the transmission unit 217 may also transmit the following travel control signal to the vehicle 100. In this case, the transmission unit 217 transmits the travel control signal for causing the vehicle 100 to travel at a steering angle different from the steering angle indicated by the reference control signal to the vehicle 100. With such a method, it is possible to easily generate a plurality of training images Ti in which the vehicle 100 is oriented in different directions relative to the passage TR. In addition, when there are two or more paths on which the vehicle 100 can travel, the transmission unit 217 may also transmit the following travel control signal to the vehicle 100. In this case, the transmission unit 217 transmits the travel control signal generated using a reference path RR different from the reference path RR used in generating the reference control signal to the vehicle 100. With such a method, it is possible to cause more vehicles 100 to travel to the passage TR included in the shooting area that is the subject of the learning process.In summary, the information processing devices 200 and 200a to 200h can reduce the load required for preparing the training image Ti or make it easier to generate the desired training image Ti by transmitting the travel control signal different from the reference control signal to the vehicle 100.

[0218] K-5. Other implementation methods 5: In each of the first embodiment to the ninth embodiment, the information processing devices 200, 200a to 200h perform the processing from the acquisition of the vehicle position information to the generation of the travel control signal. In contrast, at least a part of the processing from the acquisition of the vehicle position information to the generation of the travel control signal may be performed by the vehicle 100. For example, the following (1) to (3) may be used.

[0219] (1) The information processing devices 200, 200a to 200h can obtain vehicle position information, determine the target position to which the vehicle 100 should go next, and generate a path from the current position of the vehicle 100 indicated by the obtained vehicle position information to the target position. The information processing devices 200, 200a to 200h can generate a path to the target position between the current position and the destination, or can generate a path to the destination. The information processing devices 200, 200a to 200h can send the generated path to the vehicle 100. The vehicle 100 can generate a travel control signal in a manner such that the vehicle 100 travels on the path received from the information processing devices 200, 200a to 200h, and control the actuator group 120 using the generated travel control signal.

[0220] (2) The information processing devices 200 and 200a to 200h can obtain vehicle position information and transmit the obtained vehicle position information to the vehicle 100. The vehicle 100 can determine the target position to which the vehicle 100 should go next, generate a path from the current position of the vehicle 100 indicated by the received vehicle position information to the target position, generate a travel control signal so that the vehicle 100 travels on the generated path, and control the actuator group 120 using the generated travel control signal.

[0221] (3) In the above-mentioned methods (1) and (2), the vehicle 100 may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used for at least one of the generation of the route and the generation of the driving control signal. The internal sensor is a sensor mounted on the vehicle 100. The internal sensor may include, for example, a sensor for detecting the motion state of the vehicle 100, a sensor for detecting the action state of each part of the vehicle 100, and a sensor for detecting the surrounding environment of the vehicle 100. Specifically, the internal sensor may include, for example, a camera, a LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyroscope sensor, and the like. For example, in the above-mentioned method (1), the information processing device 200, 200a to 200h may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor on the route when generating the route. In the above-mentioned method (1), the vehicle 100 may obtain the detection result of the internal sensor, and reflect the detection result of the internal sensor on the driving control signal when generating the driving control signal. In the above-mentioned method (2), the vehicle 100 can obtain the detection results of the internal sensors, and reflect the detection results of the internal sensors in the route when generating the route. In the above-mentioned method (2), the vehicle 100 can obtain the detection results of the internal sensors, and reflect the detection results of the internal sensors in the travel control signal when generating the travel control signal.

[0222] K-6. Other implementation methods 6: In the tenth embodiment, the vehicle 100v may be equipped with an internal sensor, and the detection result outputted from the internal sensor may be used for at least one of the generation of the route and the generation of the driving control signal. For example, the vehicle 100v may obtain the detection result of the internal sensor, and when generating the route, the detection result of the internal sensor may be reflected in the route. The vehicle 100v may obtain the detection result of the internal sensor, and when generating the driving control signal, the detection result of the internal sensor may be reflected in the driving control signal.

[0223] K-7. Other implementation methods 7: In the tenth embodiment described above, the vehicle 100v uses the detection result of the external camera 300, which is a sensor located outside the vehicle 100v, i.e., an external sensor, to obtain the vehicle position information. In contrast, the vehicle 100v may be equipped with an internal sensor, and the vehicle 100v may use the detection result of the internal sensor to obtain the vehicle position information, determine the target position to which the vehicle 100v should go next, generate a path from the current position of the vehicle 100v represented by the obtained vehicle position information to the target position, generate a travel control signal for traveling on the generated path, and use the generated travel control signal to control the actuator group 120. In this case, the vehicle 100v can travel without using the detection result of the external sensor at all. It should be noted that the vehicle 100v may also obtain the target arrival time and congestion information from the outside of the vehicle 100v, so that the target arrival time and congestion information are reflected in at least one of the path and the travel control signal.

[0224] K-8. Other implementation methods 8: In each of the first embodiment to the ninth embodiment, the information processing device 200, 200a to 200h automatically generates a travel control signal to be transmitted to the vehicle 100. In contrast, the information processing device 200, 200a to 200h may generate a travel control signal to be transmitted to the vehicle 100 in accordance with an operation of an external operator located outside the vehicle 100. For example, the external operator may operate an operating device including a display that displays the captured images Im1 to Im4 outputted from an external sensor, a steering wheel for remotely operating the vehicle 100, an accelerator pedal, an actuator pedal, and a communication device for communicating with the information processing device 200, 200a to 200h by wired communication or wireless communication, and the information processing device 200, 200a to 200h may generate a travel control signal in accordance with the operation applied to the operating device.

[0225] K-9. Other implementation methods 9: In each of the above-mentioned embodiments, the vehicle 100, 100v may be provided with a structure capable of being moved by unmanned driving, for example, it may be in the form of a stand having the structure described below. Specifically, in order for the vehicle 100, 100v to perform the three functions of "driving", "turning" and "stopping" by unmanned driving, it is sufficient to have at least a vehicle control device 110 and an actuator group 120. In the case where the vehicle 100, 100v obtains information from the outside for unmanned driving, the vehicle 100, 100v may also be provided with a communication device 130. That is, the vehicle 100, 100v capable of being moved by unmanned driving may not be equipped with at least a part of the interior components such as the driver's seat and the instrument panel, may not be equipped with at least a part of the exterior components such as the bumper and the fender, and may not be equipped with the body shell. In this case, the remaining parts such as the body shell can be assembled to the vehicle 100, 100v until the vehicle 100, 100v is shipped from the factory FC, or the remaining parts such as the body shell can be assembled to the vehicle 100, 100v after the vehicle 100, 100v is shipped from the factory FC without being assembled to the vehicle 100, 100v. Each part can be assembled from any direction such as the upper side, lower side, front side, rear side, right side or left side of the vehicle 100, 100v, and can be assembled from the same direction or from different directions. It should be noted that the stand method can also be positioned in the same way as the vehicle 100, 100v in the above-mentioned embodiments.

[0226] K-10. Other implementation methods 10: Vehicles 100 and 100v can be manufactured by combining a plurality of modules. A module means a unit composed of a plurality of components gathered according to the position and function of the vehicle 100 and 100v. For example, the rack of the vehicle 100 and 100v can be manufactured by combining a front module constituting the front part of the rack, a central module constituting the central part of the rack, and a rear module constituting the rear part of the rack. It should be noted that the number of modules constituting the rack is not limited to 3, and may be less than 2 or more than 4. In addition, the components constituting the rack or the components constituting the parts of the vehicle 100 and 100v that are different from the rack may be modularized in addition to the components constituting the rack. In addition, various modules may include any exterior components such as bumpers and grilles, and any interior components such as seats and consoles. In addition, not limited to the vehicle 100 and 100v, any form of mobile body can be manufactured by combining a plurality of modules. Such a module can be manufactured, for example, by joining a plurality of components by welding, fixings, etc., or by molding at least a part of the components constituting the module as one component by casting. The molding technique of molding a component, especially a relatively large component, in one piece is also called Gigacasting or Megacasting. For example, the front module, the center module, and the rear module mentioned above can also be manufactured using Gigacasting.

[0227] K-11. Other implementation methods 11: The transportation of the vehicles 100 and 100 v by the driving of the unmanned vehicles 100 and 100 v is also referred to as "self-propelled transportation". In addition, the structure for realizing self-propelled transportation is also referred to as "vehicle remote control autonomous driving transportation system". In addition, the production method of producing the vehicles 100 and 100 v by self-propelled transportation is also referred to as "self-propelled production". In self-propelled production, for example, in the factory FC that manufactures the vehicles 100 and 100 v, at least a part of the transportation of the vehicles 100 and 100 v is realized by self-propelled transportation.

[0228] K-12. Other implementations 12: In the above-mentioned embodiments, part or all of the functions and processes implemented in software may also be implemented in hardware. In addition, part or all of the functions and processes implemented in hardware may also be implemented in software. As hardware for implementing the various functions in the above-mentioned embodiments, various circuits such as integrated circuits and discrete circuits may be used.

[0229] The present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various structures within the scope of its main purpose. For example, the technical features of the embodiments corresponding to the technical features in each method recorded in the invention content column can be appropriately replaced or combined in order to solve part or all of the above-mentioned problems or to achieve part or all of the above-mentioned effects. In addition, as long as the technical feature is not described as a necessary technical feature in this specification, it can be appropriately deleted. Description of Reference Numerals

[0230] 10e…measurement point, 50, 50a~50h, 50v…information processing system, 100, 100v…vehicle, 110, 110v…vehicle control device, 111, 111v…processor of the vehicle control device, 112, 112v…memory of the vehicle control device, 113…input / output interface of the vehicle control device, 114…internal bus of the vehicle control device, 115, 115v…vehicle control unit, 116, 214, 214a~214h…control information acquisition unit, 117, 215, 215c, 215d…position acquisition unit, 118, 216…signal generation unit, 120…actuator group, 130…communication device of the vehicle, 200, 200a~200h, 200 v…information processing device, 201, 201a~201h, 201v…processor of information processing device, 202, 202a~202h, 202v…memory of information processing device, 203…input / output interface of information processing device, 204…internal bus of information processing device, 205…communication device of information processing device, 211, 211b~211g…accuracy acquisition unit, 212, 212a~212h…learning data acquisition unit, 213, 213a~213h…learning unit, 217…transmission unit, 218…appearance acquisition unit, 219…updating unit, 300…external camera, A1…needed area, A2…unnecessary area, Bi…background image, Br…actual background image, B v…virtual background image, C…center of gravity, Cb1~Cb8…stereo coordinate values, Ci…synthesized image, Cr1~Cr4…rectangular coordinate values, D…first distance, ΔD…observation error, Da…shape data, Db1…rectangular database, Db2…stereo database, Dc…third distance, DM…machine learning model, DM1…first learning model, DM2…second learning model, DM3…third learning model, DM4…fourth learning model, DM5…fifth learning model, DM6…sixth learning model, DM7…seventh learning model, Do…observed distance, Dp…estimated distance, Dt…second distance, FC…factory, GC…global coordinate system, h…height of the vehicle’s measuring point from the road surface, H…external phase The height of the vehicle from the road surface, Im1…original image, Im2…corrected image, Im3…rotated image, Im4…processed image, Im1~Im4…captured images, Im5…first mask image, Im6…second mask image, Lc…contour correct answer label, Lr…region correct answer label, Ms…mask region, P0…basic coordinate point, P1…first coordinate point, P2…second coordinate point, P3…image coordinate point, PG1, PG1v, PG2, PG2a~PG2h, PG2v…program, Pa…camera parameter, Pc…captured coordinate point, Pf…fixed coordinate point, PL1…first location, PL2…second location, Pv…vehicle coordinate point, R0…basic circumscribed rectangle, R1…first circumscribed rectangle,R2…second circumscribed rectangle, RR…reference path, Rb…vehicle circumscribed cuboid, Rc…vehicle circumscribed rectangle, Rs…road surface, SA…surrounding area, TA…object area, TR…path, Ti…training image, V…movement vector, Vb1~Vb8…vertices of vehicle circumscribed cuboid, Vc1~Vc4…vertices of vehicle circumscribed rectangle, Vi…virtual vehicle image. ,

Claims

1. An information processing device for moving a mobile object capable of being moved by unmanned driving, the information processing device comprising: an accuracy acquisition unit that acquires the accuracy of a machine learning model that, when receiving an image captured by capturing the moving body using an external camera disposed at a location different from the moving body, outputs at least one of a control signal that specifies an action of the moving body in order to move the moving body by the unmanned driving and a parameter used when generating the control signal; a learning data acquisition unit, which acquires a learning data group for learning the machine learning model when the accuracy is less than a predetermined threshold; and a learning unit that performs a learning process for learning the machine learning model using the acquired learning data group, The learning data group includes one or more learning data sets that establish corresponding training images and correct labels including a background image representing an object area and a moving body image representing the moving body, wherein the object area includes a path along which the moving body moves and a surrounding area of ​​the path.

2. The information processing device according to claim 1, wherein: The machine learning model is pre-learned using an initial learning data set, The learning data acquisition unit acquires an additional learning data group as the learning data group for additional learning of the machine learning model when the accuracy of the captured image acquired by a part of the external cameras among the plurality of external cameras is less than the threshold value when the captured image is input into the machine learning model. The learning unit performs an additional learning process of additionally learning the machine learning model using the acquired additional learning data group. The initial learning data set includes a plurality of initial learning data sets corresponding to the respective shooting areas of the plurality of external cameras, wherein the plurality of initial learning data sets establish correspondence between training images including the background image and the moving object image corresponding to the shooting area and correct labels, The additional learning data set includes one or more additional learning data sets that establish correspondence between the training images and the correct answer labels, including the background image and the moving object image corresponding to the shooting area of ​​the external camera that obtained the shot image with the accuracy less than the threshold.

3. The information processing device according to claim 1, wherein: The machine learning model is pre-learned using an initial learning data set, The learning data acquisition unit acquires a new learning data set as the learning data set for newly learning a machine learning model different from the machine learning model, when the accuracy of the captured image acquired by a part of the external cameras among the plurality of external cameras is less than the threshold value when the captured image is input into the machine learning model. The learning unit performs a new learning process to learn a new machine learning model using the acquired new learning data set. The initial learning data set includes a plurality of initial learning data sets corresponding to the respective shooting areas of the plurality of external cameras, wherein the plurality of initial learning data sets establish correspondence between training images including the background image and the moving object image corresponding to the shooting area and correct labels, The new learning data set includes one or more new learning data sets that establish correspondence between the training images and the correct answer labels, including the background image and the moving object image corresponding to the shooting area of ​​the external camera that obtained the shot image with the accuracy less than the threshold.

4. The information processing device according to claim 1, wherein: The training image is at least one of the captured image and the synthesized image. The synthesized image is generated by synthesizing a virtual moving object image that virtually reproduces the moving object with the background image of either an actual background image obtained by photographing the target area using the external camera or a virtual background image that virtually reproduces the target area.

5. The information processing device according to claim 4, wherein: The information processing device further includes an appearance acquisition unit that acquires appearance information indicating an appearance state of the moving object. The learning unit executes the learning process using the virtual moving object image that virtually reproduces the moving object in the appearance state determined by the appearance information.

6. The information processing device according to claim 1, wherein: The learning unit executes the learning process while the unmanned driving control is being executed.

7. The information processing device according to claim 1, wherein: The information processing device further includes a control information acquisition unit that acquires control information of at least one of the control signal and the parameter by inputting the captured image into the machine learning model. When the learning process is completed, the control information acquisition unit acquires the control information by inputting the captured image to the machine learning model after the learning process.

8. The information processing device according to claim 1, wherein: The information processing device further includes a control information acquisition unit that acquires control information of at least one of the control signal and the parameter by inputting the captured image into the machine learning model. The accuracy acquisition unit further acquires the accuracy of the machine learning model after the learning process. When the learning process is completed and the accuracy of the machine learning model after the learning process becomes greater than the threshold, the control information acquisition unit acquires the control information by inputting the captured image into the machine learning model after the learning process.

9. The information processing device according to claim 8, wherein: The accuracy acquisition unit acquires the accuracy of the machine learning model after the learning process using the plurality of captured images acquired by using the external camera to capture the moving object moving behind the moving object included in the training image. When the ratio of the number of captured images for which the accuracy of the machine learning model after the learning process is above the threshold to the total number of the multiple captured images used to calculate the accuracy of the machine learning model after the learning process is above a reference value, the control information acquisition unit acquires the control information by inputting the captured images into the machine learning model after the learning process.

10. The information processing device according to claim 1, wherein: The information processing device further comprises: a control information acquisition unit that acquires control information of at least one of the control signal and the parameter by inputting the captured image into the machine learning model; and a transmitting unit, transmitting the control signal to the moving object, The training image is the captured image obtained by capturing the moving object that is performing the unmanned driving control using the external camera. When the captured image used as the training image in the learning process is obtained, the transmitting unit transmits to the moving object the control signal different from the control signal obtained by the control information obtaining unit and the control signal generated based on the parameter obtained by the control information obtaining unit.

11. An information processing system for moving a mobile object capable of being moved by unmanned driving, the information processing system comprising: Mobile body; A plurality of external cameras, arranged at a location different from the mobile object; and The information processing device according to any one of claims 1 to 10.

12. An information processing method for moving a mobile object capable of being moved by unmanned driving, the information processing method comprising: an accuracy acquisition step of acquiring the accuracy of a machine learning model, wherein the machine learning model outputs at least one of a control signal that specifies an action of the mobile body in order to move the mobile body by the unmanned driving and a parameter used in generating the control signal when a captured image of the mobile body is input and captured by an external camera installed at a location different from the mobile body; a learning data acquisition step of acquiring a learning data group for learning the machine learning model when the accuracy is less than a predetermined threshold; and a learning step of performing a learning process of causing the machine learning model to learn using the acquired learning data set, The learning data group includes one or more learning data sets that establish corresponding training images and correct labels including a background image representing an object area and a moving body image representing the moving body, wherein the object area includes a path along which the moving body moves and a surrounding area of ​​the path.

Images