Information processing apparatus, information processing system, and information processing method

Abstract

Provided is a technique whereby at least one of a control signal and a parameter to be used when generating the control signal can be acquired with high accuracy even when the appearance state of a target region changes. An information processing device is provided with: a change information acquisition unit that acquires change information; and at least one of a learning unit and a notification control unit, the learning unit learning a machine learning model using a plurality of pieces of training data in which a training image including the object region after the change of the appearance state and the moving body is associated with a positive solution label when it is determined that the appearance state has changed using the change information. When the change information is used to determine that the appearance state has changed, the notification control unit notifies the manager of learning request information for learning the machine learning model.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. Background Art

[0002] Conventionally, there is known a technology for automatically driving a vehicle by remotely controlling the vehicle while monitoring the vehicle's travel using a camera outside the vehicle (Patent Document 1). Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application No. 2017-538619 Summary of the Invention Problems to be solved by the invention

[0004] To enable unmanned movement of mobile objects such as vehicles, images containing the mobile object are sometimes fed into a machine learning model. The model then detects the mobile object from the image and obtains parameters for generating control signals that dictate the mobile object's movements. However, due to, for example, dirt on the path where the mobile object moves, the appearance of the target area, including the path and its surrounding areas, may change. The inventors of this application have discovered that if the appearance of the target area changes, the accuracy of mobile object detection may decrease if a machine learning model trained using images before the change in appearance is fed with images after the change in appearance. Means for solving problems

[0005] The present disclosure can be implemented as the following aspects.

[0006] (1) According to a first aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body capable of moving by unmanned driving comprises: a change information acquisition unit that acquires change information related to a change in appearance state in a target area, the target area being included in a shooting area captured by an external camera installed at a location different from the mobile body and including a path along which the mobile body moves and a surrounding area of ​​the path; and at least one of a learning unit and a notification control unit that, when judging that the appearance state has changed using the change information, causes a machine learning model to output at least one of a control signal that specifies the action of the mobile body moving by unmanned driving and a parameter used in generating the control signal when inputting a captured image captured by capturing the target area and the mobile body using the external camera. The learning unit learns using a plurality of training data in which training images including the target area and the mobile body after the appearance state has changed and correct labels are associated with each other; and the notification control unit, when judging that the appearance state has changed using the change information, causes a manager to be notified of learning request information for causing the machine learning model to learn. According to this method, when it is determined that the appearance state of the object area has changed using the change information, the information processing device can perform the following processing. In this case, the information processing device can use a plurality of training data corresponding to the training image of the object area containing the moving body and the appearance state change and the correct answer label to make the machine learning model learn. In this way, when the appearance state of the object area has changed, the accuracy of the control signal output from the machine learning model and the parameters used in generating the control signal can be suppressed. In addition, according to this method, when it is determined that the appearance state of the object area has changed using the change information, the information processing device can notify the administrator of the learning request information for making the machine learning model learn. In this way, the information processing device can notify the administrator of the determination that the machine learning model needs to learn. As a result, the information processing device can urge the administrator to execute the measures for making the machine learning model learn. (2) According to a second aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body capable of moving unmanned includes a learning unit that learns a machine learning model that outputs at least one of a control signal that specifies the movement of the mobile body capable of moving unmanned, and a parameter used when generating the control signal, when inputting a captured image of a target area and the mobile body captured by an external camera, wherein the target area is included in a captured area of ​​the external camera, which is set at a location different from the mobile body, and includes a path along which the mobile body moves and a surrounding area of ​​the path, and the target area is classified into a plurality of appearance states based on changes in appearance. The learning unit learns the machine learning model for each appearance state using M training data corresponding to training images and correct labels containing the target area and the mobile body classified into one of the appearance states, and N training data corresponding to training images and correct labels containing the target area and the mobile body classified into the remaining other appearance states, where M is an integer greater than or equal to 2 and N is an integer greater than or equal to 0 and less than M. According to this method, the information processing device can train a machine learning model by using multiple training images containing a moving object and an object region classified as one appearance state, and multiple training images containing a moving object and an object region classified as another appearance state. Thus, the information processing device can prepare a machine learning model suitable for obtaining at least one of a control signal and a parameter used to generate the control signal using captured images containing a moving object and an object region classified as one appearance state. (3) According to a third aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body capable of moving by unmanned driving includes a learning unit that learns a machine learning model that outputs at least one of a control signal that specifies the action of the mobile body moving by unmanned driving and a parameter used in generating the control signal when inputting a captured image obtained by capturing a target area and the mobile body using an external camera and an appearance state of the target area, wherein the target area is included in a captured area of ​​the external camera set at a location different from the mobile body and includes a path along which the mobile body moves and a surrounding area of ​​the path, and the target area is classified into a plurality of appearance states based on changes in appearance. The learning unit learns the machine learning model using a plurality of training data corresponding to training images containing the target area and the mobile body, correct labels, and accompanying information indicating the appearance state of the target area contained in the training images. According to this aspect, the information processing device can learn the machine learning model using a plurality of training data corresponding to training images containing the target area and the mobile body, correct labels, and accompanying information indicating the appearance state of the target area contained in the training images. Thereby, the information processing device can prepare a machine learning model that learns the appearance states of the object area in an associated manner. (4) According to a fourth aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body capable of moving by unmanned driving comprises: a state information acquisition unit that acquires state information related to an appearance state in a target area, the target area being included in a shooting area captured by an external camera installed at a location different from the mobile body and including a path along which the mobile body moves and an area surrounding the path; and a control information acquisition unit that acquires at least one of a control signal and a parameter used in generating the control signal by inputting the captured image into a machine learning model that has been learned and outputs at least one of a control signal that specifies an action of the mobile body moving by unmanned driving and a parameter used in generating the control signal when a captured image captured by the external camera is input. When the appearance state determined by the state information is different from the appearance state at the time of learning the machine learning model, the control information acquisition unit inputs the captured image into the machine learning model that has been learned using a plurality of training data in which training images of the target area and the mobile body, including the appearance state determined by the state information, are associated with correct labels. According to this method, when the appearance state of the object area determined by the state information differs from the appearance state of the object area in the training image used when learning the machine learning model, the information processing device can perform the following processing. In this case, the information processing device can obtain at least one of a control signal and a parameter used when generating the control signal by inputting a captured image to a machine learning model that has been learned using a plurality of training data that establish a correspondence between the training image containing the moving object and the object area whose appearance state is determined by the state information and the correct answer label. In this way, when the appearance state of the object area determined by the state information differs from the appearance state of the object area in the training image used when learning the machine learning model, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (5) According to the fifth aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body that can move by unmanned driving comprises: a state information acquisition unit that acquires state information related to the appearance state in a target area, wherein the target area is included in a shooting area of ​​an external camera set at a place different from the mobile body, and includes a path for the mobile body to move and a surrounding area of ​​the path; a model acquisition unit that acquires a machine learning model that outputs a control signal that specifies the action of the mobile body that moves by the unmanned driving and at least one of the parameters used when generating the control signal when a shot image obtained by shooting the target area and the mobile body using the external camera is input; and a control information acquisition unit that acquires the control signal by inputting the shot image to the machine learning model acquired by the model acquisition unit. and at least one of the parameters, wherein the object area is classified into a plurality of appearance states based on changes in appearance, the machine learning model is prepared for each appearance state, and the plurality of machine learning models prepared for each appearance state are pre-learned using M training data corresponding to training images containing the object area and the moving object classified into one of the appearance states and correct labels, and N training data corresponding to training images containing the object area and the moving object classified into the remaining other appearance states and correct labels, respectively, where M is an integer greater than 2 and N is an integer greater than 0 and less than M. The model acquisition unit acquires the machine learning model selected based on the appearance state from the plurality of machine learning models prepared for each appearance state. According to this aspect, the information processing device can acquire the machine learning model selected based on the appearance state identified by the state information from the plurality of machine learning models prepared for each appearance state of the object area. Furthermore, the information processing device can acquire at least one of the control signal and the parameters used in generating the control signal by inputting the captured image into the machine learning model corresponding to the appearance state of the object area included in the captured image. This can suppress the decrease in accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (6) According to a sixth aspect of the present disclosure, an information processing device is provided. The information processing device for moving a mobile body capable of moving by unmanned driving comprises: a state information acquisition unit for acquiring state information related to an appearance state in a target area, the target area being included in a shooting area captured by an external camera installed at a location different from the mobile body and including a path along which the mobile body moves and an area surrounding the path; and a control information acquisition unit for acquiring at least one of a control signal for defining an action of the mobile body moving by unmanned driving and a parameter used in generating the control signal by inputting a captured image captured by capturing the target area and the mobile body using the external camera and the appearance state determined by the state information into a learned machine learning model, wherein the target area is classified into a plurality of appearance states according to changes in appearance, and the machine learning model is pre-learned using a plurality of training data in which training images containing the target area and the mobile body, correct labels, and accompanying information indicating the appearance state of the target area contained in the training images are associated with each other. According to this method, an information processing device inputs a captured image and the appearance state of the object area determined by state information into a machine learning model that has been pre-learned using a plurality of training data sets that correspond to training images containing an object area and a moving object, correct labels, and accompanying information indicating the appearance state of the object area contained in the training images. This allows the information processing device to obtain at least one of a control signal and parameters used to generate the control signal. This prevents a decrease in the accuracy of the control signal output from the machine learning model and the parameters used to generate the control signal. (7) In the above-described method, the factor that affects the change in the appearance state may be at least one of dirt generated in the target area, unevenness generated in the target area, an object different from the mobile body arranged in a manner capable of moving in the target area, and the road surface color of the passage. According to this method, when the factor that affects the change in the appearance state of the target area is at least one of dirt generated in the target area, unevenness generated in the target area, an object arranged in a manner capable of moving in the target area, and the road surface color of the passage, the following can be achieved. In this case, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (8) In the above aspect, the machine learning model can be learned while the unmanned driving control is being executed. According to this aspect, the information processing device can cause the machine learning model to learn while the mobile object is executing the unmanned driving control. (9) In the above-described embodiment, the training image may be at least one of a synthetic image and the captured image, the synthetic image being generated by synthesizing a virtual moving body image that virtually reproduces the moving body with a background image of either an actual background image obtained by capturing the object area using the external camera or a virtual background image that virtually reproduces the object area. According to this embodiment, the information processing device can use the synthetic image generated by synthesizing the virtual moving body image with a background image of either an actual background image or a virtual background image as a training image used for learning the machine learning model. (10) According to a seventh aspect of the present disclosure, an information processing system is provided. The information processing system for moving a mobile object capable of moving unmanned comprises the mobile object, an external camera installed in a location different from the mobile object, and the information processing device described in the above aspect. According to this aspect, the information processing system can learn a machine learning model for at least one of an output control signal and a parameter used when generating the control signal, or can use the learned machine learning model to move the mobile object unmanned. (11) According to an eighth aspect of the present disclosure, an information processing method is provided. The information processing method for moving a mobile body capable of moving by unmanned driving comprises: a change information acquisition step for acquiring change information related to a change in the appearance state of a target area, the target area being included in a shooting area of ​​an external camera installed at a location different from the mobile body and including a path along which the mobile body moves and a surrounding area of ​​the path; and at least one of a learning step and a notification step for learning a machine learning model that outputs at least one of a control signal that specifies the action of the mobile body moving by unmanned driving and a parameter used in generating the control signal when inputting a captured image of the target area and the mobile body captured by the external camera using a plurality of training data in which training images including the target area and the mobile body after the appearance state change and correct labels are associated with each other; and a notification step for notifying a manager of learning request information for causing the machine learning model to learn when the change information is used to determine that the appearance state has changed. According to this method, when it is determined that the appearance state of the object area has changed using the change information, the following processing can be performed. In this case, the machine learning model can be learned using a plurality of training data corresponding to the training image and the correct answer label containing the moving body and the object area after the appearance state has changed. In this way, when the appearance state of the object area has changed, the accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal can be suppressed. In addition, according to this method, when it is determined that the appearance state of the object area has changed using the change information, the learning request information for making the machine learning model learn can be reported to the manager. In this way, the judgment that the machine learning model needs to learn can be notified to the manager. As a result, the manager can be urged to execute the measures for making the machine learning model learn. (12) According to a ninth aspect of the present disclosure, an information processing method is provided. An information processing method for enabling a mobile body capable of moving by unmanned driving comprises: a preparation step of preparing a plurality of training images including a target area and the mobile body, the target area including a captured area captured by an external camera set at a location different from the mobile body and including a path along which the mobile body moves and an area surrounding the path; and a learning step of learning a machine learning model that outputs at least one of a control signal that specifies the action of the mobile body moved by the unmanned driving and a parameter used when generating the control signal when a captured image obtained by capturing the target area and the mobile body using the external camera is input, wherein the target area is classified into a plurality of appearance states according to changes in appearance, and in the learning step, M training data corresponding to the training images including the target area and the mobile body classified into one of the appearance states and the correct answer labels are established, and N training data corresponding to the training images including the target area and the mobile body classified into the remaining other appearance states and the correct answer labels are established, and the machine learning model is learned for each of the appearance states, where M is an integer greater than or equal to 2 and N is an integer greater than or equal to 0 and less than M. According to this method, a machine learning model can be trained using more training images containing a moving object and an object region classified as one appearance state than training images containing a moving object and an object region classified as another appearance state. This allows for the preparation of a machine learning model suitable for obtaining at least one of a control signal and a parameter used in generating the control signal using captured images containing a moving object and an object region classified as one appearance state. (13) According to the tenth aspect of the present disclosure, an information processing method is provided. The information processing method for moving a mobile body capable of moving by unmanned driving comprises: a preparation step of preparing a plurality of training images including a target area and the mobile body, the target area including a shooting area of ​​an external camera installed at a location different from the mobile body, and including a path along which the mobile body moves and a surrounding area of ​​the path; and a learning step of learning a machine learning model that outputs at least one of a control signal that specifies the action of the mobile body moving by the unmanned driving and a parameter used when generating the control signal when inputting a shot image obtained by shooting the target area and the mobile body using the external camera and an appearance state of the target area, wherein the target area is classified into a plurality of appearance states according to changes in appearance, and in the learning step, the machine learning model is learned using a plurality of training data corresponding to the training images including the target area and the mobile body, the correct answer labels, and the accompanying information indicating the appearance state of the target area included in the training images. According to this method, a machine learning model can be trained using a plurality of training data sets corresponding to the training images containing the object area and the moving object, the correct answer labels, and the accompanying information indicating the appearance of the object area contained in the training images. This allows for the preparation of a machine learning model that has learned the appearance of the object area in a correlated manner. (14) According to an eleventh aspect of the present disclosure, an information processing method is provided. An information processing method for enabling a mobile body capable of moving by unmanned driving comprises: a state information acquisition step for acquiring state information related to an appearance state in an object area, wherein the object area is included in a shooting area of ​​an external camera set at a location different from the mobile body, and includes a path for the mobile body to move and a surrounding area of ​​the path; and a control information acquisition step for acquiring at least one of the control signal and the parameter by inputting the shot image into a machine learning model that has been learned and outputs a control signal that specifies the action of the mobile body that moves by the unmanned driving when a shot image obtained by shooting the object area and the mobile body using the external camera is input. When the appearance state determined by the state information is different from the appearance state of the object area in the training image input when learning the machine learning model, in the control information acquisition step, the shot image is input into the machine learning model that has been learned using a plurality of training data that have established correspondence between the training image of the object area and the mobile body that includes the appearance state determined by the state information and a correct answer label. According to this method, when the appearance state of the object area determined by the state information differs from the appearance state of the object area in the training image used when learning the machine learning model, the following processing can be performed. In this case, at least one of a control signal and a parameter used in generating the control signal can be obtained by inputting a captured image into a machine learning model that has been trained using a plurality of training data that establish a correspondence between the training image containing the moving object and the object area whose appearance state is determined by the state information and the correct answer label. In this way, when the appearance state of the object area determined by the state information differs from the appearance state of the object area in the training image used when learning the machine learning model, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used in generating the control signal. (15) According to the twelfth embodiment of the present disclosure, an information processing method is provided. The information processing method for moving a mobile body capable of moving by unmanned driving comprises: a state information acquisition step for acquiring state information related to the appearance state in a target area, wherein the target area is included in a shooting area of ​​an external camera set at a place different from the mobile body, and includes a path for the mobile body to move and a peripheral area of ​​the path; a model acquisition step for acquiring a machine learning model that outputs a control signal that specifies the action of the mobile body that moves by the unmanned driving and at least one of the parameters used when generating the control signal when a shot image obtained by shooting the target area and the mobile body using the external camera is input; and a control information acquisition step for acquiring the control signal by inputting the shot image into the machine learning model acquired in the model acquisition step. At least one of a signal and a parameter is obtained, wherein the object area is classified into a plurality of appearance states based on a change in appearance, and the machine learning model is prepared for each appearance state. The plurality of machine learning models prepared for each appearance state are pre-learned using M training data corresponding to training images containing the object area and the moving object classified into one of the appearance states and correct labels, and N training data corresponding to training images containing the object area and the moving object classified into the remaining other appearance states and correct labels, respectively, where M is an integer greater than 2 and N is an integer greater than 0 and less than M. In the model acquisition step, the machine learning model selected based on the appearance state from the plurality of machine learning models prepared for each appearance state is acquired. According to this method, a machine learning model selected based on the appearance state determined by the state information from the plurality of machine learning models prepared for each appearance state of the object area can be acquired. Furthermore, by inputting a captured image into the machine learning model corresponding to the appearance state of the object area included in the captured image, at least one of a control signal and a parameter used in generating the control signal can be acquired. This can suppress the decrease in accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (16) According to the thirteenth aspect of the present disclosure, an information processing method is provided. The information processing method for moving a mobile body capable of moving by unmanned driving comprises: a state information acquisition step for acquiring state information related to an appearance state in a target area, the target area being included in a shooting area of ​​an external camera installed at a location different from the mobile body and including a path along which the mobile body moves and a surrounding area of ​​the path; and a control information acquisition step for acquiring at least one of a control signal for determining the action of the mobile body moving by unmanned driving and a parameter used in generating the control signal by inputting a learned machine learning model that outputs a captured image obtained by capturing the target area and the mobile body using the external camera and the appearance state determined by the state information, wherein the target area is classified into a plurality of appearance states according to changes in appearance, and the machine learning model is pre-learned using a plurality of training data in which training images including the target area and the mobile body, correct labels, and accompanying information indicating the appearance state of the target area included in the training images are associated with each other. According to this method, a captured image and the appearance state of the object area determined by state information are input to a machine learning model that has been pre-learned using a plurality of training data sets that correspond to training images containing an object area and a moving object, correct labels, and accompanying information indicating the appearance state of the object area contained in the training images. This allows the acquisition of at least one of a control signal and parameters used in generating the control signal. This can suppress the reduction in accuracy of the control signal output from the machine learning model and the parameters used in generating the control signal. The present disclosure can be implemented in various forms other than the aforementioned information processing device, information processing system, and information processing method. For example, it can be implemented in forms such as methods for manufacturing an information processing device, an information processing system, and a mobile object, methods for controlling an information processing device, an information processing system, and a mobile object, computer programs for implementing the control methods, and non-transitory recording media recording the computer programs. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a diagram showing the configuration of the information processing system in the first embodiment. Figure 2 This is a flowchart illustrating a vehicle travel control method in the first embodiment. Figure 3 This is a flowchart illustrating a method for relearning the first learning model in the first embodiment. Figure 4 This is a flowchart showing a method for acquiring shape data in the first embodiment. Figure 5 This is a flowchart showing an example of a method for calculating the position of a vehicle. Figure 6 It shows that the Figure 5 Schematic diagram of various image examples showing the method. Figure 7 This is a diagram for explaining the details of the coordinate point calculation process. Figure 8 This is a diagram for explaining the calculation method of basic coordinate points. Figure 9 This is the first diagram for explaining the details of the position conversion process. Figure 10 This is a second diagram for explaining the details of the position conversion step. Figure 11 It is a diagram showing the configuration of an information processing system in the second embodiment. Figure 12 It is a diagram showing the configuration of an information processing system in the third embodiment. Figure 13 This is a flowchart illustrating a method for relearning the first learning model in the third embodiment. Figure 14 It is a diagram showing the configuration of an information processing system in a fourth embodiment. Figure 15 is a flowchart illustrating a learning method of the second learning model. Figure 16 This is a flowchart showing a method for acquiring shape data using the second learning model. Figure 17 It is a diagram showing the configuration of an information processing system in the fifth embodiment. Figure 18 : is a flowchart showing a learning method of the third learning model. Figure 19 This is a flowchart showing a method for acquiring shape data using the third learning model. Figure 20 This is a diagram showing an example of a method for generating training images. Figure 21 It is a diagram showing the configuration of an information processing system in the seventh embodiment. Figure 22 This is a diagram for explaining the coordinate values ​​of a rectangle. Figure 23 : is a flowchart showing a re-learning method of the fourth learning model. Figure 24 This is a flowchart showing a method for acquiring rectangle coordinate values ​​using the fourth learning model. Figure 25It is a diagram showing the configuration of an information processing system in the eighth embodiment. Figure 26 This is a diagram for explaining three-dimensional coordinate values. Figure 27 : is a flowchart showing a re-learning method of the fifth learning model. Figure 28 This is a flowchart showing a method for acquiring three-dimensional coordinate values ​​using the fifth learning model. Figure 29 It is a diagram showing the configuration of an information processing system in a ninth embodiment. Figure 30 : is a flowchart showing a re-learning method of the sixth learning model. Figure 31 This is a flowchart showing a method for acquiring vehicle position information using the sixth learning model. Figure 32 It is a diagram showing the configuration of an information processing system in a tenth embodiment. Figure 33 : is a flowchart showing the re-learning method of the seventh learning model. Figure 34 This is a flowchart showing a method for acquiring a travel control signal using the seventh learning model. Figure 35 It is a diagram showing the configuration of an information processing system in the eleventh embodiment. Figure 36 is a flowchart illustrating a transfer learning method of a first learning model. Figure 37 This is a flowchart showing a method for acquiring shape data in the eleventh embodiment. Figure 38 It is a diagram showing the configuration of an information processing system in the twelfth embodiment. Figure 39 This is a flowchart showing a learning method of a learning model in the twelfth embodiment. Figure 40 This is a flowchart showing a method for acquiring shape data in the twelfth embodiment. Figure 41 It is a diagram showing the configuration of an information processing system in a thirteenth embodiment. Figure 42 This is a flowchart illustrating a vehicle travel control method in the thirteenth embodiment. DETAILED DESCRIPTION

[0008] A. First embodiment: A-1. Information Processing System Structure: Figure 1This diagram illustrates the configuration of an information processing system 1 in a first embodiment. Information processing system 1 is a system for moving a mobile object independently of a passenger's driving operation. Information processing system 1 includes one or more mobile objects, one or more external cameras 9 installed in a location separate from the mobile objects, and an information processing device 5.

[0009] "Mobile body" means an object that can move, such as a vehicle 10 or an electric vertical take-off and landing aircraft (a so-called flying car). In the present embodiment, the mobile body is a vehicle 10. The vehicle 10 may be a vehicle 10 that travels on wheels or a vehicle 10 that travels on an infinite track, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, a construction vehicle, etc. The vehicle 10 includes electric vehicles (BEV: Battery Electric Vehicle), gasoline vehicles, hybrid vehicles, and fuel cell vehicles. In the case where the mobile body is other than the vehicle 10, the expressions "vehicle" and "car" in the present disclosure can be appropriately replaced with "mobile body", and the expression "travel" can be appropriately replaced with "movement".

[0010] The vehicle 10 includes a drive device 11, a steering device 12, a braking device 13, a communication device 14, and a vehicle control device 15. The drive device 11 accelerates the vehicle 10. The drive device 11 includes wheels, a driving force source for rotating the wheels, and a power supply device for supplying power to the driving force source. For example, when the vehicle 10 is an electric vehicle, the driving force source is an electric motor for driving, and the power supply device is a battery such as a lithium-ion battery. The steering device 12 changes the direction of travel of the vehicle 10. The braking device 13 decelerates the vehicle 10. The communication device 14 communicates with external devices using wireless communication, etc. The external devices are devices such as the information processing device 5 and other vehicles 10 that are located in a location different from the vehicle 10. The communication device 14 is, for example, a wireless communication device.

[0011] The vehicle control device 15 controls the operation of the vehicle 10. The vehicle control device 15 includes a CPU 150, a storage unit 157, and an input / output interface 159. In the vehicle control device 15, the CPU 150, the storage unit 157, and the input / output interface 159 are connected via a bus or the like so as to enable bidirectional communication. The input / output interface 159 is used to communicate with various devices mounted on the vehicle 10. The storage unit 157 of the vehicle control device 15 stores various information, including various programs P15 that control the operation of the vehicle control device 15. The storage unit 157 of the vehicle control device 15 includes, for example, RAM, ROM, and a hard disk drive (HDD). The CPU 150 of the vehicle control device 15 functions as the operation control unit 153 by expanding the various programs P15 stored in the storage unit 157. The operation control unit 153 controls the driving of the vehicle 10 by controlling the operation of actuators that change the acceleration / deceleration and steering angle of the vehicle 10. "Driving control" refers to various controls used to drive the actuators that perform the three functions of "driving," "turning," and "stopping" of the vehicle 10, such as adjusting the acceleration, speed, and steering angle of the vehicle 10. In this embodiment, the actuators include the actuators of the drive device 11, the actuators of the steering device 12, and the actuators of the braking device 13. It should be noted that at least some of the functions of the CPU 150 of the vehicle control device 15 may also be implemented by hardware circuits.

[0012] The vehicle 10 is configured to be able to travel by unmanned driving. "Unmanned driving" means driving that does not rely on the driving operation of the passengers. Driving operation means an operation related to at least any one of "driving", "turning" and "stopping" of the vehicle 10. Unmanned driving is achieved by automatic or manual remote control using a device located outside the vehicle 10 or by autonomous control of the vehicle 10. In the vehicle 10 that travels by unmanned driving, there may also be passengers who do not perform driving operations. Passengers who do not perform driving operations include, for example, people who simply sit in the seats of the vehicle 10, and people who perform operations different from driving operations such as assembly, inspection, and switch operations while riding in the vehicle 10. It should be noted that driving that relies on the driving operations of passengers is sometimes referred to as "manned driving".

[0013] In this specification, "remote control" includes "full remote control" in which all operations of vehicle 10 are completely determined from outside vehicle 10, and "partial remote control" in which part of the operations of vehicle 10 are determined from outside vehicle 10. Furthermore, "autonomous control" includes "full autonomous control" in which vehicle 10 autonomously controls its own operations without receiving any information from devices outside vehicle 10, and "partial autonomous control" in which vehicle 10 autonomously controls its own operations using information received from devices outside vehicle 10.

[0014] In this embodiment, the information processing system 1 is used in a factory that manufactures a vehicle 10. The reference coordinate system of the factory is a global coordinate system. That is, any position in the factory is represented by the coordinates of X, Y, and Z in the global coordinate system. The factory has a first place and a second place. The first place and the second place are connected by a passage Ro on which the vehicle 10 can travel. In the factory, a plurality of external cameras 9 are arranged along the passage Ro. The external camera 9 captures a shooting area RG including a target area and the vehicle 10 from the outside of the vehicle 10. The target area is an area including the passage Ro on which the vehicle 10 travels and the surrounding area of ​​the passage Ro. The position of each external camera 9 in the factory is adjusted in advance. The vehicle 10 moves from the first place to the second place through the passage Ro by unmanned driving. It should be noted that in other embodiments, the vehicle 10 can also travel by unmanned driving in a place different from the factory.

[0015] The information processing device 5 is used to drive the vehicle 10 unmanned. The information processing device 5 is, for example, a server located in a different location from the vehicle 10. The information processing device 5 learns the machine learning model Md, or obtains and uses the learned machine learning model Md to drive the vehicle 10 unmanned. The machine learning model Md, when fed with a captured image or the like, outputs at least one of a driving control signal and signal generation parameters. The captured image is acquired by capturing the target area and the vehicle 10 using the external camera 9. The captured image can be either an original image, which is raw data output from the external camera 9, or a processed image obtained by processing the original image. The driving control signal is a control signal that specifies the behavior of the vehicle 10 for driving the vehicle 10 unmanned. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 10 as control parameters. In other embodiments, the driving control signal may include the speed of the vehicle 10 as a control parameter instead of, or in addition to, the acceleration of the vehicle 10. The signal generation parameters are parameters used when generating the driving control signal. Signal generation parameters include, for example, vehicle position information, coordinate values ​​for calculating at least one of the position and orientation of vehicle 10, and shape data for calculating at least one of the position and orientation of vehicle 10. Shape data represents the external shape of vehicle 10. Shape data is generated by detecting vehicle 10 from a captured image. Shape data is, for example, a mask image obtained by masking the region representing vehicle 10 within each region constituting the captured image, thereby adding the mask region to the captured image. Each region constituting the captured image is, for example, a single pixel constituting the captured image.

[0016] In this embodiment, a first learning model Md1 is used as a machine learning model Md. The first learning model Md1 outputs shape data as a parameter for signal generation. Thus, the information processing device 5 obtains shape data by inputting captured images into the first learning model Md1. Furthermore, the information processing device 5 uses the shape data to calculate the position and orientation of the vehicle 10. Furthermore, the information processing device 5 uses vehicle position information and other information to generate a travel control signal and transmits it to the vehicle 10. Thus, the information processing device 5 causes the vehicle 10 to travel under remote control.

[0017] Here, the appearance of the target area may change over time, for example, due to dirt buildup on the road Ro on which the vehicle 10 travels. Therefore, the target area is classified into multiple appearance states based on changes in appearance. If the appearance of the target area has changed, and a captured image containing the vehicle 10 and the target area after the change in appearance is input to the first learning model Md1, which was learned using training images containing the vehicle 10 and the target area before the change in appearance, the detection accuracy of the vehicle 10 from the captured image may decrease. Therefore, if it is determined that the appearance of the target area has changed, the information processing device 5 relearns the first learning model Md1 and uses the relearned first learning model Md1 to obtain shape data.

[0018] The following description assumes that the factor affecting the change in the appearance state of the target area is dirt generated in the target area. For example, at the time point when the vehicle 10 starts driving based on unmanned driving, the state before the change in the appearance state of the target area is a state in which no dirt has been generated in the target area. The state after the change in the appearance state of the target area is a state in which dirt has been generated in the target area. In this case, during the period when the vehicle 10 performs unmanned driving control, it is determined that dirt has been generated in the target area. At the time point after the relearned first learning model Md1 is stored in the storage unit 53 of the information processing device 5, the state before the change and the state after the change are as follows. In this case, the state before the change is a state in which dirt has been generated in the target area. The state after the change is a state in which no dirt has been generated in the target area due to the dirt in the target area being removed by sweeping, etc.

[0019] The information processing device 5 includes a communication unit 51, a CPU 52, and a storage unit 53. In the information processing device 5, the communication unit 51, the CPU 52, and the storage unit 53 are connected to each other via a bus or the like so as to be capable of bidirectional communication.

[0020] The communication unit 51 of the information processing device 5 connects other devices other than the information processing device 5 to the information processing device 5 in a communicable manner. The communication unit 51 of the information processing device 5 is, for example, a wireless communication device.

[0021] The storage unit 53 of the information processing device 5 stores various information including various programs P5 for controlling the operation of the information processing device 5, a reference path Ip, and a first learning model Md1. The storage unit 53 of the information processing device 5 includes, for example, RAM, ROM, and a hard disk drive (HDD).

[0022] The first learning model Md1 is a machine learning model Md that has been learned using artificial intelligence by detecting the appearance shape of the vehicle 10 from the captured image and outputting shape data. The first learning model Md1 is pre-learned by supervised learning using an initial learning data set. The initial learning data set includes a plurality of training data in which a training image including the vehicle 10 and the object area before the appearance state changes and a correct answer label corresponding to the output of the machine learning model Md are established. In this embodiment, the correct answer label is a region correct answer label. The region correct answer label is a correct answer label indicating whether each region in the training image is a region representing the vehicle 10 or a region representing a region other than the vehicle 10. For the first learning model Md1, for example, a convolutional neural network (hereinafter referred to as CNN) that implements either semantic segmentation or instance segmentation can be used. When learning the CNN, for example, by backpropagation (error backpropagation method), the parameters of the CNN are updated in a manner that reduces the error between the output result of the first learning model Md1 and the region correct answer label.

[0023] The CPU 52 of the information processing device 5 expands various programs P5 stored in the storage unit 53 to function as a change information acquisition unit 521 , a learning unit 522 , a state information acquisition unit 523 , a control information acquisition unit 524 , a calculation unit 525 , and a remote control unit 526 .

[0024] The change information acquisition unit 521 acquires change information. The change information is information related to a change in the appearance state of the object area. Specifically, the change information is information used to determine whether the appearance state of the object area has changed from the appearance state of the object area in the training image used when learning the first learning model Md1. In the case where the factor that affects the change in the appearance state of the object area is dirt generated in the object area, the change information is, for example, the following information. In this case, the change information is information used to determine whether the appearance state of the object area has changed due to the generation of dirt in the object area that did not exist when learning the first learning model Md1 stored in the storage unit 53 of the information processing device 5. In this embodiment, the change information acquisition unit 521 acquires a captured image as the change information.

[0025] If the change information indicates a change in the appearance of the target area, the learning unit 522 causes the first learning model Md1 to perform relearning as learning. Relearning is an example of additional learning. In this embodiment, the learning unit 522 causes the first learning model Md1, which was learned using the initial learning dataset, to relearn using the relearning dataset. In this embodiment, the relearning dataset includes a plurality of training data sets in which training images including the vehicle 10 and the target area after the change in appearance are associated with correct region labels.

[0026] For example, when the vehicle 10 begins unmanned driving and no dirt is present in the target area, the first learning model Md1 stored in the storage unit 53 of the information processing device 5 is pre-learned through supervised learning using the following initial learning dataset. In this case, the initial learning dataset includes a plurality of training data sets that associate training images including the vehicle 10 and the target area without dirt with correct region labels. If the learning unit 522 determines that the appearance of the target area has changed during unmanned driving control, that is, that dirt has occurred in the target area, it performs the following processing. In this case, the learning unit 522 relearns the first learning model Md1 learned using the initial learning dataset using the following relearning dataset during unmanned driving control. In this case, the relearning dataset includes a plurality of training data sets that associate training images including the vehicle 10 and the target area with dirt and correct region labels.

[0027] In this embodiment, the learning unit 522 uses change information to determine whether the appearance of the target area has changed since the first learning model Md1 was learned. Specifically, the learning unit 522 uses the captured image as change information to determine whether the appearance of the target area has changed since the first learning model Md1 was learned. For example, the learning unit 522 determines whether the appearance of the target area has changed since the first learning model Md1 was learned by detecting factors that affect the change in the appearance of the target area from the captured image. Specifically, if factors that affect the change in the appearance of the target area are detected from the captured image, the learning unit 522 determines that the appearance of the target area has changed since the first learning model Md1 was learned. On the other hand, if factors that affect the change in the appearance of the target area are not detected from the captured image, the learning unit 522 determines that the appearance of the target area has not changed since the first learning model Md1 was learned. More specifically, if the state before the change is a state where no dirt has occurred in the target area and the state after the change is a state where dirt has occurred in the target area, the learning unit 522 makes the following determination. In this case, if no dirt is detected from the captured image, the learning unit 522 determines that no dirt has occurred in the target area and the appearance of the target area has not changed. On the other hand, if dirt is detected from the captured image, the learning unit 522 determines that dirt has occurred in the target area and the appearance of the target area has changed.

[0028] It should be noted that the learning unit 522 may also use the pixel values ​​of the captured image to determine whether the appearance of the target area has changed since the first learning model Md1 was learned. For example, if the difference between the total luminance values ​​of each pixel in the original image and a predetermined luminance value is greater than a predetermined threshold, the learning unit 522 determines that the appearance of the target area has changed since the first learning model Md1 was learned. On the other hand, if the difference between the total luminance values ​​of each pixel in the original image and the predetermined luminance value is less than the threshold, the learning unit 522 determines that the appearance of the target area has not changed since the first learning model Md1 was learned.

[0029] The state information acquisition unit 523 acquires state information. State information is information related to the appearance of the target area. In this embodiment, the state information is used to determine whether the appearance of the target area is a state where dirt is not present or a state where dirt is present. In this embodiment, the state information acquisition unit 523 acquires a captured image as state information.

[0030] The control information acquisition unit 524 uses the state information to determine the appearance of the target area and inputs the captured image to the first learning model Md1 corresponding to the appearance of the target area, thereby acquiring at least one of a driving control signal and a signal generation parameter. If the appearance of the target area determined by the state information differs from the appearance of the target area in the training image used when learning the first learning model Md1, the control information acquisition unit 524 performs the following processing. In this case, the control information acquisition unit 524 inputs the captured image to the relearned first learning model Md1, which has been relearned using a plurality of training data sets in which the training image including the vehicle 10 and the target area whose appearance is determined by the state information is associated with the correct region label. On the other hand, if the appearance of the target area determined by the state information is the same as the appearance of the target area in the training image stored in the storage unit 53 and used when learning the first learning model Md1, the control information acquisition unit 524 performs the following processing. In this case, the control information acquisition unit 524 inputs the captured image to the first learning model Md1 that has not been relearned using a plurality of training data in which training images and correct region labels are associated with the target region including the vehicle 10 and the appearance state specified by the passing state information.

[0031] The calculation unit 525 obtains vehicle position information by calculating the position and orientation of the vehicle 10. The vehicle position information is position information that forms the basis for generating the travel control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 10 in the factory's reference coordinate system.

[0032] The position of the vehicle 10 is, for example, the position of a measuring point 10e preset for a specific portion of the vehicle 10. In this embodiment, the calculation unit 525 uses shape data obtained by inputting a captured image into the first learning model Md1 to calculate the coordinates of the measuring point 10e of the vehicle 10 in the coordinate system of the captured image, that is, the local coordinate system. The position of the vehicle 10 is calculated by converting the calculated coordinates into coordinates in the global coordinate system.

[0033] The orientation of the vehicle 10 is, for example, a direction represented by a vector extending from the rear side to the front side of the vehicle 10 along the front-rear axis passing through the center of gravity of the vehicle 10. In the present embodiment, the calculation unit 525 calculates the orientation of the vehicle 10 based on the orientation of the movement vector of the vehicle 10 calculated from the positional changes of the feature points of the vehicle 10 between frames of the captured image, using, for example, an optical flow method.

[0034] The remote control unit 526 remotely controls the driving of the vehicle 10 by generating and transmitting a driving control signal to the vehicle 10. To generate the driving control signal, the remote control unit 526 first determines the target location to which the vehicle 10 should next proceed. In this embodiment, the target location is represented by X, Y, and Z coordinates in the factory's global coordinate system. As previously mentioned, the storage unit 53 of the information processing device 5 pre-stores a reference path Ip, the route that the vehicle 10 should travel. The remote control unit 526 uses the vehicle's position information and the reference path Ip to determine the target location to which the vehicle 10 should next proceed. The reference path Ip is the route that the vehicle 10 should travel. A route is represented by a node representing a departure point, nodes representing passing points, a node representing a destination, and links connecting these nodes. The remote control unit 526 determines the target location on the reference path Ip that is ahead of the vehicle 10's current location. The remote control unit 526 generates a driving control signal to cause the vehicle 10 to travel toward the determined target location. As previously mentioned, in this embodiment, the driving control signal includes the acceleration and steering angle of vehicle 10 as parameters. The remote control unit 526 calculates the driving speed of vehicle 10 based on the change in the position of vehicle 10 and compares the calculated driving speed with a predetermined target speed for vehicle 10. Generally speaking, the remote control unit 526 determines the acceleration to accelerate vehicle 10 when the driving speed is lower than the target speed, and to decelerate vehicle 10 when the driving speed is higher than the target speed. Furthermore, when vehicle 10 is on reference path Ip, the remote control unit 526 determines the steering angle and acceleration to prevent vehicle 10 from deviating from the reference path Ip. When vehicle 10 is not on reference path Ip, in other words, if vehicle 10 has deviated from the reference path Ip, the remote control unit 526 determines the steering angle and acceleration to return vehicle 10 to the reference path Ip. In this way, the remote control unit 526 generates a driving control signal that includes the acceleration and steering angle of vehicle 10 as parameters. Then, the remote control unit 526 transmits the generated travel control signal to the vehicle 10. It should be noted that at least a part of the functions of the CPU 52 of the information processing device 5 may be realized by a hardware circuit.

[0035] A-2. Vehicle motion control method: Figure 2 This is a flowchart showing a method of causing the vehicle 10 to travel in the information processing system 1 according to the first embodiment. Figure 2 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0036] The calculation unit 525 of the information processing device 5 obtains vehicle position information using the detection results output by the external camera 9 located outside the vehicle 10 (step S110). The remote control unit 526 determines the target location to which the vehicle 10 should next proceed (step S120). The remote control unit 526 uses the vehicle position information and the reference path Ip to determine the target location to which the vehicle 10 should next proceed. The remote control unit 526 determines the target location on the reference path Ip that is ahead of the current location of the vehicle 10.

[0037] The remote control unit 526 generates a driving control signal for driving the vehicle 10 toward the determined target position (step S130). The remote control unit 526 calculates the driving speed of the vehicle 10 based on the change in the position of the vehicle 10, and compares the calculated driving speed with the predetermined target speed of the vehicle 10. When the driving speed is lower than the target speed, the remote control unit 526 determines the acceleration so as to accelerate the vehicle 10. When the driving speed is higher than the target speed, the remote control unit 526 determines the acceleration so as to decelerate the vehicle 10. When the vehicle 10 is on the reference path Ip, the remote control unit 526 determines the steering angle so as to prevent the vehicle 10 from deviating from the reference path Ip. When the vehicle 10 is not on the reference path Ip, in other words, when the vehicle 10 deviates from the reference path Ip, the remote control unit 526 determines the steering angle so as to return the vehicle 10 to the reference path Ip.

[0038] The remote control unit 526 transmits the generated travel control signal to the vehicle 10 (step S140). The information processing device 5 repeatedly acquires vehicle position information, determines a target position, generates and transmits a travel control signal in a predetermined cycle.

[0039] The vehicle control device 15 mounted on the vehicle 10 receives a travel control signal from the information processing device 5 (step S150). The motion control unit 153 of the vehicle control device 15 uses the received travel control signal to control the actuators, thereby causing the vehicle 10 to travel at the acceleration and steering angle indicated by the travel control signal (step S160). The vehicle control device 15 repeatedly receives the travel control signal and controls the actuators at a predetermined cycle. According to the information processing system 1 of this embodiment, since the vehicle 10 can be driven by remote control, the vehicle 10 can be moved without using transportation equipment such as cranes and conveyors.

[0040] A-3. Information Processing Methods: Figure 3 This is a flowchart illustrating a method for relearning the first learning model Md1 in the first embodiment. Figure 3 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0041] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring a captured image as change information to the external camera 9 (step S211). Upon receiving the request signal, the external camera 9 transmits the captured image to the information processing device 5 (step S212).

[0042] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522 uses the captured image to determine whether the appearance of the target area has changed since the first learning model Md1 stored in the storage unit 53 was learned. If it is determined that the appearance of the target area has changed since the first learning model Md1 stored in the storage unit 53 was learned (step S213: Yes), the learning unit 522 relearns the first learning model Md1 (step S214). On the other hand, if the appearance of the target area has not changed since the first learning model Md1 stored in the storage unit 53 was learned (step S213: No), the learning unit 522 does not relearn the first learning model Md1, and the process ends.

[0043] Figure 4 This is a flowchart showing a method for acquiring shape data in the first embodiment. Figure 4 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0044] In the state information acquisition process, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring a captured image as state information (step S311). Upon receiving the request signal, the external camera 9 transmits the captured image to the information processing device 5 (step S312). The state information acquisition unit 523 analyzes the captured image to determine the appearance state of the target area (step S313).

[0045] Following the state information acquisition step, the control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524 acquires shape data by inputting a captured image into the first learning model Md1 that was learned using a training image containing the vehicle 10 and the object area whose appearance state is determined by the state information. Specifically, if the appearance state determined by the state information differs from the appearance state of the object area in the training image used when learning the first learning model Md1 stored in the storage unit 53 (step S314: Yes), the control information acquisition unit 524 performs the following processing. In this case, the control information acquisition unit 524 inputs a captured image into the first learning model Md1 that has been relearned using a plurality of training data sets that associate the training image containing the vehicle 10 and the object area whose appearance state is determined by the state information with the correct region labels. Thus, the control information acquisition unit 524 acquires shape data (step S315). On the other hand, if the appearance state determined by the state information is identical to the appearance state of the target area in the training image used when learning the first learning model Md1 stored in the storage unit 53 (step S314: No), the control information acquisition unit 524 performs the following processing. In this case, the control information acquisition unit 524 acquires shape data by inputting the captured image to the first learning model Md1 that has not been relearned using a plurality of training data that associates the training image containing the vehicle 10 and the target area whose appearance state is determined by the state information with the correct region labels (step S316).

[0046] A-4. Vehicle position calculation method: Figure 5 This is a flowchart showing an example of a method for generating shape data and a method for calculating the position of the vehicle 10 using the shape data. Figure 6 It shows that the Figure 5 Schematic diagram of examples of various images Im1 to Im6 when using the method shown. Figure 6 In the Figure 5 In this embodiment, the measuring point 10e of the vehicle 10 is the rear end of the left side of the vehicle 10. It should be noted that the measuring point 10e of the vehicle 10 may be other than the rear end of the left side of the vehicle 10.

[0047] In the image acquisition step (step S410), the control information acquisition unit 524 acquires the original image Im1 output from the external camera 9. In the distortion correction step (step S420), the control information acquisition unit 524 corrects the distortion of the original image Im1 to generate a corrected image Im2, which is a processed image. In the rotation processing step (step S430), the control information acquisition unit 524 rotates the corrected image Im2. Thus, the control information acquisition unit 524 generates a rotated image Im3, which is a processed image. In the cropping processing step (step S440), the control information acquisition unit 524 deletes the unnecessary area A2, excluding the necessary area A1 consisting of the vehicle 10 and its surrounding area, from the rotated image Im3, from the various areas that constitute the rotated image Im3. Thus, the control information acquisition unit 524 generates a processed image Im4, which is a processed image. Each of steps S420 to S440 is a pre-processing step for improving the detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4. Therefore, at least any one of the steps S420 to S440 may be omitted.

[0048] In the detection process (step S450), the control information acquisition unit 524 inputs the processed image Im4 into the first learning model Md1. The control information acquisition unit 524 thereby detects the vehicle 10 from the processed image Im4 and acquires the first mask image Im5 as shape data Da. The first mask image Im5 is an image obtained by masking the region representing the vehicle 10 among the regions constituting the processed image Im4, thereby adding a mask region Ms to the processed image Im4.

[0049] In the perspective transformation step (step S460), the calculation unit 525 generates a second mask image Im6 by performing a perspective transformation on the first mask image Im5. The calculation unit 525 uses, for example, predetermined perspective transformation parameters to transform the first mask image Im5 into a bird's-eye view image as viewed from above the vehicle 10, approximately perpendicular to the road surface Rs. The perspective transformation parameters are, for example, parameters related to the positional information and internal parameters of the external camera 9 obtained through calibration. Thus, the calculation unit 525 generates the second mask image Im6, represented in the image coordinate system, based on the first mask image Im5 represented in the camera coordinate system. The camera coordinate system is a coordinate system with the focal point of the external camera 9 as its origin and axes represented by the Xc axis and the Yc axis, which is orthogonal to the Xc axis. The image coordinate system is a coordinate system with a point on the image plane as its origin and axes represented by the Xi axis and the Yi axis, which is orthogonal to the Xi axis.

[0050] In the coordinate point calculation step (step S470 ), the calculation unit 525 calculates an image coordinate point indicating the position of the vehicle 10 in the image coordinate system. Figure 71 is a diagram for explaining the details of the coordinate point calculation process. To calculate the image coordinate point, the calculation unit 525 first calculates the base coordinate point P0 based on the first circumscribed rectangle R1 set for the mask region Ms in the first mask image Im5, which is the image before perspective transformation. Figure 8 This figure illustrates the method for calculating the base coordinate point P0. To calculate the base coordinate point P0, the calculation unit 525 sets a base circumscribed rectangle R0 for the mask area Ms in the first mask image Im5. Next, the calculation unit 525 rotates the first mask image Im5 by a required amount, with the center of gravity C of the mask area Ms as the rotation center, so that the direction of the motion vector V of the vehicle 10 corresponding to the mask area Ms in the first mask image Im5 is oriented in a predetermined direction. For example, the predetermined direction is the upward direction on the screen of the display device displaying the first mask image Im5. Next, the calculation unit 525 sets a first circumscribed rectangle R1 for the rotated mask area Ms of the first mask image Im5 so that its long side is parallel to the motion vector V. Next, the calculation unit 525 reversely rotates the first mask image Im5, to which the first circumscribed rectangle R1 is attached, by the aforementioned rotation amount, with the center of gravity C of the mask area Ms as the rotation center. Thus, the calculation unit 525 uses the coordinate point of one of the four vertices of the first circumscribed rectangle R1 that has the coordinates closest to the position measurement point 10 e of the vehicle 10 as the base coordinate point P0 .

[0051] Then, if Figure 7 As shown, the calculation unit 525 performs perspective transformation on the inversely rotated first mask image Im5, i.e., the first mask image Im5 after calculating the base coordinate point P0. As a result, the calculation unit 525 uses the coordinate point corresponding to the base coordinate point P0 as the first coordinate point P1 in the first circumscribed rectangle R1 deformed by the perspective transformation.

[0052] Next, the calculation unit 525 sets a second circumscribed rectangle R2 for the mask region Ms in the second mask image Im6, which is the result of perspective transformation of the first mask image Im5. The calculation unit 525 then sets the vertex of the second circumscribed rectangle R2 that represents the same position as the first coordinate point P1 as the second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 represent the same position and are therefore correlated with each other.

[0053] Next, the calculation unit 525 corrects the coordinates (Xi1, Yi1) of the first coordinate point P1 by replacing them with the coordinates (Xi2, Yi2) of the second coordinate point P2 based on the size relationship between the coordinate values ​​of the first coordinate point P1 and the coordinate values ​​of the second coordinate point P2. When the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is larger than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the calculation unit 525 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 in the Yi direction of the first coordinate point P1 is larger than the coordinate value Yi2 in the Yi direction of the second coordinate point P2 (Yi1>Yi2), the calculation unit 525 replaces the coordinate value Yi1 in the Yi direction of the first coordinate point P1 with the coordinate value Yi2 in the Yi direction of the second coordinate point P2. Figure 7 In the example shown, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Furthermore, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Consequently, image coordinate point P3 has coordinates (Xi2, Yi1). Thus, the calculation unit 525 calculates image coordinate point P3, which represents the position of vehicle 10 in the image coordinate system, by correcting first coordinate point P1 using second coordinate point P2.

[0054] In the position conversion process (step S480), the calculation unit 525 calculates the vehicle coordinate point representing the position of the measuring position 10e of the vehicle 10 in the global coordinate system by converting the image coordinate point P3 into the vehicle coordinate point. The calculation unit 525 converts the image coordinate point P3 into the vehicle coordinate point using the relationship equations (1) to (3) described later, which include the vehicle coordinate point as the target variable and include the image coordinate point P3, the shooting parameter, and the vehicle parameter as the explanatory variables. The shooting parameter is a parameter related to the distance of the external camera 9 from the predetermined reference point. In this embodiment, the shooting parameter is the height H (described later) of the external camera 9 from the road surface Rs. Figure 9 The vehicle parameter is a parameter related to the distance between the measuring point 10e of the vehicle 10 and the reference point. In this embodiment, the vehicle parameter is the height h (described later) of the measuring point 10e of the vehicle 10 from the road surface Rs. Figure 9 ).

[0055] Figure 9 This is the first figure for explaining the details of the position conversion process. Figure 9 , the vehicle 10 is shown as viewed from the left side. Figure 10 This is a second diagram for explaining the details of the position conversion process. Figure 10, the vehicle 10 is shown as viewed from the roof side. Figure 9 and Figure 10 The global coordinate system shown is a coordinate system with a fixed coordinate point Pf, representing an arbitrary reference position on the road surface Rs, as its origin, and having coordinate axes represented by the Xg axis and the Yg axis, which is orthogonal to the Xg axis. The captured coordinate point Pc represents the position of the external camera 9 that outputs the original image Im1 used to calculate the image coordinate point P3 in the global coordinate system. The fixed coordinate point Pf and the captured coordinate point Pc are pre-stored in, for example, the storage unit 53 of the information processing device 5.

[0056] like Figure 9 As shown, let the observation distance between the position of the external camera 9 and the position of the vehicle 10 (image coordinate point P3) on the XgYg plane be Do. Let the observation error be ΔD. Let the height [m] of the external camera 9 from the road surface Rs, as an imaging parameter, be H. Let the height [m] of the measurement point 10e of the vehicle 10 from the road surface Rs, as a vehicle parameter, be h. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Formula (1) That is, the larger the observation distance Do is, the larger the observation error ΔD is.

[0057] When the actual distance between the position of the external camera 9 and the position of the measuring point 10 e of the vehicle 10 is defined as a first distance D, the first distance D can be expressed by the following equation (2). D=Do×(1-h / H) Formula (2) That is, the first distance D is determined by the observation distance Do, the height H of the external camera 9 as an imaging parameter, and the height h of the measurement point 10 e of the vehicle 10 as a vehicle parameter.

[0058] like Figure 10 As shown, when the distance between the reference position and the estimated position of the vehicle 10 is defined as the estimated distance Dp and the actual distance between the reference position and the vehicle 10 is defined as the second distance Dt, the second distance Dt can be expressed by the following equation (3). Dt=Dp×(1-h / H) Formula (3)

[0059] Here, the estimated distance Dp can be calculated using the third distance Dc, which is the actual distance calculated from the fixed coordinate point Pf and the captured coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the calculation unit 525 calculates the vehicle coordinate point Pv using the second distance Dt calculated by correcting the estimated distance Dp using the above-mentioned equation (3) and the fixed coordinate point Pf. The calculated vehicle coordinate point Pv represents the position of the measuring point 10e of the vehicle 10 in the global coordinate system and therefore corresponds to the position of the vehicle 10 in real space.

[0060] According to the first embodiment described above, the information processing device 5 is able to obtain change information. Furthermore, when the information processing device 5 determines that the appearance state of the object area Ta has changed using the change information, it is able to perform the following processing. In this case, the information processing device 5 can cause the first learning model Md1, which has been learned using a plurality of training data corresponding to training images and regional correct labels including the vehicle 10 and the object area Ta before the appearance state changes, to be relearned using a plurality of training data corresponding to training images and regional correct labels including the vehicle 10 and the object area Ta after the appearance state changes. In this way, when the appearance state of the object area Ta has changed, it is possible to suppress a decrease in detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0061] Furthermore, according to the first embodiment, the information processing device 5 can obtain, as change information, captured images Im1 to Im4 that include the target area Ta and the vehicle 10. Furthermore, by detecting dirt from the captured images Im1 to Im4 using image analysis, the information processing device 5 can determine whether the appearance of the target area Ta has changed since the first learning model Md1 was learned.

[0062] Furthermore, according to the first embodiment, the information processing device 5 can relearn the first learning model Md1 while executing unmanned control. It should be noted that in other embodiments, the information processing device 5 may not execute unmanned control, for example, but may learn the first learning model Md1 while executing manned control.

[0063] Furthermore, according to the first embodiment, the vehicle 10 is driven unmanned in the factory. In this way, the information processing device 5 can relearn the first learning model Md1 during the manufacturing process of the vehicle 10 .

[0064] Furthermore, according to the first embodiment described above, the information processing device 5 can obtain state information. Furthermore, when the appearance state determined by the state information differs from the appearance state of the object area Ta in the training image used when learning the first learning model Md1, the information processing device 5 can obtain shape data Da by performing the following processing. In this case, the information processing device 5 can obtain shape data Da by inputting the captured images Im1 to Im4 into the first learning model Md1 that has been relearned using a plurality of training data sets that associate training images containing the vehicle 10 and the object area Ta whose appearance state is determined by the state information with correct area labels. This can suppress a decrease in the accuracy of detecting the vehicle 10 from the captured images Im1 to Im4 when the appearance state of the object area Ta differs from that when the first learning model Md1 was learned.

[0065] Furthermore, according to the first embodiment, the information processing device 5 can acquire, as state information, captured images Im1 to Im4 including the target area Ta and the vehicle 10. Furthermore, the information processing device 5 can determine the appearance state of the target area Ta by detecting dirt from the captured images Im1 to Im4 using image analysis.

[0066] It should be noted that the information processing device 5 can relearn the first learning model Md1 each time the appearance of the target area Ta changes, and can store the relearned first learning model Md1 in the storage unit 53 as a new first learning model Md1 that is different from the first learning model Md1 before relearning. When storing the relearned first learning model Md1 in the storage unit 53 as a new first learning model Md1 that is different from the first learning model Md1 before relearning, it is preferable to do so as follows. In this case, it is preferable to associate the relearned first learning model Md1 with supplementary information indicating the appearance of the target area Ta included in the training image used for relearning. With this approach, the information processing device 5 can obtain at least one of the driving control signal and the signal generation parameters using the learned first learning model Md1 corresponding to the appearance of the target area Ta, without relearning each time the appearance of the target area Ta changes. Furthermore, the information processing device 5 can also obtain both the driving control signal and the signal generation parameters by inputting the captured images Im1 to Im4 into the first learning model Md1.

[0067] B. Second embodiment: Figure 11This figure shows the structure of an information processing system 1a in a second embodiment. The information processing system 1a includes one or more vehicles 10, an external camera 9, and an information processing device 5a. In this embodiment, the types of change information and status information differ from those in the first embodiment. As in the first embodiment, the factor that affects the change in the appearance state of the target area Ta is dirt generated in the target area Ta. Unless otherwise specified, the other structures of the information processing system 1a are the same as those in the first embodiment. Structures identical to those in the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.

[0068] The information processing device 5a includes a communication unit 51, a CPU 52a, and a storage unit 53a. The storage unit 53a of the information processing device 5a stores various information, including various programs P5a that control the operation of the information processing device 5a, a reference path Ip, a first learning model Md1, and cleaning information Ic. The cleaning information Ic indicates, for each target area Ta, the cleaning status of the passage Ro and the surrounding area of ​​the passage Ro, which may have accumulated. Specifically, the cleaning information Ic indicates the timing of cleaning and the imaging area RG of each external camera 9. The cleaning information Ic is updated each time cleaning is performed. Cleaning of dirt is performed at a predetermined interval.

[0069] The CPU 52a of the information processing device 5a functions as a change information acquisition unit 521a, a learning unit 522a, a state information acquisition unit 523a, a control information acquisition unit 524, a calculation unit 525, and a remote control unit 526 by developing various programs P5a stored in the storage unit 53a.

[0070] The change information acquisition unit 521a acquires the cleaning information Ic as the change information. The learning unit 522a uses the cleaning information Ic to determine whether the appearance state of the object area Ta has changed. For example, when the state before the change is a state in which no dirt has been generated in the object area Ta and the state after the change is a state in which dirt has been generated in the object area Ta, the learning unit 522a determines as follows. In this case, if the learning unit 522a has cleaned the object area Ta within a predetermined period, it determines that no dirt has been generated in the object area Ta and the appearance state of the object area Ta has not changed. On the other hand, if the learning unit 522a has not cleaned the object area Ta within a predetermined period, it determines that dirt has been generated in the object area Ta and the appearance state of the object area Ta has changed.

[0071] The state information acquisition unit 523a generates state information using the cleaning information Ic, thereby acquiring state information. For example, if the state before the change was that the target area Ta was free of dirt and the state after the change was that the target area Ta was contaminated, the state information acquisition unit 523a generates state information as follows. In this case, if the state information acquisition unit 523a has cleaned the target area Ta within a predetermined period, the state information acquisition unit 523a generates state information indicating that the target area Ta is free of dirt. On the other hand, if the state information acquisition unit 523a has not cleaned the target area Ta within a predetermined period, the state information acquisition unit 523a generates state information indicating that the target area Ta is contaminated.

[0072] According to the second embodiment, the information processing device 5a can obtain the sweep information Ic as change information. Furthermore, the information processing device 5a uses the sweep information Ic to determine whether the target area Ta has been swept within a predetermined period, thereby determining whether the appearance of the target area Ta has changed since the first learning model Md1 was learned.

[0073] Furthermore, according to the second embodiment, the information processing device 5 a can acquire the status information by using the sweep information Ic to determine whether the target area Ta has been swept within a predetermined period.

[0074] C. Third embodiment: Figure 12 : This is a diagram showing the structure of the information processing system 1b in the third embodiment. The information processing system 1b includes one or more vehicles 10, an external camera 9, and an information processing device 5b. In this embodiment, when the information processing device 5b determines that the appearance state of the object area Ta has changed, it notifies the administrator of the learning request information for relearning the first learning model Md1. Then, the administrator who has recognized the learning request information instructs the first learning model Md1 to relearn via the input operation unit 55, and the first learning model Md1 is relearned. Unless otherwise specified, the other structures of the information processing system 1b are the same as those of the first embodiment. The same figure marks are marked on the same structures as the first embodiment, and the description is omitted.

[0075] The information processing device 5b includes a communication unit 51, a CPU 52b, a storage unit 53b, a notification unit 54, and an input operation unit 55. The storage unit 53b of the information processing device 5b stores various information, including various programs P5b that control the operation of the information processing device 5b, a reference path Ip, and the first learning model Md1. The notification unit 54 notifies the administrator of the learning request information. The notification unit 54 is, for example, a display that displays the learning request information as text. Alternatively, the notification unit 54 may be a speaker that plays the learning request information as audio information. The input operation unit 55 is an input device for the administrator, who has recognized the learning request information, to instruct the relearning of the first learning model Md1. The input operation unit 55 is, for example, a touch panel, a mouse, or a keyboard that receives input from the administrator.

[0076] The CPU 52b of the information processing device 5b functions as a change information acquisition unit 521, a notification control unit 527, and a learning unit 522b by developing various programs P5b stored in the storage unit 53b. Furthermore, the CPU 52b of the information processing device 5b functions as a state information acquisition unit 523, a control information acquisition unit 524, a calculation unit 525, and a remote control unit 526 by developing various programs P5b stored in the storage unit 53b.

[0077] When the notification control unit 527 determines that the appearance of the target area Ta has changed using the change information, it notifies the administrator of the learning request information. In this embodiment, the notification control unit 527 uses the captured images Im1 to Im4 as change information to determine whether the appearance of the target area Ta has changed. Furthermore, when the notification control unit 527 determines that the appearance of the target area Ta has changed, it notifies the administrator of the learning request information via the notification unit 54.

[0078] When receiving a re-learning instruction from the administrator via the input operation unit 55 , the learning unit 522 b re-learns the first learning model Md1 .

[0079] Figure 13 This is a flowchart illustrating a method for relearning the first learning model Md1 in the third embodiment. Figure 13 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0080] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S221). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5b (step S222).

[0081] Following the change information acquisition step, the notification step is executed. In the notification step, the notification control unit 527 uses the captured images Im1 to Im4 to determine whether the appearance of the target area Ta has changed. If it is determined that the appearance of the target area Ta has changed (step S223: Yes), the notification control unit 527 notifies the administrator of the learning request information via the notification unit 54 (step S224). Following the notification step, the learning step is executed. During the learning step, if a re-learning instruction is received from the administrator via the input operation unit 55 (step S225: Yes), the learning unit 522b re-learns the first learning model Md1 (step S226). On the other hand, if it is determined that the appearance of the target area Ta has not changed (step S223: No), the information processing device 5b does not notify the administrator of the learning request information, does not re-learn the first learning model Md1, and ends the process.

[0082] According to the third embodiment, when the information processing device 5b uses change information to determine that the appearance of the target area Ta has changed, it can notify the administrator of learning request information. This allows the information processing device 5b to urge the administrator to take measures to relearn the first learning model Md1. Thus, the administrator, having recognized the learning request information, can quickly relearn the first learning model Md1 by instructing the information processing device 5b to relearn via the input operation unit 55.

[0083] It should be noted that the information processing device 5b may not include the learning unit 522b but may include the notification control unit 527. Even in such a configuration, when the information processing system 1b determines that the appearance state of the target area Ta has changed using the change information, it is possible to relearn the first learning model Md1 by connecting another device, such as a machine learning device (not shown), that implements the functions of the learning unit 522b to the information processing device 5b in a communicable manner.

[0084] D. Fourth embodiment: Figure 14: This is a diagram showing the structure of the information processing system 1c in the fourth embodiment. The information processing system 1c includes one or more vehicles 10, an external camera 9, and an information processing device 5c. In this embodiment, the information processing device 5c causes the second learning model Md2 serving as the machine learning model Md to learn for each appearance state of the object area Ta. Furthermore, the information processing device 5c obtains shape data Da as a parameter for signal generation by inputting captured images Im1 to Im4 into the second learning model Md2 corresponding to the appearance state of the object area Ta determined by the state information among the plurality of second learning models Md2 prepared for each appearance state of the object area Ta. The other structures of the information processing system 1c are the same as those of the first embodiment unless otherwise specified. The same figure marks are given to the same structures as the first embodiment and the description is omitted.

[0085] The information processing device 5c includes a communication unit 51, a CPU 52c, and a storage unit 53c. The storage unit 53c of the information processing device 5c stores various information including various programs P5c for controlling the operation of the information processing device 5c, a reference path Ip, and a plurality of second learning models Md2 prepared for each appearance state of the object area Ta.

[0086] Multiple second learning models Md2, when input with captured images Im1-Im4, detect the vehicle 10 from each of the captured images Im1-Im4 and output shape data Da. The second learning models Md2 are learned through supervised learning using a prioritized learning dataset corresponding to the appearance state of the target area Ta, among multiple prioritized learning datasets prepared for each appearance state of the target area Ta. Each prioritized learning dataset includes M (M is an integer greater than or equal to 2) first training data and N (N is an integer greater than or equal to 0 and less than M) second training data. The first training data is training data in which training images of the target area Ta and the vehicle 10, including one of the multiple appearance states classified as the target area Ta, are associated with correct labels for the region. The second training data is training data in which training images of the target area Ta and the vehicle 10, including the appearance states other than the one of the multiple appearance states classified as the target area Ta, are associated with correct labels for the region. The number M of first training data can be, for example, 1.5 times or more, or 2 times or more, of the number N of second training data. Furthermore, when there are multiple appearance states different from the one appearance state, the number M of first training data may be greater than the total number of second training data, for example. In other words, the second learning model Md2 is a machine learning model Md that prioritizes learning the feature values ​​of the object area Ta classified as belonging to the one appearance state by learning more of the first training data than the second training data.

[0087] The multiple second learning models Md2 prepared for each appearance state of the target area Ta include, for example, a no-dirt model Md2a and a dirty model Md2b. The no-dirt model Md2a is a second learning model Md2 that learns more from a first training image containing the vehicle 10 and the target area Ta in an unstained state than from a second training image containing the vehicle 10 and the target area Ta in an stained state. The no-dirt model Md2a is used when detecting the vehicle 10 from captured images Im1 to Im4 containing the vehicle 10 and the target area Ta in an unstained state. The dirty model Md2b is a second learning model Md2 that learns more from a first training image containing the vehicle 10 and the target area Ta in an stained state than from a second training image containing the vehicle 10 and the target area Ta in an unstained state. The dirty model Md2b is used when detecting the vehicle 10 from captured images Im1 to Im4 containing the vehicle 10 and the target area Ta in an stained state.

[0088] For the second learning model Md2, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the second learning model Md2 and the correct region labels.

[0089] The CPU 52c of the information processing device 5c develops various programs P5c stored in the storage unit 53c to function as a learning unit 522c, a state information acquisition unit 523, a model acquisition unit 528, a control information acquisition unit 524c, a calculation unit 525, and a remote control unit 526. The learning unit 522c learns the second learning model Md2 for each appearance state of the object area Ta. The model acquisition unit 528 acquires a second learning model Md2 selected from a plurality of second learning models Md2 prepared for each appearance state of the object area Ta, based on the appearance state of the object area Ta identified by the state information. The control information acquisition unit 524c acquires shape data Da by inputting the captured images Im1 to Im4 into the second learning model Md2 acquired by the model acquisition unit 528.

[0090] Figure 15 : is a flowchart showing a method of learning the second learning model Md2. Figure 15 The illustrated process is executed, for example, before the vehicle 10 starts driving without an operator.

[0091] In the preparation step, a priority learning dataset is prepared for each appearance state of the target area Ta (step S231). Following the preparation step, the learning step is performed. In the learning step, the learning unit 522c uses the priority learning dataset to learn the second learning model Md2 for each appearance state of the target area Ta (step S232).

[0092] Figure 16 This is a flowchart showing a method of acquiring shape data Da using the second learning model Md2. Figure 16 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0093] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring the captured images Im1 to Im4 as state information (step S331). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5c (step S332). The state information acquisition unit 523 analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S333).

[0094] Following the state information acquisition step, the model acquisition step is executed. In this step, the model acquisition unit 528 acquires a second learning model Md2 selected based on the appearance state of the target area Ta identified by the state information, from among a plurality of second learning models Md2 prepared for each appearance state of the target area Ta (step S334).

[0095] Following the model acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524c acquires shape data Da by inputting the captured images Im1 to Im4 to the second learning model Md2 acquired by the model acquisition unit 528 (step S335).

[0096] According to the fourth embodiment, the information processing device 5c learns the second learning model Md2 using more training images that include the vehicle 10 and the object area Ta classified as one appearance state than training images that include the vehicle 10 and the object area Ta classified as the remaining other appearance states. This allows the information processing device 5c to prepare a second learning model Md2 suitable for detecting the vehicle 10 from captured images Im1 to Im4 that include the vehicle 10 and the object area Ta classified as one appearance state.

[0097] Furthermore, according to the fourth embodiment, a second learning model Md2 is prepared for each appearance state of the target area Ta. Thus, the information processing device 5c can obtain state information and select a second learning model Md2 based on the appearance state of the target area Ta identified by the state information from among the multiple second learning models Md2 prepared for each appearance state of the target area Ta. Furthermore, by inputting the captured images Im1 to Im4 into the second learning model Md2 corresponding to the appearance state of the target area Ta included in those captured images, the information processing device 5c can accurately detect the vehicle 10 from the captured images Im1 to Im4. This improves the accuracy of detecting the vehicle 10 from the captured images Im1 to Im4.

[0098] E. Fifth embodiment: Figure 17 : This is a diagram showing the structure of the information processing system 1d in the fifth embodiment. The information processing system 1d includes one or more vehicles 10, an external camera 9, and an information processing device 5d. In this embodiment, the information processing device 5d associates the appearance state with the training image and learns the third learning model Md3 as the machine learning model Md. In addition, the information processing device 5d obtains shape data Da as a parameter for signal generation by inputting the captured images Im1 to Im4 into the third learning model Md3. The other structures of the information processing system 1d are the same as those of the first embodiment unless otherwise specified. The same figure marks are marked on the same structures as the first embodiment and the description is omitted.

[0099] The information processing device 5d includes a communication unit 51, a CPU 52d, and a storage unit 53d. The storage unit 53d of the information processing device 5d stores various information including various programs P5d for controlling the operation of the information processing device 5d, a reference path Ip, and a third learning model Md3.

[0100] The third learning model Md3, when fed with captured images Im1-Im4, detects the vehicle 10 from the captured images Im1-Im4 and outputs shape data Da. The third learning model Md3 is learned through supervised learning using a simultaneous learning dataset. The simultaneous learning dataset includes a plurality of training data sets that associate training images containing the target area Ta and the vehicle 10, correct region labels, and supplementary information indicating the appearance of the target area Ta contained in the training images.

[0101] For the third learning model Md3, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the third learning model Md3 and the correct region labels.

[0102] The CPU 52d of the information processing device 5d develops various programs P5d stored in the storage unit 53d to function as a learning unit 522d, a state information acquisition unit 523, a control information acquisition unit 524d, a calculation unit 525, and a remote control unit 526. The learning unit 522d uses the collective learning dataset to learn the third learning model Md3. The control information acquisition unit 524d acquires shape data Da by inputting the captured images Im1 to Im4 and the appearance of the object area Ta identified by the state information into the third learning model Md3.

[0103] Figure 18 3 is a flowchart illustrating a method for learning the third learning model Md3. Figure 18 The illustrated process is executed, for example, before the vehicle 10 starts driving without an operator.

[0104] In the preparation process, a data set for collective learning is prepared (step S241). Following the preparation process, a learning process is executed. In the learning process, the learning unit 522d learns the third learning model Md3 using the data set for collective learning (step S242).

[0105] Figure 19 This is a flowchart showing a method of acquiring shape data Da using the third learning model Md3. Figure 19 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0106] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring the captured images Im1 to Im4 as state information (step S341). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5d (step S342). The state information acquisition unit 523 analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S343).

[0107] Following the state information acquisition step, the control information acquisition step is executed. In this step, the control information acquisition unit 524d inputs the captured images Im1 to Im4 and the appearance of the target area Ta identified by the state information into the third learning model Md3. The control information acquisition unit 524d then acquires the shape data Da (step S344).

[0108] According to the fifth embodiment, the information processing device 5d can learn the third learning model Md3 using a plurality of training data sets that associate training images including the target area Ta and the vehicle 10, the correct region labels, and accompanying information indicating the appearance of the target area Ta included in the training images. Thus, the information processing device 5d can prepare the third learning model Md3 that has been learned by associating the appearance of the target area Ta.

[0109] Furthermore, according to the fifth embodiment, a third learning model Md3 is prepared and learned using a plurality of training data sets corresponding to training images including the target area Ta and the vehicle 10, the correct region labels, and accompanying information indicating the appearance of the target area Ta included in the training images. Consequently, the information processing device 5d acquires state information and inputs the captured images Im1-Im4 and the appearance of the target area Ta determined by the state information into the third learning model Md3, thereby enabling high-precision detection of the vehicle 10 from the captured images Im1-Im4. This improves the accuracy of detecting the vehicle 10 from the captured images Im1-Im4.

[0110] F. Sixth embodiment: Figure 20 is a diagram illustrating an example of a method for generating a training image Ti. In the present embodiment, the training image Ti used in learning the machine learning model Md is a synthetic image Ci. The synthetic image Ci is a two-dimensional image generated by synthesizing a virtual vehicle image Vi with a background image Bi of either an actual background image Br or a virtual background image Bv. The background image Bi does not include the vehicle 10 and includes the object area Ta. The actual background image Br is a background image Bi obtained by actually photographing the object area Ta using an external camera 9. The virtual background image Bv is a background image Bi that virtually reproduces the object area Ta. The virtual background image Bv is, for example, either two-dimensional CAD data or three-dimensional CAD data representing the object area Ta. The virtual vehicle image Vi is an image that virtually reproduces the vehicle 10. The virtual vehicle image Vi is either two-dimensional CAD data or three-dimensional CAD data representing the vehicle 10.

[0111] In this embodiment, the synthesized image Ci is generated by combining a virtual vehicle image Vi with a virtual background image Bv. Specifically, the synthesized image Ci is generated by, for example, superimposing the virtual vehicle image Vi onto each target area Ta of a factory manufacturing line recreated in virtual space. In other words, in this embodiment, the training image Ti is generated by leveraging digital twins.

[0112] It should be noted that if the virtual background image Bv is 3D data, it is converted from 3D data to 2D data during the generation of the synthesized image Ci. If the virtual vehicle image Vi is 3D data, it is converted from 3D data to 2D data during the generation of the synthesized image Ci. Furthermore, during the synthesis of the virtual vehicle image Vi with the background image Bi, the placement of the virtual vehicle image Vi relative to the background image Bi or the pixel values ​​of each of the images Bi, Ci, and Vi may be adjusted.

[0113] According to the sixth embodiment described above, the machine learning model Md can be made to learn by using the synthesized image Ci generated by synthesizing the virtual vehicle image Vi with the background image Bi as the training image Ti.

[0114] Furthermore, according to the sixth embodiment, when synthesizing the virtual vehicle image Vi with the background image Bi, by adjusting the placement of the virtual vehicle image Vi relative to the background image Bi, it is possible to vary at least one of the position and orientation of the vehicle 10. This makes it possible to easily generate multiple training images Ti in which at least one of the position and orientation of the vehicle 10 relative to the route Ro is varied.

[0115] Furthermore, according to the sixth embodiment, during the process of synthesizing the virtual vehicle image Vi with the background image Bi, the pixel values ​​of at least one of the background image Bi, the virtual vehicle image Vi, and the synthesized image Ci can be changed from their initial values. This makes it easy to generate multiple training images Ti with different shooting conditions, such as brightness. This also makes it easy to generate multiple training images Ti that assume that pixel values, such as brightness, vary depending on weather, time of day, season, lighting conditions, and the like.

[0116] Furthermore, according to the sixth embodiment, the machine learning model Md can be trained by using a synthesized image Ci generated by synthesizing a virtual vehicle image Vi with a virtual background image Bv as a training image Ti. In this way, by changing the structure of the virtual background image Bv, it is possible to easily reproduce object areas Ta with different appearances, thereby easily generating multiple training images Ti each containing an object area Ta with different appearances.

[0117] It should be noted that, when the training image Ti is a synthetic image Ci, as in the present embodiment, the correct answer label for the machine learning model Md that outputs the shape data Da can be a contour correct answer label Lc, rather than a region correct answer label Lr. The contour correct answer label Lc represents the contour of the vehicle 10 extracted from the virtual vehicle image Vi. This approach allows for easy identification of whether each region constituting the training image Ti represents the vehicle 10 or represents a region other than the vehicle 10, by extracting the contour of the virtual vehicle image Vi. This reduces the burden of preparing the machine learning model Md for learning.

[0118] G. Seventh Implementation Method: Figure 21 This diagram illustrates the configuration of an information processing system 1e in a seventh embodiment. The information processing system 1e includes one or more vehicles 10, an external camera 9, and an information processing device 5e. In this embodiment, the information processing device 5e inputs captured images Im1 to Im4 into a fourth learning model Md4, which serves as a machine learning model Md, to obtain rectangular coordinate values, which serve as signal generation parameters. The remaining configuration of the information processing system 1e is the same as that of the first embodiment, unless otherwise specified. Configurations identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0119] Figure 22 This figure illustrates rectangle coordinate values ​​Cr1 to Cr4. These coordinate values ​​are the coordinates of the four vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc. The vehicle circumscribed rectangle Rc is a quadrilateral defined to enclose the area occupied by vehicle 10 when projected onto road surface Rs in images Ti and Im1 to Im4. These coordinate values ​​are used to calculate at least one of the position and orientation of vehicle 10.

[0120] like Figure 21 As shown, the information processing device 5e includes a communication unit 51, a CPU 52e, and a storage unit 53e. The storage unit 53e of the information processing device 5e stores various information including various programs P5e for controlling the operation of the information processing device 5e, a reference path Ip, a fourth learning model Md4, a rectangle database Db1, and camera parameters Pa.

[0121] The fourth learning model Md4 outputs rectangle coordinate values ​​Cr1-Cr4 when input with captured images Im1-Im4. The learning dataset used to learn the fourth learning model Md4 includes a plurality of training data sets in which training images Ti, each containing the target area Ta and the vehicle 10, are associated with rectangle coordinate correct labels. The rectangle coordinate correct labels represent the coordinate values ​​Cr1-Cr4 of the four vertices Vc1-Vc4 of the vehicle circumscribed rectangle Rc set for the vehicle 10 in the training image Ti. Each coordinate value Cr1-Cr4 is associated with supplementary information indicating which of the four vertices Vc1-Vc4 of the vehicle circumscribed rectangle Rc the coordinate value Cr1-Cr4 represents. In this embodiment, the rectangle coordinate values ​​Cr1-Cr4 output by the fourth learning model Md4 are coordinate values ​​in the image coordinate system. Therefore, the rectangle coordinate correct labels represent each coordinate value Cr1-Cr4 in the image coordinate system.

[0122] For the fourth learning model Md4, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the fourth learning model Md4 and the correct label of the rectangular coordinates.

[0123] The rectangle database Db1 is used to calculate the coordinates of the position measurement point 10e of the vehicle 10 based on the rectangle coordinates Cr1 to Cr4. The rectangle database Db1 is a database showing the relative positional relationship between the vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc and the position measurement point 10e of the vehicle 10.

[0124] The camera parameters Pa are parameters related to the external camera 9. In this embodiment, the camera parameters Pa are used to transform the coordinate values ​​of the measuring point 10e of the vehicle 10 represented by the image coordinate system into the global coordinate system. The camera parameters Pa include, for example, the installation position, installation posture, and focal length of the external camera 9.

[0125] The CPU 52e of the information processing device 5e functions as a change information acquisition unit 521, a learning unit 522e, a state information acquisition unit 523, a control information acquisition unit 524e, a calculation unit 525e, and a remote control unit 526 by developing various programs P5e stored in the storage unit 53e.

[0126] The learning unit 522e uses the change information to determine whether the appearance of the target area Ta has changed since the fourth learning model Md4 was learned. If the appearance of the target area Ta has changed, the learning unit 522e relearns the fourth learning model Md4 using a relearning dataset that includes a plurality of training data corresponding to training images Ti of the vehicle 10 and the target area Ta after the change in appearance, and rectangular coordinate correct answer labels.

[0127] The control information acquisition unit 524e specifies the appearance state of the target area Ta using the state information, and inputs the captured images Im1 to Im4 to the fourth learning model Md4 corresponding to the appearance state of the target area Ta, thereby acquiring rectangular coordinate values ​​Cr1 to Cr4 represented by the image coordinate system.

[0128] The calculation unit 525e uses the rectangle database Db1 to calculate the coordinate values ​​of the position measurement point 10e of the vehicle 10 based on the rectangle coordinate values ​​Cr1 to Cr4. Furthermore, the calculation unit 525e uses the camera parameters Pa to transform the coordinate values ​​of the position measurement point 10e of the vehicle 10, represented in the image coordinate system, into coordinate values ​​in the global coordinate system. To obtain the camera parameters Pa, the calculation unit 525e can obtain, from among the camera identifiers used to identify the multiple external cameras 9, a camera identifier that indicates the external camera 9 that captured the captured images Im1 to Im4.

[0129] The calculation unit 525e also calculates the orientation of the vehicle 10. The calculation unit 525e calculates the orientation of the vehicle 10 using, for example, the shape of the vehicle circumscribed rectangle Rc and supplementary information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribed rectangle Rc the coordinate values ​​Cr1 to Cr4 represent.

[0130] Figure 23 : is a flowchart showing a re-learning method of the fourth learning model Md4. Figure 23 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0131] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S251). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5e (step S252).

[0132] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522e uses the captured images Im1 to Im4 to determine whether the appearance of the object area Ta has changed since the fourth learning model Md4 was learned. If it is determined that the appearance of the object area Ta has changed since the fourth learning model Md4 was learned (step S253: Yes), the learning unit 522e relearns the fourth learning model Md4 (step S254). On the other hand, if it is determined that the appearance of the object area Ta has not changed since the fourth learning model Md4 was learned (step S253: No), the learning unit 522e does not relearn the fourth learning model Md4 and ends the process.

[0133] Figure 24 This is a flowchart showing a method for acquiring rectangle coordinate values ​​Cr1 to Cr4 using the fourth learning model Md4. Figure 24 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0134] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring the captured images Im1 to Im4 as state information (step S351). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5e (step S352). The state information acquisition unit 523 analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S353).

[0135] Following the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524e acquires rectangle coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the fourth learning model Md4 (step S354).

[0136] According to the seventh embodiment, the information processing device 5e can learn the fourth learning model Md4 that outputs rectangle coordinate values ​​Cr1 to Cr4 when the captured images Im1 to Im4 are input. Thus, the information processing device 5e can obtain the rectangle coordinate values ​​Cr1 to Cr4 and generate a driving control signal by inputting the captured images Im1 to Im4 to the fourth learning model Md4.

[0137] It should be noted that the rectangular coordinate values ​​Cr1 to Cr4 can be coordinate values ​​in the camera coordinate system or in the global coordinate system. When the rectangular coordinate values ​​Cr1 to Cr4 are coordinate values ​​expressed in either the camera coordinate system or the global coordinate system, the fourth learning model Md4 is learned, for example, as follows. In this case, the fourth learning model Md4 learns, for example, by associating at least one of the camera identifier representing the external camera 9 that acquired the training image Ti and the camera parameters Pa of the external camera 9 that acquired the training image Ti with the training image Ti. In addition, in this case, in the rectangular coordinate correct answer label, each coordinate value Cr1 to Cr4 is expressed, for example, in the same coordinate system as the coordinate system of the output rectangular coordinate values ​​Cr1 to Cr4.

[0138] The information processing device 5e may also obtain the coordinate values ​​of the wheel contact surface as signal generation parameters instead of the rectangular coordinate values ​​Cr1 to Cr4. Even in this embodiment, the information processing device 5e can obtain the rectangular coordinate values ​​Cr1 to Cr4 and generate a travel control signal.

[0139] H. Eighth Implementation Method: Figure 25 This diagram illustrates the configuration of an information processing system 1f in an eighth embodiment. The information processing system 1f includes one or more vehicles 10, an external camera 9, and an information processing device 5f. In this embodiment, the information processing device 5f inputs captured images Im1 to Im4 to a fifth learning model Md5, which serves as a machine learning model Md, to obtain 3D coordinate values, which serve as signal generation parameters. The remaining configuration of the information processing system 1f is the same as that of the first embodiment, unless otherwise specified. Configurations identical to those of the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0140] Figure 26 This diagram illustrates three-dimensional coordinate values ​​Cb1 to Cb8. These coordinate values ​​are the coordinates of the eight vertices Vb1 to Vb8 of the vehicle circumscribing rectangular parallelepiped Rb. The vehicle circumscribing rectangular parallelepiped Rb is a rectangular parallelepiped that surrounds the vehicle 10 in images Ti and Im1 to Im4. These coordinate values ​​are used to calculate at least one of the position and orientation of the vehicle 10.

[0141] like Figure 25 As shown, the information processing device 5f includes a communication unit 51, a CPU 52f, and a storage unit 53f. The storage unit 53f of the information processing device 5f stores various information including various programs P5f for controlling the operation of the information processing device 5f, a reference path Ip, a fifth learning model Md5, a stereo database Db2, and camera parameters Pa.

[0142] The fifth learning model Md5 outputs three-dimensional coordinate values ​​Cb1-Cb8 when input with captured images Im1-Im4. The learning dataset used in learning the fifth learning model Md5 includes a plurality of training data sets in which training images Ti including the target area Ta and the vehicle 10 are associated with correct stereo coordinate labels. The correct stereo coordinate labels are correct labels indicating the coordinate values ​​Cb1-Cb8 of the eight vertices Vb1-Vb8 of the vehicle circumscribed rectangular parallelepiped Rb set for the vehicle 10 included in the training images Ti. The correct stereo coordinate labels are associated with each coordinate value Cb1-Cb8, indicating which of the eight vertices Vb1-Vb8 of the vehicle circumscribed rectangular parallelepiped Rb the coordinate value Cb1-Cb8 represents. In this embodiment, the three-dimensional coordinate values ​​Cb1-Cb8 output by the fifth learning model Md5 are coordinate values ​​in the camera coordinate system. Thus, in the stereo coordinate correct answer label, each coordinate value Cb1 to Cb8 is represented by the camera coordinate system. Furthermore, the fifth learning model Md5 learns at least one of the camera parameter Pa and the camera identifier used to obtain the camera parameter Pa in association with the training image Ti.

[0143] For the fifth learning model Md5, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the fifth learning model Md5 and the correct stereo coordinate labels.

[0144] The 3D database Db2 is used to calculate the coordinates of the measuring point 10e of the vehicle 10 based on the 3D coordinates Cb1 to Cb8. The 3D database Db2 is a database showing the relative positional relationship between the vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb and the measuring point 10e of the vehicle 10.

[0145] The CPU 52f of the information processing device 5f functions as a change information acquisition unit 521, a learning unit 522f, a state information acquisition unit 523, a control information acquisition unit 524f, a calculation unit 525f, and a remote control unit 526 by developing various programs P5f stored in the storage unit 53f.

[0146] The learning unit 522f uses the change information to determine whether the appearance of the target area Ta has changed since the fifth learning model Md5 was learned. If the appearance of the target area Ta has changed, the learning unit 522f retrains the fifth learning model Md5 using a retraining dataset that includes a plurality of training data sets corresponding to training images Ti of the vehicle 10 and the target area Ta after the change in appearance, and correct stereo coordinate labels.

[0147] The control information acquisition unit 524f specifies the appearance state of the target area Ta using the state information, and inputs the captured images Im1 to Im4 to the fifth learning model Md5 corresponding to the appearance state of the target area Ta, thereby acquiring stereo coordinate values ​​Cb1 to Cb8 represented by the camera coordinate system.

[0148] The calculation unit 525f uses the stereo database Db2 to calculate the coordinates of the position measurement point 10e of the vehicle 10 based on the stereo coordinate values ​​Cb1 to Cb8. Furthermore, the calculation unit 525f uses the camera parameters Pa to transform the coordinates of the position measurement point 10e of the vehicle 10 represented in the image coordinate system into coordinates in the camera coordinate system. To obtain the camera parameters Pa, the calculation unit 525f may obtain a camera identifier.

[0149] The calculation unit 525f also calculates the orientation of the vehicle 10. The calculation unit 525f calculates the orientation of the vehicle 10 using, for example, the shape of the vehicle circumscribed rectangular parallelepiped Rb and accompanying information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb the coordinate values ​​Cb1 to Cb8 represent.

[0150] Figure 27 3 is a flowchart illustrating a method of relearning the fifth learning model Md5. Figure 27 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0151] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S261). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5f (step S262).

[0152] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522f uses the captured images Im1 to Im4 to determine whether the appearance of the target area Ta has changed since the fifth learning model Md5 was learned. If it is determined that the appearance of the target area Ta has changed since the fifth learning model Md5 was learned (step S263: Yes), the learning unit 522f relearns the fifth learning model Md5 (step S264). On the other hand, if it is determined that the appearance of the target area Ta has not changed since the fifth learning model Md5 was learned (step S263: No), the learning unit 522f does not relearn the fifth learning model Md5, and the process ends.

[0153] Figure 28This is a flowchart showing a method for acquiring the three-dimensional coordinate values ​​Cb1 to Cb8 using the fifth learning model Md5 . Figure 28 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0154] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring the captured images Im1 to Im4 as state information (step S361). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5f (step S362). The state information acquisition unit 523 analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S363).

[0155] Following the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524f acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the fifth learning model Md5 (step S364).

[0156] According to the eighth embodiment, the information processing device 5f can learn the fifth learning model Md5, which outputs three-dimensional coordinate values ​​Cb1 to Cb8 when the captured images Im1 to Im4 are input. Thus, the information processing device 5f can obtain the three-dimensional coordinate values ​​Cb1 to Cb8 and generate a driving control signal by inputting the captured images Im1 to Im4 to the fifth learning model Md5.

[0157] It should be noted that the three-dimensional coordinate values ​​Cb1 to Cb8 may be coordinate values ​​in an image coordinate system or in a global coordinate system.

[0158] I. Ninth embodiment: Figure 29 This diagram illustrates the configuration of an information processing system 1g in a ninth embodiment. The information processing system 1g includes one or more vehicles 10, an external camera 9, and an information processing device 5g. In this embodiment, the information processing device 5g inputs captured images Im1 to Im4 to a sixth learning model Md6, which serves as a machine learning model Md, to obtain vehicle position information as a signal generation parameter. The remaining configuration of the information processing system 1g is the same as that of the first embodiment, unless otherwise specified. Configurations identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0159] The information processing device 5g includes a communication unit 51, a CPU 52g, and a storage unit 53g. The storage unit 53g of the information processing device 5g stores various information, including various programs P5g for controlling the operation of the information processing device 5g, a reference path Ip, a sixth learning model Md6, and camera parameters Pa. In this embodiment, a program for generating a travel control signal is prepared for each external camera 9. Consequently, a reference path Ip is prepared for each external camera 9. Furthermore, the target position determined during the process of generating the travel control signal is represented by X, Y, and Z coordinates in the camera coordinate system.

[0160] The sixth learning model Md6 outputs vehicle position information when input with captured images Im1 to Im4. In this embodiment, the sixth learning model Md6 outputs the coordinate values ​​of the location point 10e of the vehicle 10 and the orientation of the vehicle 10 when input with captured images Im1 to Im4. The learning dataset used for learning the sixth learning model Md6 includes a plurality of training data sets in which training images Ti, each including the target area Ta and the vehicle 10, are associated with position coordinate correct labels and orientation correct labels. The orientation correct label indicates the orientation of the vehicle 10 as shown in the training image Ti. The position coordinate correct label indicates the coordinate values ​​of the location point 10e of the vehicle 10 as shown in the training image Ti. In this embodiment, the coordinate values ​​of the location point 10e of the vehicle 10 output by the sixth learning model Md6 are in the camera coordinate system. Therefore, the coordinate values ​​of the location point 10e of the vehicle 10 are represented in the camera coordinate system in the location coordinate correct labels. Furthermore, the sixth learning model Md6 learns at least one of the camera parameter Pa and the camera identifier used to obtain the camera parameter Pa in association with the training image Ti.

[0161] For the sixth learning model Md6, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the sixth learning model Md6 and the correct position coordinate labels and the direction of the correct label.

[0162] The CPU 52g of the information processing device 5g functions as a change information acquisition unit 521, a learning unit 522g, a state information acquisition unit 523, a control information acquisition unit 524g, and a remote control unit 526 by developing various programs P5g stored in the storage unit 53g.

[0163] The learning unit 522g uses the change information to determine whether the appearance of the target area Ta has changed since the sixth learning model Md6 was learned. If the appearance of the target area Ta has changed, the learning unit 522g relearns the sixth learning model Md6 using a relearning dataset that includes a plurality of training images Ti, each of which includes the vehicle 10 and the target area Ta after the change in appearance, and a corresponding plurality of training data sets, including correct position coordinate labels and correct orientation labels.

[0164] The control information acquisition unit 524g uses the state information to determine the appearance of the target area Ta and inputs the captured images Im1 to Im4 into the sixth learning model Md6 corresponding to the appearance of the target area Ta. The control information acquisition unit 524g thereby obtains the coordinates of the measurement point 10e of the vehicle 10 and the orientation of the vehicle 10, expressed in the camera coordinate system.

[0165] Figure 30 : is a flowchart showing a re-learning method of the sixth learning model Md6. Figure 30 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0166] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S271). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5f (step S272).

[0167] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522g uses the captured images Im1 to Im4 to determine whether the appearance of the object area Ta has changed since the sixth learning model Md6 was learned. If it is determined that the appearance of the object area Ta has not changed since the sixth learning model Md6 was learned (step S273: Yes), the learning unit 522g relearns the sixth learning model Md6 (step S274). On the other hand, if it is determined that the appearance of the object area Ta has not changed since the sixth learning model Md6 was learned (step S273: No), the learning unit 522g does not relearn the sixth learning model Md6, and the process ends.

[0168] Figure 31 This is a flowchart showing a method for acquiring vehicle position information using the sixth learning model Md6. Figure 31 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0169] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring the captured images Im1 to Im4 as state information (step S371). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5g (step S372). The state information acquisition unit 523 analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S373).

[0170] Following the state information acquisition step, the control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524g inputs the captured images Im1 to Im4 to the sixth learning model Md6 to acquire the coordinate values ​​of the measuring point 10e indicating the position of the vehicle 10 (step S374).

[0171] According to the ninth embodiment, the information processing device 5g can learn the sixth learning model Md6 that outputs the position and orientation of the vehicle 10 when the captured images Im1 to Im4 are input. Thus, by inputting the captured images Im1 to Im4 to the sixth learning model Md6, the information processing device 5g can obtain vehicle position information and generate a travel control signal.

[0172] According to the ninth embodiment, a program for generating a travel control signal is prepared for each external camera 9. Thus, the information processing device 5g can acquire vehicle position information and generate a travel control signal without performing conversion processing from the camera coordinate system to the global coordinate system.

[0173] J. Tenth embodiment: Figure 32 This diagram illustrates the configuration of an information processing system 1h in a tenth embodiment. The information processing system 1h includes one or more vehicles 10, an external camera 9, and an information processing device 5h. In this embodiment, the information processing device 5h obtains a driving control signal by inputting captured images Im1 to Im4 and a camera identifier into a seventh learning model Md7, which serves as a machine learning model Md. The remaining configuration of the information processing system 1h is the same as that of the first embodiment, unless otherwise specified. Configurations identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0174] The information processing device 5h includes a communication unit 51, a CPU 52h, and a storage unit 53h. The storage unit 53h of the information processing device 5h stores various information including various programs P5h for controlling the operation of the information processing device 5h and the seventh learning model Md7.

[0175] The seventh learning model Md7 outputs a driving control signal when fed with captured images Im1-Im4 and the camera identifier representing the external camera 9 that captured these images. The learning dataset used to learn the seventh learning model Md7 includes a plurality of training data sets that associate multiple training images Ti encompassing the target area Ta and the vehicle 10, the camera identifier representing the external camera 9 that captured the training images Ti, and correct signal labels. Correct signal labels represent target values ​​for each parameter included in the driving control signal. These correct signal labels are generated based on the reference path Ip and the target speed within the captured area RG of the external camera 9 that captured the training images Ti.

[0176] For the seventh learning model Md7, for example, a CNN that implements either semantic segmentation or instance segmentation can be used. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the seventh learning model Md7 and the correct signal label.

[0177] The CPU 52h of the information processing device 5h functions as a change information acquisition unit 521, a learning unit 522h, a state information acquisition unit 523, a control information acquisition unit 524h, and a remote control unit 526h by developing various programs P5h stored in the storage unit 53h.

[0178] The learning unit 522h uses the change information to determine whether the appearance of the target area Ta has changed since the seventh learning model Md7 was learned. If the appearance of the target area Ta has changed, the learning unit 522h performs the following processing. In this case, the learning unit 522h retrains the seventh learning model Md7 using a retraining dataset that includes a plurality of training data sets corresponding to training images Ti of the vehicle 10 and the target area Ta after the change in appearance, camera identifiers, and correct signal labels.

[0179] The control information acquisition unit 524h uses the state information to determine the appearance of the target area Ta. It then inputs the captured images Im1 to Im4 and the camera identifier into the seventh learning model Md7 corresponding to the appearance of the target area Ta, thereby acquiring a driving control signal. The remote control unit 526h transmits the driving control signal acquired by the control information acquisition unit 524h to the vehicle 10.

[0180] Figure 33 : is a flowchart showing a method of re-learning the seventh learning model Md7. Figure 33 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0181] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S281). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5h (step S282).

[0182] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522h uses the captured images Im1 to Im4 to determine whether the appearance of the target area Ta has changed since the seventh learning model Md7 was learned. If it is determined that the appearance of the target area Ta has changed since the seventh learning model Md7 was learned (step S283: Yes), the learning unit 522h relearns the seventh learning model Md7 (step S284). On the other hand, if it is determined that the appearance of the target area Ta has not changed since the seventh learning model Md7 was learned (step S283: No), the learning unit 522h does not relearn the seventh learning model Md7, and the process ends.

[0183] Figure 34 This is a flowchart showing a method of acquiring a travel control signal using the seventh learning model Md7. Figure 34 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0184] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 for acquiring the captured images Im1 to Im4 as state information (step S381). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5h (step S382). The state information acquisition unit 523 analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S383).

[0185] Following the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524h acquires a travel control signal by inputting the captured images Im1 to Im4 and the camera identifier to the seventh learning model Md7 (step S384).

[0186] According to the tenth embodiment, the information processing device 5h can learn the seventh learning model Md7 that outputs a travel control signal when the captured images Im1 to Im4 and the camera identifier are input. Thus, the information processing device 5h can obtain the travel control signal by inputting the captured images Im1 to Im4 and the camera identifier into the seventh learning model Md7.

[0187] It should be noted that the information input to the seventh learning model Md7 is not limited to the information described above. The information processing device 5h can also obtain a driving control signal by inputting other information, such as speed information indicating the driving speed of the vehicle 10, path information indicating the reference path Ip of the vehicle 10, and a pinion angle, into the seventh learning model Md7 in addition to the captured images Im1 to Im4 and the camera identifier. This allows the information processing device 5h to obtain a more appropriate driving control signal tailored to the driving conditions of the vehicle 10 and the reference path Ip.

[0188] K. Eleventh embodiment: Figure 35 This diagram illustrates the configuration of an information processing system 1i in an eleventh embodiment. The information processing system 1i includes one or more vehicles 10, an external camera 9, and an information processing device 5i. In this embodiment, when it is determined that the appearance of the target area Ta has changed, the information processing device 5i causes the first learning model Md1 to perform transfer learning, which is a form of learning. Transfer learning is an example of additional learning. Unless otherwise specified, the remaining configuration of the information processing system 1i is the same as that of the first embodiment. Configurations identical to those of the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.

[0189] The information processing device 5i includes a communication unit 51, a CPU 52i, and a storage unit 53i. The storage unit 53i of the information processing device 5i stores various information including various programs P5i for controlling the operation of the information processing device 5i and a first learning model Md1.

[0190] The CPU 52i of the information processing device 5i develops various programs P5i stored in the storage unit 53i to function as a change information acquisition unit 521, a learning unit 522i, a state information acquisition unit 523, a control information acquisition unit 524i, a calculation unit 525, and a remote control unit 526.

[0191] The learning unit 522i uses the change information to determine whether the appearance state of the object area Ta has changed since the first learning model Md1 was learned. And, when it is determined that the appearance state of the object area Ta has changed, the learning unit 522i performs the following processing. In this case, the learning unit 522i causes the first learning model Md1 that has been learned using the initial learning data set to perform transfer learning using the transfer learning data set. The transfer learning data set includes a plurality of training data that establish correspondences between the training image Ti including the vehicle 10 and the object area Ta after the appearance state has changed and the correct regional answer label Lr. In transfer learning, the learning unit 522i performs the following processing, for example. Specifically, the learning unit 522i adds a layer of the CNN that constitutes the first learning model Md1. And, using the transfer learning data set, the learning unit 522i updates the parameters related to the added layer instead of updating the trained parameters.

[0192] The control information acquisition unit 524i uses the state information to determine the appearance of the object area Ta and inputs the captured images Im1 to Im4 to the first learning model Md1 corresponding to the appearance of the object area Ta, thereby acquiring shape data Da. Specifically, if the appearance of the object area Ta determined by the state information differs from the appearance of the object area Ta in the training image Ti used when learning the first learning model Md1, the control information acquisition unit 524i performs the following processing. In this case, the control information acquisition unit 524i inputs the captured images Im1 to Im4 to the first learning model Md1 that has undergone transfer learning using a plurality of training data sets that associate the training image Ti containing the vehicle 10 and the object area Ta whose appearance was determined by the state information with the correct region label Lr. On the other hand, if the appearance of the object area Ta determined by the state information is the same as the appearance of the object area Ta in the training image Ti used when learning the first learning model Md1, the control information acquisition unit 524i performs the following processing. In this case, the control information acquisition unit 524i inputs the captured images Im1 to Im4 to the first learning model Md1 without performing transfer learning using a plurality of training data corresponding to the training image Ti of the target area Ta including the vehicle 10 and the appearance state determined by the state information and the area correct answer label Lr.

[0193] Figure 36 3 is a flowchart illustrating a transfer learning method of the first learning model Md1 . Figure 36 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0194] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S291). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5i (step S292).

[0195] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522i uses the captured images Im1 to Im4 to determine whether the appearance of the object area Ta has changed since the first learning model Md1 stored in the storage unit 53i was learned. If it is determined that the appearance of the object area Ta has changed since the first learning model Md1 stored in the storage unit 53i was learned (step S293: Yes), the learning unit 522i performs transfer learning on the first learning model Md1 (step S294). On the other hand, if it is determined that the appearance of the object area Ta has not changed since the first learning model Md1 stored in the storage unit 53i was learned (step S293: No), the learning unit 522i does not perform transfer learning on the first learning model Md1, and the process ends.

[0196] Figure 37 This is a flowchart showing a method of acquiring shape data Da in the eleventh embodiment. Figure 37 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0197] In the state information acquisition process, the state information acquisition unit 523 sends a request signal to the external camera 9 to acquire the captured images Im1 to Im4 as state information (step S391). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5i (step S392). The state information acquisition unit 523i analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S393).

[0198] Following the state information acquisition step, the control information acquisition step is executed. In this step, the control information acquisition unit 524i acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1, which was learned using training images Ti containing the vehicle 10 and the target area Ta whose appearance was determined by the state information. Specifically, if the appearance determined by the state information differs from the appearance of the target area Ta in the training images Ti used when learning the first learning model Md1, stored in the storage unit 53i (step S394 returns Yes), the control information acquisition unit 524i performs the following processing. In this case, the control information acquisition unit 524i inputs the captured images Im1 to Im4 to the first learning model Md1, which has undergone transfer learning using a plurality of training data sets that associate training images Ti containing the vehicle 10 and the target area Ta whose appearance was determined by the state information with the correct region labels Lr. Thus, the control information acquisition unit 524i acquires shape data Da (step S395). On the other hand, if the appearance state determined by the state information is identical to the appearance state of the target area Ta in the training image Ti used when learning the first learning model Md1 stored in the storage unit 53i (step S394: No), the control information acquisition unit 524i performs the following processing. In this case, the control information acquisition unit 524i obtains shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 that has not undergone transfer learning using a plurality of training data that associate the training image Ti containing the vehicle 10 and the target area Ta whose appearance state is determined by the state information with the correct region label Lr (step S396).

[0199] According to the eleventh embodiment described above, when the information processing device 5i determines that the appearance of the object area Ta has changed using change information, it can perform the following processing. In this case, the information processing device 5i can cause the first learning model Md1, which has been trained using multiple training data sets that correspond to training images Ti containing the vehicle 10 and the object area Ta before the appearance change, and the correct regional labels Lr, to perform transfer learning using multiple training data sets that correspond to training images Ti containing the vehicle 10 and the object area Ta after the appearance change, and the correct regional labels Lr. This can suppress a decrease in detection accuracy when detecting the vehicle 10 from captured images Im1 to Im4 when the appearance of the object area Ta has changed.

[0200] Furthermore, according to the eleventh embodiment, when the appearance state determined by the state information differs from the training images Ti used when learning the first learning model Md1, the information processing device 5i can obtain shape data Da by performing the following processing. In this case, the information processing device 5i can obtain shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1, which has been transfer-learned using a plurality of training data sets that associate the training images Ti and the correct region labels Lr, which include the vehicle 10 and the target region Ta whose appearance state is determined by the state information. This prevents a decrease in the accuracy of detecting the vehicle 10 from the captured images Im1 to Im4 when the appearance state of the target region Ta differs from that when the first learning model Md1 was learned.

[0201] L. Twelfth Implementation Method: Figure 38 This diagram illustrates the configuration of an information processing system 1j in a twelfth embodiment. The information processing system 1j includes one or more vehicles 10, an external camera 9, and an information processing device 5j. In this embodiment, upon determining that the appearance of the target area Ta has changed, the information processing device 5j discards the first learning model Md1 and relearns the eighth learning model Md8, which is the machine learning model Md. The remaining configuration of the information processing system 1j is the same as that of the first embodiment, unless otherwise specified. Configurations identical to those of the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0202] The information processing device 5j includes a communication unit 51, a CPU 52j, and a storage unit 53j. The storage unit 53j of the information processing device 5j stores various information including various programs P5j for controlling the operation of the information processing device 5j and the first learning model Md1.

[0203] The CPU 52j of the information processing device 5j functions as a change information acquisition unit 521, a learning unit 522j, a state information acquisition unit 523, a control information acquisition unit 524j, a calculation unit 525, and a remote control unit 526 by developing various programs P5j stored in the storage unit 53j.

[0204] The learning unit 522j uses the change information to determine whether the appearance of the object area Ta has changed since the first learning model Md1 was learned. If it is determined that the appearance of the object area Ta has changed, the learning unit 522j performs the following processing. In this case, the learning unit 522j discards the first learning model Md1 learned using the initial learning dataset. Furthermore, the learning unit 522j uses a new learning dataset to newly learn the eighth learning model Md8. The new learning dataset includes a plurality of training data sets that associate training images Ti, each including the vehicle 10 and the object area Ta after the appearance change, with the correct region labels Lr. For example, a CNN that implements either semantic segmentation or instance segmentation can be used for the eighth learning model Md8. During CNN learning, for example, backpropagation (error backpropagation) is used to update the CNN parameters in a manner that reduces the error between the output of the eighth learning model Md8 and the correct region labels Lr. The learning unit 522j stores the generated eighth learning model Md8 in the storage unit 53j.

[0205] The control information acquisition unit 524j uses the state information to determine the appearance of the object area Ta and inputs the captured images Im1 to Im4 to the machine learning model Md corresponding to the appearance of the object area Ta, thereby acquiring shape data Da. Specifically, if the appearance of the object area Ta determined by the state information differs from the appearance of the object area Ta in the training images Ti used when learning the first learning model Md1, the control information acquisition unit 524j performs the following processing. In this case, the control information acquisition unit 524j inputs the captured images Im1 to Im4 to the eighth learning model Md8 to acquire shape data Da. On the other hand, if the appearance of the object area Ta determined by the state information is the same as the appearance of the object area Ta in the training images Ti used when learning the first learning model Md1, the control information acquisition unit 524j performs the following processing. In this case, the control information acquisition unit 524j inputs the captured images Im1 to Im4 to the first learning model Md1 to acquire shape data Da.

[0206] Figure 39 This is a flowchart showing a method for learning the learning models Md1 and Md8 in the twelfth embodiment. Figure 39 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0207] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S301). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5i (step S302).

[0208] Following the change information acquisition step, the learning step is executed. In the learning step, the learning unit 522j uses the captured images Im1 to Im4 to determine whether the appearance of the target area Ta has changed since the first learning model Md1 stored in the storage unit 53j was learned. If it is determined that the appearance of the target area Ta has changed since the first learning model Md1 stored in the storage unit 53j was learned (step S303: Yes), the learning unit 522j discards the first learning model Md1 (step S304). Then, the learning unit 522j learns a new eighth learning model Md8 (step S305). On the other hand, if it is determined that the appearance of the target area Ta has not changed since the first learning model Md1 stored in the storage unit 53j was learned (step S303: No), the learning unit 522j does not discard the first learning model Md1 and learns a new eighth learning model Md8, terminating the process.

[0209] Figure 40 This is a flowchart showing a method of acquiring shape data Da in the twelfth embodiment. Figure 40 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 starts traveling without human driver.

[0210] In the state information acquisition process, the state information acquisition unit 523 sends a request signal to the external camera 9 to acquire the captured images Im1 to Im4 as state information (step S401). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5j (step S402). The state information acquisition unit 523j analyzes the captured images Im1 to Im4 to determine the appearance state of the target area Ta (step S403).

[0211] Following the state information acquisition step, the control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524j obtains shape data Da by inputting the captured images Im1 to Im4 to the machine learning model Md corresponding to the appearance state of the object area Ta. Specifically, if the appearance state determined by the state information differs from the appearance state of the object area Ta in the training images Ti used when learning the first learning model Md1 stored in the storage unit 53j (step S404: Yes), the control information acquisition unit 524j performs the following processing. In this case, the control information acquisition unit 524j obtains shape data Da by inputting the captured images Im1 to Im4 to the eighth learning model Md8 (step S405). On the other hand, if the appearance state determined by the state information is the same as the appearance state of the object area Ta in the training images Ti used when learning the first learning model Md1 stored in the storage unit 53j (step S404: No), the control information acquisition unit 524j performs the following processing. In this case, the control information acquisition unit 524 j acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 (step S406 ).

[0212] According to the twelfth embodiment, when the information processing device 5j determines that the appearance of the object area Ta has changed using change information, it can perform the following processing. In this case, the information processing device 5j can discard the first learning model Md1, which was learned using multiple training data sets corresponding to the training images Ti containing the vehicle 10 and the object area Ta after the appearance change, and the correct regional labels Lr, and relearn the eighth learning model Md8 using multiple training data sets corresponding to the training images Ti containing the vehicle 10 and the object area Ta after the appearance change, and the correct regional labels Lr. This can suppress a decrease in detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4 when the appearance of the object area Ta has changed.

[0213] Furthermore, according to the twelfth embodiment, when the appearance state determined by the state information differs from the training images Ti used when learning the first learning model Md1, the information processing device 5j can obtain shape data Da by performing the following processing. In this case, the information processing device 5j can obtain shape data Da by inputting the captured images Im1 to Im4 into the eighth learning model Md8, which has been trained using a plurality of training data sets that associate the training images Ti and the correct region labels Lr, including the vehicle 10 and the target region Ta whose appearance state is determined by the state information. This prevents a decrease in the accuracy of detecting the vehicle 10 from the captured images Im1 to Im4 when the appearance state of the target region Ta differs from that when the first learning model Md1 was learned.

[0214] M. Thirteenth embodiment: Figure 41 This diagram illustrates the configuration of an information processing system 1v in a thirteenth embodiment. The information processing system 1v includes one or more vehicles 10v, an external camera 9, and an information processing device 5v. In this embodiment, a vehicle control device 15v mounted on the vehicle 10v uses the detection results output by the external camera 9 to obtain vehicle position information and generate a driving control signal. The remaining configuration of the information processing system 1v is the same as that of the first embodiment, unless otherwise specified. Configurations identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0215] The information processing device 5v includes a communication unit 51, a CPU 52v, and a storage unit 53v. The storage unit 53v of the information processing device 5v stores various information including various programs P5v for controlling the operation of the information processing device 5v and the first learning model Md1.

[0216] The CPU 52v of the information processing device 5v expands various programs P5v stored in the storage unit 53v to function as a change information acquisition unit 521, a learning unit 522, a state information acquisition unit 523, and an updating unit 529. The updating unit 529 updates the first learning model Md1 stored in the vehicle control device 15v each time the first learning model Md1 is relearned, so that the first learning model Md1 stored in the vehicle control device 15v is in the latest state.

[0217] The vehicle control device 15v includes a CPU 150v, a storage unit 157v, and an input / output interface 159. The storage unit 157v of the vehicle control device 15v stores various programs P15v for controlling the operation of the vehicle control device 15v and various information including a reference path Ip.

[0218] The CPU 150v of the vehicle control device 15v develops various programs P15v stored in the storage unit 157v to function as a control information acquisition unit 154, a calculation unit 155, a signal generation unit 156, and a motion control unit 153v. The control information acquisition unit 154 acquires shape data Da by inputting the captured images Im1 to Im4 into the first learning model Md1. The calculation unit 155 uses the shape data Da to acquire vehicle position information. The signal generation unit 156 uses the vehicle position information and the reference path Ip to determine the target location to which the vehicle 10v should next proceed. The signal generation unit 156 then generates a travel control signal for causing the vehicle 10v to travel toward the determined target location. The motion control unit 153v uses the generated travel control signal to control the actuators, thereby causing the vehicle 10v to travel.

[0219] Figure 42 1 is a flowchart showing a method of causing a vehicle 10 v to travel in an information processing system 1 v according to a thirteenth embodiment. Figure 42 The illustrated flow is repeatedly executed at predetermined time intervals, for example, starting from the time when the vehicle 10 v starts traveling without human driver.

[0220] The calculation unit 155 of the vehicle control device 15v mounted on the vehicle 10v uses the shape data Da to obtain vehicle position information (step S510). The signal generation unit 156 determines the target position to which the vehicle 10v should next go (step S520). The signal generation unit 156 generates a travel control signal for causing the vehicle 10v to travel toward the determined target position (step S530). The motion control unit 153 controls the actuator using the generated travel control signal to cause the vehicle 10v to travel at the acceleration and steering angle indicated by the travel control signal (step S540). The vehicle control device 15v repeats the acquisition of vehicle position information, determination of the target position, generation of the travel control signal, and control of the actuator in a predetermined cycle.

[0221] According to the thirteenth embodiment, the information processing system 1 v can cause the vehicle 10 v to travel under autonomous control without remotely controlling the vehicle 10 v using the information processing device 5 v.

[0222] N. Other implementation methods: N-1. Other Implementation Methods 1: Factors influencing changes in the appearance of the target area Ta may also include factors other than dirt generated in the target area Ta. Factors influencing changes in the appearance of the target area Ta may also include, for example, irregularities generated in the target area Ta. The irregularities generated in the target area Ta may be depressions in the passage Ro caused by thinning of the road surface Rs due to, for example, increased travel frequency of the vehicle 10 or 10v. The irregularities generated in the target area Ta may also be protrusions in the passage Ro caused by bulging of the road surface Rs. With this approach, the learning units 522, 522a-522j can train the machine learning model Md based on the occurrence of irregularities in the target area Ta. This can suppress a decrease in the accuracy of the control information output by the machine learning model Md when the shape of the road surface Rs changes. Furthermore, the control information acquisition units 154, 524, 524c-524j can acquire control information using the machine learning model Md corresponding to the occurrence of irregularities in the target area Ta, enabling the vehicle 10 or 10v to travel unmanned.

[0223] N-2. Other Implementation Methods 2: The factors that influence the change in the appearance of the target area Ta may be other than dirt generated in the target area Ta. For example, the factors that influence the change in the appearance of the target area Ta may be objects other than the vehicle 10 that are arranged so as to be movable within the target area Ta. That is, the factors that influence the change in the appearance of the target area Ta may be at least one of an object other than the vehicle 10 that is arranged so as to be movable within the target area Ta and an object that is removed from the target area Ta. Examples of objects that are arranged so as to be movable within the target area Ta include equipment such as manufacturing equipment, road cones, and signboards, whose placement can be changed as appropriate. With this configuration, the information processing devices 5, 5a-5j, and 5v can train the machine learning model Md based on the placement of objects within the target area Ta. This can suppress a decrease in the accuracy of the control information output by the machine learning model Md when the placement of objects within the target area Ta changes. Furthermore, by obtaining control information using the machine learning model Md corresponding to the placement of objects within the target area Ta, the information processing devices 5, 5a-5j, and 5v can enable the vehicles 10 and 10v to travel unmanned.

[0224] N-3. Other Implementation Methods 3: The factors that affect the change in the appearance of the object area Ta may be other than the dirt generated in the object area Ta. The factors that affect the change in the appearance of the object area Ta may be, for example, the color of the road surface of the passage Ro. The color of the road surface of the passage Ro may be, for example, the color of a demarcation line such as a white line, or the color of the ground including the passage Ro, or the color of the road surface Rs after paving. If this is the case, the information processing devices 5, 5a to 5j, 5v can make the machine learning model Md learn according to the color of the road surface of the passage Ro. Thus, for example, when the color of the demarcation line becomes lighter due to scratches caused by years of degradation or becomes darker due to repainting, etc., depending on the passage of time and the maintenance status of the demarcation line, the following can be achieved. In this case, the accuracy of the driving control signal and the signal generation parameters output from the machine learning model Md can be suppressed from decreasing. In addition, if the paint color of the ground surface including the passage Ro changes due to the passage of time and the maintenance status of the road surface, such as due to scratches and lightening due to the increase in the number of times the vehicles 10 and 10v travel, or due to darkening due to repainting, or due to changes in hue compared to before repainting, the following can be achieved. In this case, the accuracy of the driving control signal and signal generation parameters output from the machine learning model Md can be suppressed. In addition, if the color of the paved road surface Rs changes due to the paving status of the passage Ro, such as due to darkening due to repaving, the accuracy of the driving control signal and signal generation parameters output from the machine learning model Md can be suppressed. Moreover, by obtaining control information using the machine learning model Md corresponding to the road surface color of the passage Ro, the information processing devices 5, 5a to 5j, 5v can cause the vehicles 10 and 10v to travel unmanned.

[0225] N-4. Other Implementation Methods 4: At least part of the functions of the information processing devices 5, 5a to 5j, and 5v may be a function of the vehicle control device 15, 15v, or a function of the external camera 9. For example, if all the functions of the information processing devices 5, 5a to 5j, and 5v are implemented by the external camera 9, the information processing devices 5, 5a to 5j, and 5v can obtain the coordinate values ​​Cr1 to Cr4 and Cb1 to Cb8 in the camera coordinate system without using a camera identifier.

[0226] N-5. Other Implementation Methods 5: In each of the first through twelfth embodiments, the information processing devices 5, 5a through 5j perform the processing from acquiring vehicle position information to generating a travel control signal. Alternatively, at least a portion of the processing from acquiring vehicle position information to generating a travel control signal may be performed by the vehicle 10. For example, the following embodiments (1) through (3) may be employed.

[0227] (1) The information processing devices 5, 5a to 5j can obtain vehicle position information, determine the target position to which the vehicle 10 should next go, and generate a route from the current location of the vehicle 10 indicated by the obtained position information to the target position. The information processing devices 5, 5a to 5j can generate a route to the target position between the current location and the destination, or can generate a route to the destination. The information processing devices 5, 5a to 5j can transmit the generated route to the vehicle 10. The vehicle 10 can generate a travel control signal so that the vehicle 10 travels along the route received from the information processing devices 5, 5a to 5j, and use the generated travel control signal to control the actuator.

[0228] (2) The information processing devices 5, 5a to 5j can obtain vehicle position information and transmit the obtained position information to the vehicle 10. The vehicle 10 can determine a target position to which the vehicle 10 should proceed, generate a route from the current position of the vehicle 10 indicated by the received position information to the target position, generate a travel control signal so that the vehicle 10 travels along the generated route, and control the actuator using the generated travel control signal.

[0229] (3) In the above-mentioned methods (1) and (2), the vehicle 10 may be equipped with internal sensors, and the detection results outputted from the internal sensors may be used for at least one of the generation of the route and the generation of the driving control signal. The internal sensors are sensors mounted on the vehicle 10. Specifically, the internal sensors may include, for example, cameras, LiDAR, millimeter-wave radars, ultrasonic sensors, GPS sensors, acceleration sensors, gyroscope sensors, etc. For example, in the above-mentioned method (1), the information processing devices 5, 5a to 5j may obtain the detection results of the internal sensors and reflect the detection results of the internal sensors on the route when generating the route. In the above-mentioned method (1), the vehicle 10 may obtain the detection results of the internal sensors and reflect the detection results of the internal sensors on the driving control signal when generating the driving control signal. In the above-mentioned method (2), the vehicle 10 may obtain the detection results of the internal sensors and reflect the detection results of the internal sensors on the route when generating the route. In the above-mentioned method (2), the vehicle 10 may obtain the detection results of the internal sensors and reflect the detection results of the internal sensors on the route when generating the driving control signal.

[0230] (4) In the thirteenth embodiment, the vehicle 10v may be equipped with internal sensors, and detection results output by the internal sensors may be used for at least one of route generation and travel control signal generation. For example, the vehicle 10v may obtain detection results from the internal sensors and reflect the detection results from the internal sensors in the route generation. The vehicle 10v may obtain detection results from the internal sensors and reflect the detection results from the internal sensors in the travel control signal generation.

[0231] (5) In the thirteenth embodiment described above, the vehicle 10v uses the detection results of an external sensor installed in a location different from the vehicle 10v to obtain vehicle position information. In contrast, the vehicle 10v may be equipped with an internal sensor, and the vehicle 10v may use the detection results of the internal sensor to obtain vehicle position information, determine the target position to which the vehicle 10v should go next, generate a path from the current location of the vehicle 10v represented by the obtained vehicle position information to the target position, generate a driving control signal for driving on the generated path, and use the generated driving control signal to control the actuator. In this case, the vehicle 10v can travel without using the detection results of the external sensor at all. It should be noted that the vehicle 10v may also obtain the target arrival time and congestion information from outside the vehicle 10v, and reflect the target arrival time and congestion information on at least one of the path and the driving control signal.

[0232] (6) In each of the first to twelfth embodiments, the information processing devices 5, 5a to 5j automatically generate travel control signals to be transmitted to the vehicle 10. Alternatively, the information processing devices 5, 5a to 5j may generate travel control signals to be transmitted to the vehicle 10 in response to an operation by an external operator located outside the vehicle 10. For example, the operator may operate an operating device including a display that displays images output from external sensors, a steering wheel for remotely operating the vehicle 10, an accelerator pedal, a brake pedal, and a communication device for communicating with the information processing devices 5, 5a to 5j via wired or wireless communication, and the information processing devices 5, 5a to 5j may generate travel control signals in accordance with the operation applied to the operating device.

[0233] N-6. Other Implementation Methods 6: In each of the above-mentioned embodiments, the vehicle 10, 10v only needs to have a structure capable of moving without a driver, for example, it can also be in the form of a platform having a structure described below. Specifically, in order to perform the three functions of "driving", "turning" and "stopping" without a driver, the vehicle 10, 10v only needs to have at least a vehicle control device 15, 15v, a drive device 11, a steering device 12 and a braking device 13. In the case where the vehicle 10, 10v obtains information from the outside for unmanned driving, the vehicle 10, 10v only needs to have a communication device 14. In other words, the vehicle 10, 10v that can move without a driver can be equipped with at least a portion of interior components such as a driver's seat and an instrument panel, can be equipped with at least a portion of exterior components such as a bumper and a fender, and can also be equipped with a body shell. In this case, the remaining components, such as the body shell, can be assembled onto the vehicle 10, 10v before the vehicle 10, 10v is shipped from the factory, or the remaining components, such as the body shell, can be assembled onto the vehicle 10, 10v after the vehicle 10, 10v is shipped from the factory. Each component can be assembled from any direction, such as the top, bottom, front, rear, right, or left side of the vehicle 10, 10v, and can be assembled from the same direction or from different directions. It should be noted that the gantry system can also be positioned in the same manner as the vehicles 10, 10v in the aforementioned embodiments.

[0234] N-7. Other Implementation Methods 7: Vehicles 10 and 10v can be manufactured by combining multiple modules. A module refers to a unit composed of multiple components assembled according to the location and function of vehicles 10 and 10v. For example, the platform of vehicles 10 and 10v can be manufactured by combining a front module that constitutes the front portion of the platform, a central module that constitutes the center portion of the platform, and a rear module that constitutes the rear portion of the platform. It should be noted that the number of modules that constitute the platform is not limited to three and can also be two or fewer or four or more. In addition, components that constitute portions of vehicles 10 and 10v other than the platform can also be modularized in addition to or in place of the components that constitute the platform. Various modules can include any exterior components such as bumpers and grilles, and any interior components such as seats and consoles. Moreover, not limited to vehicles 10 and 10v, any type of mobile vehicle can be manufactured by combining multiple modules. Such modules can be manufactured, for example, by joining multiple components together using welding, fasteners, etc., or by integrally molding at least a portion of the components that constitute the module into a single component through casting. The molding technique of integrally molding a component, especially a relatively large component, is also called Gigacasting or Megacasting. For example, the front module, center module, and rear module mentioned above can be manufactured using Gigacasting.

[0235] N-8. Other Implementation Methods 8: Transporting vehicles 10, 10v using unmanned driving of the vehicles 10, 10v is also referred to as "self-propelled transport." Furthermore, the structure for implementing self-propelled transport is also referred to as a "vehicle remote-controlled autonomous transport system." Furthermore, the production method of vehicles 10, 10v using self-propelled transport is also referred to as "self-propelled production." In self-propelled production, for example, at least a portion of the transport of the vehicles 10, 10v is achieved by self-propelled transport in a factory manufacturing the vehicles 10, 10v.

[0236] The present disclosure is not limited to the above-mentioned embodiments and can be implemented in various structures within the scope of its main purpose. For example, the technical features of the embodiments corresponding to the technical features in the various methods described in the "Summary of the Invention" column can be appropriately replaced or combined in order to solve part or all of the above-mentioned problems or to achieve part or all of the above-mentioned effects. In addition, as long as the technical features are not described as essential technical features in this specification, they can be appropriately deleted. Description of Reference Numerals

[0237] 1. 1a to 1j, 1v…Information processing system, 5. 5a to 5j, 5v…Information processing device, 9…External camera, 10. 10v…Vehicle, 10e…Position measurement device, 11…Drive device, 12…Steering device, 13…Braking device, 14…Communication device, 15. 15v…Vehicle control device, 51…Communication unit, 52. 52a to 52j, 52v…CPU of information processing device, 53. 53a to 53j, 53v…Storage unit of information processing device, 54…Notification unit, 55…Input operation unit, 150. 150v…CPU of vehicle control device, 153. 153v…Action control unit, 154. 524. 524c to 524j…Control information acquisition unit, 155. 52 5, 525e, 525f…Calculation unit, 156…Signal generation unit, 157, 157v…Storage unit of the vehicle control device, 159…Input / output interface, 521, 521a…Change information acquisition unit, 522, 522a to 522j…Learning unit, 523, 523a…Status information acquisition unit, 526…Remote control unit, 527…Notification control unit, 528…Model acquisition unit, 529…Update unit, A1…Required area, A2…Unnecessary area, Bi…Background image, Br…Actual background image, Bv…Virtual background image, C…Center of gravity, Cb1 to Cb8…Stereo coordinate values, Ci…Composited image, Cr1 to Cr4…Rectangular coordinate values, D…First distance, ΔD…Observation error, Da …shape data, Db1…rectangular database, Db2…stereo database, Dc…third distance, Do…observed distance, Dp…estimated distance, Dt…second distance, h…height of the vehicle's measuring point from the road surface, H…height of the external camera from the road surface, Ic…sweep information, Im1…original image, Im2…corrected image, Im3…rotated image, Im4…processed image, Im1–Im4…captured images, Im5…first mask image, Im6…second mask image, Ip…reference path, Lc…correct contour label, Lr…correct region label, Md…machine learning model, Md1…first learning model, Md2, Md2a, Md2b…second learning model, Md3…third learning model , Md4…the fourth learning model, Md5…the fifth learning model, Md6…the sixth learning model, Md7…the seventh learning model, Md8…the eighth learning model, Ms…the mask area, P0…the basic coordinate point, P1…the first coordinate point, P2…the second coordinate point, P3…the image coordinate point, P5, P5a~P5j, P5v, P15, P15v…the program, Pa…the camera parameter, Pc…the shooting coordinate point, Pf…the fixed coordinate point, Pv…the vehicle coordinate point, R0…the basic bounding rectangle, R1…the first bounding rectangle, R2…the second bounding rectangle, RG…the shooting area, Rb…the vehicle bounding rectangle, Rc…the vehicle bounding rectangle, Ro…the passage, Rs…the road surface, Ta…the object area, Ti…the training image,V…movement vector, Vb1~Vb8, Vc1~Vc4…vertices, Vi…virtual vehicle image.

Claims

1. An information processing device for moving a mobile object capable of being moved by unmanned driving, the information processing device comprising: a change information acquisition unit that acquires change information related to a change in an appearance state in a target area, the target area being included in a photographing area of ​​an external camera disposed at a location different from that of the moving object, and including a path along which the moving object moves and a peripheral area of ​​the path; and At least one of a learning unit and a notification control unit, when the learning unit uses the change information to determine that the appearance state has changed, causes the machine learning model to output a control signal that specifies the action of the moving body moved by the unmanned driving and at least one of the parameters used when generating the control signal when a captured image obtained by capturing the object area and the moving body using the external camera is input, and learns using a plurality of training data that establish correspondences between training images of the object area and the moving body after the appearance state has changed and correct labels, and when the notification control unit uses the change information to determine that the appearance state has changed, causes the manager to be informed of learning requirement information for causing the machine learning model to learn.

2. An information processing device for moving a mobile body capable of moving by unmanned driving, The information processing device includes a learning unit that learns a machine learning model that outputs a control signal that specifies the action of the moving body that can be moved by the unmanned driving and at least one of the parameters used when generating the control signal when a captured image obtained by capturing an object area and the moving body using an external camera is input, wherein the object area is included in the capturing area of ​​the external camera set at a place different from the moving body, and includes a path for the moving body to move and a surrounding area of ​​the path. The object area is classified into a plurality of appearance states according to changes in appearance, The learning unit uses training images including the object area and the moving body classified into one of the appearance states and the correct labels to establish corresponding M training data, and training images including the object area and the moving body classified into the remaining other appearance states and the correct labels to establish corresponding N training data, and enables the machine learning model to learn for each of the appearance states, where M is an integer greater than 2 and N is an integer greater than 0 and less than M.

3. An information processing device for moving a mobile body capable of moving without human control, The information processing device includes a learning unit that learns a machine learning model that outputs a control signal that specifies the action of the moving body moved by the unmanned driving and at least one of the parameters used when generating the control signal when a captured image obtained by capturing a target area and the moving body using an external camera and an appearance state of the target area are input, wherein the target area is included in a capturing area of ​​the external camera set at a location different from that of the moving body, and includes a path for the moving body to move and a surrounding area of ​​the path. The object area is classified into a plurality of appearance states according to changes in appearance, The learning unit uses a plurality of training data corresponding to a training image including the object area and the moving object, a correct answer label, and accompanying information indicating the appearance state of the object area included in the training image to learn the machine learning model.

4. An information processing device for moving a mobile object capable of being moved by unmanned driving, the information processing device comprising: a state information acquisition unit that acquires state information related to an appearance state in a target area, the target area being included in a photographing area of ​​an external camera disposed at a location different from that of the moving object, and including a path along which the moving object moves and a peripheral area of ​​the path; and a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image into a machine learning model that has been learned to output at least one of a control signal that specifies the action of the moving body that moves by the unmanned driving and a parameter used when generating the control signal when the captured image obtained by capturing the target area and the moving body using the external camera is input, thereby acquiring at least one of the control signal and the parameter, When the appearance state determined by the state information is different from the appearance state during learning of the machine learning model, the control information acquisition unit inputs the captured image into the machine learning model that has been learned using a plurality of training data that have established corresponding training images of the object area and the moving body that include the appearance state determined by the state information and correct labels.

5. An information processing device for moving a mobile object capable of being moved by unmanned driving, the information processing device comprising: a state information acquisition unit that acquires state information related to an appearance state in a target area, the target area being included in a photographing area of ​​an external camera disposed at a location different from that of the moving object, and including a path along which the moving object moves and a peripheral area of ​​the path; a model acquisition unit that acquires a machine learning model that outputs at least one of a control signal that specifies the action of the moving body that moves by the unmanned driving and a parameter used when generating the control signal when a captured image obtained by capturing the target area and the moving body by using the external camera is input; and a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image into the machine learning model acquired by the model acquisition unit, The object area is classified into a plurality of appearance states according to changes in appearance, The machine learning model is prepared for each of the appearance states, The plurality of machine learning models prepared for each appearance state are pre-learned using M training data corresponding to training images and correct labels containing the object area and the moving body classified into one appearance state, and N training data corresponding to training images and correct labels containing the object area and the moving body classified into the remaining other appearance states, respectively, where M is an integer greater than 2 and N is an integer greater than 0 and less than M. The model acquisition unit acquires the machine learning model selected according to the appearance state from among the plurality of machine learning models prepared for each appearance state.

6. An information processing device for moving a mobile object capable of being moved by unmanned driving, the information processing device comprising: a state information acquisition unit that acquires state information related to an appearance state in a target area, the target area being included in a photographing area of ​​an external camera disposed at a location different from that of the moving object, and including a path along which the moving object moves and a peripheral area of ​​the path; and A control information acquisition unit, which obtains at least one of the control signal and the parameter by inputting a learned machine learning model that outputs a control signal that specifies the action of the moving body moved by the unmanned driving and at least one of the parameters used when generating the control signal when a captured image obtained by capturing the target area and the moving body with the use of the external camera and the appearance state determined by the state information are input, and then inputting the captured image and the appearance state determined by the state information to obtain at least one of the control signal and the parameter, The object area is classified into a plurality of appearance states according to changes in appearance, The machine learning model is pre-learned using a plurality of training data corresponding to a training image including the object region and the moving object, a correct answer label, and accompanying information indicating the appearance state of the object region included in the training image.

7. The information processing device according to any one of claims 1 to 6, wherein: The factors affecting the change in the appearance state are at least any one of dirt generated in the target area, unevenness generated in the target area, an object different from the moving body configured in a manner capable of moving in the target area, and the road surface color of the passage.

8. The information processing device according to any one of claims 1 to 6, wherein: The machine learning model learns during execution of the unmanned driving control.

9. The information processing device according to any one of claims 1 to 6, wherein: The training image is at least one of a synthetic image and the captured image, and the synthetic image is generated by synthesizing a background image of either an actual background image obtained by capturing the object area using the external camera and a virtual background image that virtually reproduces the object area to virtually reproduce a virtual moving body image of the moving body.

10. An information processing system for moving a mobile object capable of being moved by unmanned driving, the information processing system comprising: the mobile body; an external camera, which is disposed at a location different from the mobile object; and The information processing device according to any one of claims 1 to 6.

11. An information processing method for moving a mobile object capable of moving by unmanned driving, the information processing method comprising: a change information acquisition step of acquiring change information related to a change in an appearance state in a target area, the target area being included in a photographing area of ​​an external camera disposed at a location different from that of the moving body, and including a path along which the moving body moves and a peripheral area of ​​the path; and At least one of a learning process and a notification process, wherein the learning process, when judging that the appearance state has changed by using the change information, causes a machine learning model that outputs a control signal that specifies the action of the moving body moved by the unmanned driving and at least one of the parameters used when generating the control signal when inputting a captured image obtained by capturing the object area and the moving body by using the external camera to learn using a plurality of training data corresponding to training images and correct labels including the object area and the moving body after the appearance state has changed, and the notification process, when judging that the appearance state has changed by using the change information, causes a manager to be notified of learning requirement information for causing the machine learning model to learn.

12. An information processing method for moving a mobile object capable of being moved by unmanned driving, the information processing method comprising: a preparation step of preparing a plurality of training images including a target area and the moving body, wherein the target area includes a photographing area of ​​an external camera disposed at a location different from that of the moving body and includes a path along which the moving body moves and a peripheral area of ​​the path; and a learning step of learning a machine learning model that outputs a control signal that specifies the action of the moving body moved by the unmanned driving and at least one of a parameter used when generating the control signal when a captured image obtained by capturing the target area and the moving body using the external camera is input, The object area is classified into a plurality of appearance states according to changes in appearance, In the learning process, M corresponding training data are established using the training images containing the object area and the moving body classified into one of the appearance states and the correct labels, and N corresponding training data are established using the training images containing the object area and the moving body classified into the remaining other appearance states and the correct labels, and the machine learning model is learned for each of the appearance states, where M is an integer greater than 2 and N is an integer greater than 0 and less than M.

13. An information processing method for moving a mobile object capable of being moved by unmanned driving, the information processing method comprising: a preparation step of preparing a plurality of training images including a target area and the moving body, wherein the target area includes a photographing area of ​​an external camera disposed at a location different from that of the moving body and includes a path along which the moving body moves and a peripheral area of ​​the path; and a learning step of causing a machine learning model to learn when a captured image obtained by capturing the target area and the moving body with the external camera and an appearance state of the target area are input and outputs a control signal that specifies the action of the moving body moved by the unmanned driving and at least one of a parameter used when generating the control signal; The object area is classified into a plurality of appearance states according to changes in appearance, In the learning process, the machine learning model is learned by establishing corresponding multiple training data using the training image including the object area and the moving body, the correct answer label, and the accompanying information representing the appearance state of the object area included in the training image.

14. An information processing method for moving a mobile object capable of being moved by unmanned driving, the information processing method comprising: a state information acquisition step of acquiring state information related to an appearance state in a target area, the target area being included in a photographing area of ​​an external camera provided at a location different from that of the moving body, and including a path along which the moving body moves and a peripheral area of ​​the path; and a control information acquisition step of acquiring at least one of the control signal and the parameter by inputting the captured image into a machine learning model that has been learned to output at least one of a control signal that specifies the action of the moving body that moves by the unmanned driving and a parameter used when generating the control signal when the captured image obtained by capturing the target area and the moving body using the external camera is input, When the appearance state determined by the state information is different from the appearance state of the object area in the training image input when learning the machine learning model, in the control information acquisition step, the captured image is input into the machine learning model that has been learned using a plurality of training data that have established correspondences between the training images of the object area and the moving body including the appearance state determined by the state information and the correct labels.

15. An information processing method for moving a mobile object capable of being moved by unmanned driving, the information processing method comprising: a state information acquisition step of acquiring state information related to an appearance state in a target area, wherein the target area is included in a photographing area of ​​an external camera provided at a location different from that of the moving body, and includes a path along which the moving body moves and a peripheral area of ​​the path; a model acquisition step of acquiring a machine learning model that outputs at least one of a control signal that specifies the action of the moving body that moves by the unmanned driving and a parameter used when generating the control signal when a captured image obtained by capturing the target area and the moving body using the external camera is input; and a control information acquisition step of acquiring at least one of the control signal and the parameter by inputting the captured image into the machine learning model acquired in the model acquisition step, The object area is classified into a plurality of appearance states according to changes in appearance, The machine learning model is prepared for each of the appearance states, The plurality of machine learning models prepared for each appearance state are pre-learned using M training data corresponding to training images and correct labels containing the object area and the moving body classified into one appearance state, and N training data corresponding to training images and correct labels containing the object area and the moving body classified into the remaining other appearance states, respectively, where M is an integer greater than 2 and N is an integer greater than 0 and less than M. In the model acquisition step, the machine learning model selected according to the appearance state is acquired from the plurality of machine learning models prepared for each appearance state.

16. An information processing method for moving a mobile object capable of being moved by unmanned driving, the information processing method comprising: a state information acquisition step of acquiring state information related to an appearance state in a target area, the target area being included in a photographing area of ​​an external camera provided at a location different from that of the moving body, and including a path along which the moving body moves and a peripheral area of ​​the path; and a control information acquisition step, wherein a machine learning model that has been learned outputs at least one of a control signal that specifies the action of the moving body that moves by the unmanned driving and a parameter used when generating the control signal when a captured image obtained by capturing the target area and the moving body using the external camera and the appearance state determined by the state information are input, and the captured image and the appearance state determined by the state information are input to obtain at least one of the control signal and the parameter, The object area is classified into a plurality of appearance states according to changes in appearance, The machine learning model is pre-learned using a plurality of training data corresponding to a training image including the object region and the moving object, a correct answer label, and accompanying information indicating the appearance state of the object region included in the training image.

Images