Information processing device, information processing system, and information processing method

Abstract

To provide a technique for obtaining at least one of a control signal and a parameter for use in generating the control signal accurately even when an appearance state of a target region is changed.SOLUTION: An information processing device includes: a change information acquisition unit which acquires change information; and, when a change in an appearance state is determined using the change information, at least one of a training unit and a notification control unit, the training unit being configured to train a machine learning model using multiple pieces of training data in which a training image including a target region after the change in the appearance state and a moving object is associated with a ground truth label, the notification control unit being configured to notify an administrator of training request information for training the machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a technique is known in which a vehicle is made to travel automatically by remote control by monitoring the travel of the vehicle using a camera outside the vehicle (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-538619 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to move a moving body such as a vehicle in an unmanned driving manner, an image including the moving body may be input to a machine learning model to detect the moving body from the image and obtain parameters for generating a control signal that specifies the operation of the moving body. However, when dirt occurs on a passage through which the moving body moves, the appearance state of a target area including the passage and the surrounding area of ​​the passage may change. The present inventors have found that when the appearance state of the target area changes, if an image after the appearance state has changed is input to a machine learning model trained using an image before the appearance state changed, the accuracy of detecting the moving body may decrease. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, an information processing device is provided for use in moving a moving body that can be moved by unmanned driving, and the information processing device includes at least one of: a change information acquisition unit that acquires change information regarding a change in the appearance state of a target area included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage; a learning unit that, when it is determined that the appearance state has changed using the change information, learns a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal when an image captured by capturing the target area and the moving body by the external camera is input, using a plurality of teacher data that associates a training image including the target area and the moving body after the appearance state has changed with a correct answer label; and a notification control unit that notifies an administrator of learning request information for training the machine learning model. According to this aspect, when it is determined that the appearance state of the target area has changed using the change information, the information processing device can execute the following processes. In this case, the information processing device can train the machine learning model using multiple teacher data in which training images including the moving object and the target area after the appearance state has changed are associated with the correct label. In this way, when the appearance state of the target area changes, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used to generate the control signal. Also, according to this embodiment, when it is determined that the appearance state of the target area has changed using the change information, the information processing device can notify the administrator of learning request information for training the machine learning model. In this way, the information processing device can inform the administrator that it has been determined that the machine learning model needs to be trained. As a result, the information processing device can urge the administrator to take measures to train the machine learning model. (2) According to a second aspect of the present disclosure, there is provided an information processing device, the information processing device being used to move a moving body capable of moving by unmanned driving, comprising a learning unit that learns a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used to generate the control signal when a target area is included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage, and an imaging image obtained by imaging the moving body by the external camera is input, the target area is classified into a plurality of appearance states according to changes in appearance, and the learning unit uses M (M is an integer of 2 or more) pieces of teacher data in which a training image including the target area and the moving body classified into one of the appearance states is associated with a correct answer label, and N (N is an integer of 0 or more and less than M) pieces of teacher data in which a training image including the target area and the moving body classified into the remaining other appearance states is associated with a correct answer label to learn the machine learning model for each of the appearance states. According to this embodiment, the information processing device can train the machine learning model by using more training images including a moving object and a target area classified into one appearance state than training images including a moving object and a target area classified into the remaining other appearance states. This allows the information processing device to prepare a machine learning model suitable for acquiring at least one of a control signal and a parameter used when generating the control signal, using captured images including a moving object and a target area classified into one appearance state. (3) According to a third aspect of the present disclosure, there is provided an information processing device, the information processing device being used to move a moving body capable of moving by unmanned driving, comprising a learning unit that learns a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used when generating the control signal when a target area, which is included in an image capturing area of ​​an external camera installed at a location different from the moving body, including a passageway through which the moving body moves and a peripheral area of ​​the passageway, an image captured by capturing an image of the moving body by the external camera, and an appearance state of the target area are input, the target area is classified into a plurality of the appearance states according to a change in appearance, and the learning unit learns the machine learning model using a plurality of teacher data that associates a training image including the target area and the moving body, a correct answer label, and supplementary information indicating the appearance state of the target area included in the training image. According to this aspect, the information processing device can train a machine learning model using multiple pieces of teacher data in which a training image including a target region and a moving object, a correct answer label, and supplementary information indicating the appearance state of the target region included in the training image are associated with each other. This allows the information processing device to prepare a machine learning model that is trained in association with the appearance state of the target region. (4) According to a fourth aspect of the present disclosure, there is provided an information processing device. The information processing device used to move a moving body that can move by unmanned driving includes: a state information acquisition unit that acquires state information on an appearance state in a target area included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage; and a control information acquisition unit that acquires at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal by inputting the captured image to a trained machine learning model that outputs at least one of the control signal and the parameter when the appearance state specified by the state information is different from the appearance state when the machine learning model was trained, and the control information acquisition unit inputs the captured image to the machine learning model trained using multiple teacher data that associates a training image including the target area and the moving body in the appearance state specified by the state information with a correct answer label. According to this embodiment, when the appearance state of the target region specified by the state information is different from the appearance state of the target region in the training image used when the machine learning model is trained, the information processing device can execute the following process. In this case, the information processing device can obtain at least one of a control signal and a parameter used when generating the control signal by inputting the captured image to a machine learning model trained using a plurality of teacher data in which a training image including a target region of an appearance state specified by a moving body and the state information is associated with a correct answer label. In this way, when the appearance state of the target region specified by the state information is different from the appearance state of the target region in the training image used when the machine learning model is trained, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (5) According to a fifth aspect of the present disclosure, there is provided an information processing device. The information processing device used to move a moving body capable of moving by unmanned driving includes: a state information acquisition unit that acquires state information on the appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a model acquisition unit that acquires a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used in generating the control signal when an image acquired by imaging the target area and the moving body by the external camera is input; and a machine learning model that outputs at least one of the control signal and the parameter by inputting the image acquired by the model acquisition unit. and a control information acquisition unit that acquires at least one of the appearance states, the target region is classified into a plurality of the appearance states according to a change in appearance, the machine learning model is prepared for each of the appearance states, and each of the plurality of machine learning models prepared for each of the appearance states is trained in advance using M (M is an integer of 2 or more) pieces of teacher data in which a training image including the target region and the moving object classified into one of the appearance states is associated with a correct answer label, and N (N is an integer of 0 or more and less than M) pieces of teacher data in which a training image including the target region and the moving object classified into the remaining other appearance states is associated with a correct answer label, and the model acquisition unit acquires the machine learning model selected according to the appearance state from among the plurality of machine learning models prepared for each of the appearance states. According to this embodiment, the information processing device can acquire a machine learning model selected according to the appearance state specified by the state information from among the plurality of machine learning models prepared for each appearance state of the target region. Then, the information processing device can acquire at least one of a control signal and a parameter used when generating a control signal by inputting a captured image into a machine learning model corresponding to the appearance state of the target region included in the captured image. In this way, it is possible to prevent a decrease in accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (6) According to a sixth aspect of the present disclosure, there is provided an information processing device. The information processing device used to move a moving body capable of moving by unmanned driving includes: a state information acquisition unit that acquires state information on the appearance state of a target area included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage; and a control information acquisition unit that acquires at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used when generating the control signal when an image capture image acquired by capturing the target area and the moving body with the external camera and the appearance state specified by the state information are input to a trained machine learning model that outputs at least one of the control signal and the parameter by inputting the captured image and the appearance state specified by the state information, the target area is classified into a plurality of the appearance states according to a change in appearance, and the machine learning model has been trained in advance using a plurality of teacher data that associates a training image including the target area and the moving body with a correct answer label and supplementary information indicating the appearance state of the target area included in the training image. According to this embodiment, the information processing device inputs the captured image and the appearance state of the target region specified by the state information to a machine learning model that has been trained in advance using multiple teacher data in which a training image including a target region and a moving object, a correct answer label, and supplementary information indicating the appearance state of the target region included in the training image are associated with each other. This allows the information processing device to obtain at least one of a control signal and a parameter used when generating the control signal. In this way, it is possible to suppress a decrease in accuracy of the control signal output from the machine learning model and the parameter used when generating the control signal. (7) In the above embodiment, the element affecting the change in the appearance state may be at least one of dirt occurring in the target area, unevenness occurring in the target area, an object other than the moving body that is movably arranged in the target area, and the road surface color of the passage. According to this embodiment, when the element affecting the change in the appearance state of the target area is at least one of dirt occurring in the target area, unevenness occurring in the target area, an object that is movably arranged in the target area, and the road surface color of the passage, the following can be realized. In this case, it is possible to suppress a decrease in accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (8) In the above aspect, the machine learning model may be trained during a period in which control of the unmanned driving is being executed. According to this aspect, the information processing device can train the machine learning model during a period in which the moving body is executing control of the unmanned driving. (9) In the above embodiment, the training image may be at least one of a composite image generated by synthesizing a virtual moving body image that virtually reproduces the moving body with one of a real background image acquired by capturing an image of the target area using the external camera and a virtual background image that virtually reproduces the target area, and the captured image. According to this embodiment, the information processing device can use a composite image generated by synthesizing a virtual moving body image with one of a real background image and a virtual background image as a training image used for learning a machine learning model. (10) According to a seventh aspect of the present disclosure, an information processing system is provided. The information processing system used to move a moving body capable of moving by unmanned driving includes the moving body, an external camera installed at a location different from the moving body, and the information processing device described in the above aspect. According to this aspect, the information processing system can learn a machine learning model that outputs at least one of a control signal and a parameter used when generating the control signal, or can move the moving body by unmanned driving using the learned machine learning model. (11) According to an eighth aspect of the present disclosure, an information processing method is provided. The information processing method used to move a moving body that can move by unmanned driving includes at least one of a change information acquisition step of acquiring change information regarding a change in an appearance state in a target area included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage; a learning step of learning a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal when an image captured by capturing the target area and the moving body by the external camera is input, using a plurality of teacher data in which a training image including the target area and the moving body after the appearance state has changed and a correct answer label are associated with each other; and a notification step of notifying an administrator of learning request information for training the machine learning model. According to this aspect, when it is determined that the appearance state of the target area has changed using the change information, the following processing can be executed. In this case, the machine learning model can be trained using a plurality of teacher data in which training images including the moving object and the target area after the appearance state has changed are associated with the correct label. In this way, when the appearance state of the target area changes, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used to generate the control signal. In addition, according to this embodiment, when it is determined that the appearance state of the target area has changed using the change information, it is possible to notify the administrator of learning request information for training the machine learning model. In this way, it is possible to inform the administrator that it has been determined that the machine learning model needs to be trained. This makes it possible to prompt the administrator to take measures to train the machine learning model. (12) According to a ninth aspect of the present disclosure, there is provided an information processing method. An information processing method used to move a moving body that can move by unmanned driving includes a preparation step of preparing a target area included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a surrounding area of ​​the passageway, and a plurality of training images including the moving body; and a learning step of training a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal when an image capture image obtained by capturing an image of the target area and the moving body by the external camera is input, the target area being classified into a plurality of appearance states according to changes in appearance, and in the learning step, the machine learning model is trained for each of the appearance states using M (M is an integer of 2 or more) pieces of teacher data that associate the training image including the target area and the moving body classified into one of the appearance states with a correct answer label, and N (N is an integer of 0 or more and less than M) pieces of teacher data that associate the training image including the target area and the moving body classified into the remaining other appearance states with a correct answer label. According to this embodiment, the machine learning model can be trained using more training images including a moving object and a target area classified into one appearance state than training images including a moving object and a target area classified into the remaining other appearance states. This makes it possible to prepare a machine learning model suitable for obtaining at least one of a control signal and a parameter used when generating a control signal, using captured images including a moving object and a target area classified into one appearance state. (13) According to a tenth aspect of the present disclosure, there is provided an information processing method, which is used to move a moving body that can move by unmanned driving, and includes a preparation step of preparing a plurality of training images including the moving body, the target area being included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passageway through which the moving body moves and a peripheral area of ​​the passageway, and the moving body, and a learning step of training a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal when an image capture image obtained by capturing an image of the target area and the moving body by the external camera and an appearance state of the target area are input, the target area is classified into a plurality of the appearance states according to a change in appearance, and in the learning step, the machine learning model is trained using a plurality of teacher data that associates a training image including the target area and the moving body, a correct answer label, and supplementary information indicating the appearance state of the target area included in the training image. According to this embodiment, a machine learning model can be trained using multiple pieces of teacher data in which the training images including the target region and the moving object, the correct answer labels, and the supplementary information indicating the appearance state of the target region included in the training images are associated with each other. This makes it possible to prepare a machine learning model that is trained in association with the appearance state of the target region. (14) According to an eleventh aspect of the present disclosure, there is provided an information processing method, which is used to move a moving body that can move by unmanned driving, and includes: a state information acquiring step of acquiring state information on an appearance state of a target area included in an image capturing area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage; and a control information acquiring step of acquiring at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal by inputting the captured image to a trained machine learning model that outputs at least one of the control signal and the parameter when the appearance state specified by the state information is different from the appearance state of the target area in a training image input when the machine learning model is trained, in the control information acquiring step, the captured image is input to the machine learning model trained using multiple teacher data that associates a training image including the target area and the moving body in the appearance state specified by the state information with a correct answer label. According to this embodiment, when the appearance state of the target area specified by the state information is different from the appearance state of the target area in the training image used when the machine learning model was trained, the following processing can be executed. In this case, at least one of the control signal and the parameters used when generating the control signal can be obtained by inputting the captured image to the machine learning model trained using a plurality of teacher data in which the training image including the target area of ​​the appearance state specified by the moving object and the state information is associated with the correct answer label. In this way, when the appearance state of the target area specified by the state information is different from the appearance state of the target area in the training image used when the machine learning model was trained, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (15) According to a twelfth aspect of the present disclosure, there is provided an information processing method, which is used to move a moving body capable of moving by unmanned driving, includes a state information acquisition step of acquiring state information on an appearance state of a target area included in an image capture area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage, a model acquisition step of acquiring a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used in generating the control signal when an image capture image of the target area and the moving body is inputted by the external camera, and a process of inputting the image capture image into the machine learning model acquired in the model acquisition step to acquire at least one of the control signal and the parameter. and a control information acquisition step of acquiring at least one of the appearance states, the target region being classified into a plurality of the appearance states according to a change in appearance, the machine learning model being prepared for each of the appearance states, and the plurality of machine learning models prepared for each of the appearance states being trained in advance using M (M is an integer of 2 or more) pieces of teacher data in which a training image including the target region and the moving object classified into one of the appearance states is associated with a correct answer label, and N (N is an integer of 0 or more and less than M) pieces of teacher data in which a training image including the target region and the moving object classified into the remaining other appearance states is associated with a correct answer label, and the model acquisition step acquires the machine learning model selected according to the appearance state from among the plurality of machine learning models prepared for each of the appearance states. According to this embodiment, it is possible to acquire a machine learning model selected according to the appearance state specified by the state information from among the plurality of machine learning models prepared for each appearance state of the target region. Then, by inputting the captured image into a machine learning model corresponding to the appearance state of the target region included in the captured image, it is possible to acquire at least one of a control signal and a parameter used when generating the control signal. In this way, it is possible to prevent a decrease in accuracy of the control signal output from the machine learning model and the parameters used when generating the control signal. (16) According to a thirteenth aspect of the present disclosure, there is provided an information processing method, which is used to move a moving body capable of moving by unmanned driving, and includes: a state information acquiring step of acquiring state information on an appearance state of a target area included in an image capturing area of ​​an external camera installed at a location different from the moving body, the target area including a passage through which the moving body moves and a peripheral area of ​​the passage; and a control information acquiring step of acquiring at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used when generating the control signal by inputting the captured image and the appearance state specified by the state information to a trained machine learning model that outputs at least one of the control signal and the parameter when an image captured by the external camera and the appearance state specified by the state information are input, the target area is classified into a plurality of the appearance states according to a change in appearance, and the machine learning model is trained in advance using a plurality of teacher data that associates a training image including the target area and the moving body with a correct answer label and supplementary information indicating the appearance state of the target area included in the training image. According to this embodiment, the captured image and the appearance state of the target area specified by the state information are input to a machine learning model that has been trained in advance using multiple teacher data that correspond to a training image including a target area and a moving object, a correct answer label, and supplementary information indicating the appearance state of the target area included in the training image. This makes it possible to obtain at least one of a control signal and a parameter used when generating the control signal. In this way, it is possible to suppress a decrease in the accuracy of the control signal output from the machine learning model and the parameter used when generating the control signal. The present disclosure can be realized in various forms other than the above-mentioned information processing device, information processing system, and information processing method. For example, the present disclosure can be realized in the form of an information processing device, an information processing system, a manufacturing method for a moving body, an information processing device, an information processing system, a control method for a moving body, a computer program for realizing the control method, a non-transitory recording medium on which the computer program is recorded, etc. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the configuration of an information processing system according to a first embodiment. [Diagram 2] 3 is a flowchart showing a vehicle driving control method in the first embodiment. [Diagram 3] 5 is a flowchart showing a re-learning method of the first learning model in the first embodiment. [Figure 4] 5 is a flowchart showing a method for acquiring shape data in the first embodiment. [Diagram 5] 4 is a flowchart showing an example of a vehicle position calculation method. [Figure 6] 6A to 6C are schematic diagrams showing examples of various images obtained when the method shown in FIG. 5 is executed. [Figure 7] FIG. 11 is a diagram for explaining details of a coordinate point calculation step. [Figure 8] FIG. 4 is a diagram for explaining a method of calculating a base coordinate point. [Figure 9] FIG. 1 is a diagram for explaining details of a position conversion process. [Figure 10] FIG. 2 for explaining the details of the position conversion process. [Figure 11] FIG. 13 is a diagram showing the configuration of an information processing system according to a second embodiment. [Figure 12] FIG. 13 is a diagram showing the configuration of an information processing system according to a third embodiment. [Figure 13] 13 is a flowchart showing a re-learning method for the first learning model in the third embodiment. [Figure 14] FIG. 13 is a diagram showing the configuration of an information processing system according to a fourth embodiment. [Figure 15] 11 is a flowchart showing a learning method of the second learning model. [Figure 16] 11 is a flowchart showing a method for acquiring shape data using a second learning model. [Figure 17] FIG. 13 is a diagram showing the configuration of an information processing system according to a fifth embodiment. [Figure 18] 11 is a flowchart showing a learning method of the third learning model. [Figure 19] 13 is a flowchart showing a method of acquiring shape data using the third learning model. [Figure 20] FIG. 13 is a diagram showing an example of a method for generating training images. [Figure 21] FIG. 13 is a diagram showing the configuration of an information processing system according to a seventh embodiment. [Figure 22] FIG. 4 is a diagram for explaining rectangular coordinate values. [Diagram 23] 13 is a flowchart showing a re-learning method for the fourth learning model. [Figure 24] 13 is a flowchart showing a method for obtaining rectangular coordinate values ​​using the fourth learning model. [Diagram 25] FIG. 23 is a diagram showing the configuration of an information processing system according to an eighth embodiment. [Figure 26] FIG. 4 is a diagram for explaining three-dimensional coordinate values. [Figure 27] 13 is a flowchart showing a method for re-learning the fifth learning model. [Figure 28] 13 is a flowchart showing a method for obtaining three-dimensional coordinate values ​​using the fifth learning model. [Figure 29] FIG. 13 is a diagram showing the configuration of an information processing system according to a ninth embodiment. [Diagram 30] 13 is a flowchart showing a re-learning method for the sixth learning model. [Diagram 31] 13 is a flowchart showing a method for acquiring vehicle position information using a sixth learning model. [Diagram 32] FIG. 23 is a diagram showing the configuration of an information processing system according to a tenth embodiment. [Diagram 33] 13 is a flowchart showing a method for re-learning the seventh learning model. [Diagram 34]A flowchart showing a method of acquiring a driving control signal using the seventh learning model. [Diagram 35] FIG. 23 is a diagram showing the configuration of an information processing system according to an eleventh embodiment. [Diagram 36] 1 is a flowchart showing a transfer learning method for a first learning model. [Figure 37] 23 is a flowchart showing a method for acquiring shape data in the eleventh embodiment. [Figure 38] FIG. 23 is a diagram showing the configuration of an information processing system according to a twelfth embodiment. [Figure 39] 23 is a flowchart showing a learning method of a learning model in the 12th embodiment. [Diagram 40] 23 is a flowchart showing a method for acquiring shape data in the twelfth embodiment. [Diagram 41] FIG. 23 is a diagram showing the configuration of an information processing system according to a thirteenth embodiment. [Diagram 42] 23 is a flowchart showing a vehicle driving control method in a thirteenth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: A-1. Information processing system configuration: 1 is a diagram showing a configuration of an information processing system 1 in a first embodiment. The information processing system 1 is a system for moving a moving body without the driving operation of a passenger on the moving body. The information processing system 1 includes one or more moving bodies, one or more external cameras 9 installed in a location different from the moving body, and an information processing device 5.

[0009] The term "mobile body" refers to an object that can move, such as a vehicle 10 or an electric vertical take-off and landing aircraft (a so-called flying car). In this embodiment, the mobile body is a vehicle 10. The vehicle 10 may be a vehicle 10 that runs on wheels or a vehicle 10 that runs on caterpillar tracks, such as a passenger car, a truck, a bus, a two-wheeled vehicle, a four-wheeled vehicle, a tank, or a construction vehicle. The vehicle 10 includes an electric vehicle (BEV: Battery Electric Vehicle), a gasoline-powered vehicle, a hybrid vehicle, and a fuel cell vehicle. When the mobile body is other than the vehicle 10, the expressions "vehicle" and "car" in this disclosure can be appropriately replaced with "mobile body", and the expression "running" can be appropriately replaced with "movement".

[0010] The vehicle 10 includes a driving device 11, a steering device 12, a braking device 13, a communication device 14, and a vehicle control device 15. The driving device 11 accelerates the vehicle 10. The driving device 11 includes wheels, a driving force source that rotates the wheels, and a power supply device that supplies power to the driving force source. For example, if the vehicle 10 is an electric vehicle, the driving force source is a driving motor, and the power supply device is a battery such as a lithium-ion battery. The steering device 12 changes the traveling direction of the vehicle 10. The braking device 13 decelerates the vehicle 10. The communication device 14 communicates with an external device using wireless communication or the like. The external device is a device such as an information processing device 5 provided in a location different from the vehicle 10, and another vehicle 10. The communication device 14 is, for example, a wireless communication device.

[0011] The vehicle control device 15 controls the operation of the vehicle 10. The vehicle control device 15 includes a CPU 150, a storage unit 157, and an input / output interface 159. In the vehicle control device 15, the CPU 150, the storage unit 157, and the input / output interface 159 are connected to each other via a bus or the like so as to be able to communicate bidirectionally. The input / output interface 159 is used to communicate with various devices mounted on the vehicle 10. The storage unit 157 of the vehicle control device 15 stores various information including various programs P15 that control the operation of the vehicle control device 15. The storage unit 157 of the vehicle control device 15 includes, for example, a RAM, a ROM, and a hard disk drive (HDD). The CPU 150 of the vehicle control device 15 functions as an operation control unit 153 by developing the various programs P15 stored in the storage unit 157. The operation control unit 153 executes driving control of the vehicle 10 by controlling the operation of an actuator that changes the acceleration / deceleration and steering angle of the vehicle 10. "Driving control" refers to various controls for driving actuators that perform the three functions of the vehicle 10, "running," "turning," and "stopping," such as adjusting the acceleration, speed, and steering angle of the vehicle 10. In this embodiment, the actuators include an actuator of the drive device 11, an actuator of the steering device 12, and an actuator of the braking device 13. Note that at least a part of the functions of the CPU 150 of the vehicle control device 15 may be realized by a hardware circuit.

[0012] The vehicle 10 is configured to be capable of traveling by unmanned driving. "Unmanned driving" means driving without the driving operation of a passenger. Driving operation means at least one of the operations of "running", "turning" and "stopping" of the vehicle 10. Unmanned driving is realized by automatic or manual remote control using a device located outside the vehicle 10, or by autonomous control of the vehicle 10. The vehicle 10 traveling by unmanned driving may have a passenger who does not perform driving operation on board. The passenger who does not perform driving operation includes, for example, a person who simply sits in the seat of the vehicle 10, or a person who performs work other than driving operation, such as assembly, inspection, and operation of switches, while riding on the vehicle 10. Note that driving by a passenger who performs driving operation is sometimes called "manned driving".

[0013] In this specification, "remote control" includes "full remote control" in which all of the operations of the vehicle 10 are completely determined from outside the vehicle 10, and "partial remote control" in which some of the operations of the vehicle 10 are determined from outside the vehicle 10. In addition, "autonomous control" includes "full autonomous control" in which the vehicle 10 autonomously controls its own operations without receiving any information from devices external to the vehicle 10, and "partial autonomous control" in which the vehicle 10 autonomously controls its own operations using information received from devices external to the vehicle 10.

[0014] In this embodiment, the information processing system 1 is used in a factory that manufactures a vehicle 10. The reference coordinate system of the factory is a global coordinate system. That is, any position in the factory is expressed by X, Y, and Z coordinates in the global coordinate system. The factory includes a first location and a second location. The first location and the second location are connected by an aisle Ro along which the vehicle 10 can travel. In the factory, a plurality of external cameras 9 are installed along the aisle Ro. The external camera 9 captures an image of an imaging area RG including a target area and the vehicle 10 from outside the vehicle 10. The target area is an area including the aisle Ro along which the vehicle 10 travels and the surrounding area of ​​the aisle Ro. The position of each external camera 9 in the factory is adjusted in advance. The vehicle 10 moves from the first location to the second location through the aisle Ro by unmanned driving. Note that in other embodiments, the vehicle 10 may travel by unmanned driving in a location other than the factory.

[0015] The information processing device 5 is used to drive the vehicle 10 by unmanned driving. The information processing device 5 is, for example, a server installed at a location different from the vehicle 10. The information processing device 5 learns the machine learning model Md, or acquires and uses the learned machine learning model Md to drive the vehicle 10 by unmanned driving. When a captured image or the like is input, the machine learning model Md outputs at least one of a driving control signal and a signal generation parameter. The captured image is acquired by capturing an image of the target area and the vehicle 10 by the external camera 9. The captured image is either an original image as raw data output from the external camera 9 or a processed image obtained by processing the original image. The driving control signal is a control signal that specifies the operation of the vehicle 10 in order to drive the vehicle 10 by unmanned driving. In this embodiment, the driving control signal includes the acceleration and steering angle of the vehicle 10 as control parameters. In other embodiments, the driving control signal may include the speed of the vehicle 10 as a control parameter instead of or in addition to the acceleration of the vehicle 10. The signal generation parameters are parameters used when generating a driving control signal. The signal generation parameters are, for example, vehicle position information, coordinate values ​​used to calculate at least one of the position and orientation of the vehicle 10, or shape data used to calculate at least one of the position and orientation of the vehicle 10. The shape data is data indicating the external shape of the vehicle 10. The shape data is generated by detecting the vehicle 10 from the captured image. The shape data is, for example, a mask image in which a mask area is added to the captured image by masking an area representing the vehicle 10 among the areas constituting the captured image. Each area constituting the captured image is, for example, one pixel constituting the captured image.

[0016] In this embodiment, a first learning model Md1 is used as the machine learning model Md. The first learning model Md1 outputs shape data as a parameter for signal generation. Therefore, the information processing device 5 acquires the shape data by inputting a captured image to the first learning model Md1. Then, the information processing device 5 calculates the position and orientation of the vehicle 10 using the shape data. Then, the information processing device 5 generates a driving control signal using vehicle position information and the like, and transmits the signal to the vehicle 10. As a result, the information processing device 5 drives the vehicle 10 by remote control.

[0017] Here, the appearance state of the target area may change over time, for example, when dirt occurs on the passage Ro on which the vehicle 10 travels. As a result, the target area is classified into a plurality of appearance states according to the change in appearance. When the appearance state of the target area changes, if a captured image including the vehicle 10 and the target area after the appearance state has changed is input to the first learning model Md1 that has been trained using a training image including the vehicle 10 and the target area before the appearance state has changed, there is a risk that the detection accuracy when detecting the vehicle 10 from the captured image will decrease. Therefore, when it is determined that the appearance state of the target area has changed, the information processing device 5 re-learns the first learning model Md1 and acquires shape data using the first learning model Md1 after re-learning.

[0018] In the following, a description will be given on the assumption that the factor affecting the change in the appearance state in the target area is dirt that occurs in the target area. For example, at the time when the vehicle 10 starts traveling by unmanned driving, the pre-change state before the appearance state of the target area changes is a state in which no dirt occurs in the target area. The post-change state after the appearance state of the target area changes is a state in which dirt occurs in the target area. In this case, at the time when it is determined that dirt occurs in the target area during the period when the vehicle 10 is performing the control of unmanned driving, and the re-learned first learning model Md1 is stored in the storage unit 53 of the information processing device 5, the pre-change state and the post-change state are as follows. In this case, the pre-change state is a state in which dirt occurs in the target area. The post-change state is a state in which no dirt occurs in the target area as a result of the dirt in the target area being removed by cleaning or the like.

[0019] The information processing device 5 includes a communication unit 51, a CPU 52, and a storage unit 53. In the information processing device 5, the communication unit 51, the CPU 52, and the storage unit 53 are connected to each other via a bus or the like so as to be able to communicate bidirectionally.

[0020] The communication unit 51 of the information processing device 5 communicably connects the information processing device 5 to devices other than the information processing device 5. The communication unit 51 of the information processing device 5 is, for example, a wireless communication device.

[0021] The storage unit 53 of the information processing device 5 stores various information including various programs P5 that control the operation of the information processing device 5, a reference path Ip, and a first learning model Md1. The storage unit 53 of the information processing device 5 includes, for example, a RAM, a ROM, and a hard disk drive (HDD).

[0022] The first learning model Md1 is a trained machine learning model Md that utilizes artificial intelligence to detect the external shape of the vehicle 10 from a captured image and output shape data. The first learning model Md1 is trained in advance by supervised learning using an initial learning dataset. The initial learning dataset includes a plurality of teacher data in which a training image including the vehicle 10 and a target area before the change in the external state is associated with a correct answer label according to the output of the machine learning model Md. In this embodiment, the correct answer label is a region correct answer label. The region correct answer label is a correct answer label indicating whether each area in the training image is an area indicating the vehicle 10 or an area indicating something other than the vehicle 10. For example, a convolutional neural network (hereinafter, CNN) that realizes either semantic segmentation or instance segmentation can be used for the first learning model Md1. During training of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce an error between the output result by the first learning model Md1 and the region correct answer label.

[0023] The CPU 52 of the information processing device 5 functions as a change information acquisition unit 521, a learning unit 522, a status information acquisition unit 523, a control information acquisition unit 524, a calculation unit 525, and a remote control unit 526 by deploying various programs P5 stored in the memory unit 53.

[0024] The change information acquisition unit 521 acquires change information. The change information is information about a change in the appearance state in the target area. Specifically, the change information is information for determining whether or not the appearance state of the target area has changed from the appearance state of the target area in the training image used when the trained first learning model Md1 has been trained. When the factor affecting the change in the appearance state of the target area is dirt that occurs in the target area, the change information is, for example, the following information. In this case, the change information is information for determining whether or not the appearance state of the target area has changed due to dirt that did not exist when the first learning model Md1 stored in the storage unit 53 of the information processing device 5 is trained. In this embodiment, the change information acquisition unit 521 acquires a captured image as the change information.

[0025] When the change information indicates a change in the appearance state of the target area, the learning unit 522 executes re-learning as learning for the first learning model Md1. Re-learning is an example of additional learning. In this embodiment, the learning unit 522 re-learns the first learning model Md1, which has been learned using an initial learning dataset, using a re-learning dataset. In this embodiment, the re-learning dataset includes multiple teacher data sets that associate training images including the vehicle 10 and the target area after the appearance state has changed with area correct labels.

[0026] For example, when the vehicle 10 starts traveling by unmanned driving, if there is no dirt in the target area, the first learning model Md1 stored in the storage unit 53 of the information processing device 5 is previously trained by supervised learning using the following initial learning dataset. In this case, the initial learning dataset includes a plurality of teacher data in which a training image including the vehicle 10 and the target area where no dirt is generated is associated with a region correct answer label. When the learning unit 522 determines that the appearance state of the target area has changed during the period in which the control of the unmanned driving is being executed, that is, when it determines that there is dirt in the target area, the learning unit 522 executes the following process. In this case, the learning unit 522 re-trains the first learning model Md1 trained using the initial learning dataset during the period in which the control of the unmanned driving is being executed, using the following re-learning dataset. In this case, the re-learning dataset includes a plurality of teacher data in which a training image including the vehicle 10 and the target area where dirt is generated is associated with a region correct answer label.

[0027] In this embodiment, the learning unit 522 uses the change information to determine whether the appearance state of the target area has changed since the first learning model Md1 was learned. In detail, the learning unit 522 uses the captured image as change information to determine whether the appearance state of the target area has changed since the first learning model Md1 was learned. For example, the learning unit 522 detects an element that affects the change in the appearance state of the target area from the captured image to determine whether the appearance state of the target area has changed since the first learning model Md1 was learned. Specifically, when an element that affects the change in the appearance state of the target area is detected from the captured image, the learning unit 522 determines that the appearance state of the target area has changed since the first learning model Md1 was learned. On the other hand, when an element that affects the change in the appearance state of the target area is not detected from the captured image, the learning unit 522 determines that the appearance state of the target area has not changed since the first learning model Md1 was learned. More specifically, when the pre-change state is a state in which no dirt has occurred in the target area and the post-change state is a state in which dirt has occurred in the target area, the learning unit 522 makes the following judgment. In this case, when no dirt is detected in the captured image, the learning unit 522 judges that no dirt has occurred in the target area and that the appearance state of the target area has not changed. On the other hand, when dirt is detected in the captured image, the learning unit 522 judges that dirt has occurred in the target area and that the appearance state of the target area has changed.

[0028] The learning unit 522 may use pixel values ​​of the captured image to determine whether the appearance state of the target region has changed since the first learning model Md1 was learned. For example, when the difference between the sum of the luminance values ​​of each pixel of the original image and a predetermined luminance value is equal to or greater than a predetermined threshold, the learning unit 522 determines that the appearance state of the target region has changed since the first learning model Md1 was learned. On the other hand, when the difference between the sum of the luminance values ​​of each pixel of the original image and a predetermined luminance value is less than a threshold, the learning unit 522 determines that the appearance state of the target region has not changed since the first learning model Md1 was learned.

[0029] The state information acquisition unit 523 acquires state information. The state information is information related to the appearance state of the target area. In this embodiment, the state information is information for determining whether the appearance state of the target area is a state in which the target area is not dirty or a state in which the target area is dirty. In this embodiment, the state information acquisition unit 523 acquires a captured image as the state information.

[0030] The control information acquisition unit 524 uses the state information to specify the appearance state of the target area, and inputs the captured image to the first learning model Md1 according to the appearance state of the target area to acquire at least one of a driving control signal and a signal generation parameter. When the appearance state of the target area specified by the state information is different from the appearance state of the target area in the training image used when the first learning model Md1 is learned, the control information acquisition unit 524 executes the following process. In this case, the control information acquisition unit 524 inputs the captured image to the first learning model Md1 after re-learning, which is re-learned using a plurality of teacher data in which the vehicle 10 and the training image including the target area of ​​the appearance state specified by the state information are associated with the area correct label. On the other hand, when the appearance state of the target area specified by the state information is the same as the appearance state of the target area in the training image used when the first learning model Md1 stored in the storage unit 53 is learned, the control information acquisition unit 524 executes the following process. In this case, the control information acquisition unit 524 inputs the captured image to the first learning model Md1, which has not been re-learned using multiple teacher data that associates a training image including a target area of ​​the vehicle 10 and the appearance state identified by the state information with an area correct label.

[0031] The calculation unit 525 acquires vehicle position information by calculating the position and orientation of the vehicle 10. The vehicle position information is position information that is the basis for generating a driving control signal. In this embodiment, the vehicle position information includes the position and orientation of the vehicle 10 in the reference coordinate system of the factory.

[0032] The position of the vehicle 10 is, for example, the position of a positioning point 10e that is preset for a specific part of the vehicle 10. In this embodiment, the calculation unit 525 calculates the coordinates of the positioning point 10e of the vehicle 10 in the coordinate system of the captured image, i.e., the local coordinate system, using shape data acquired by inputting the captured image to the first learning model Md1, and converts the calculated coordinates into coordinates in the global coordinate system to calculate the position of the vehicle 10.

[0033] The orientation of the vehicle 10 is, for example, a direction represented by a vector directed from the rear side to the front side of the vehicle 10 along the longitudinal axis passing through the center of gravity of the vehicle 10. In this embodiment, the calculation unit 525 calculates the orientation of the vehicle 10 by using, for example, an optical flow method, based on the direction of a movement vector of the vehicle 10 calculated from a position change of a feature point of the vehicle 10 between frames of captured images.

[0034] The remote control unit 526 generates a travel control signal and transmits it to the vehicle 10, thereby remotely controlling the travel operation of the vehicle 10. In order to generate the travel control signal, the remote control unit 526 first determines a target position to which the vehicle 10 should next head. In this embodiment, the target position is represented by X, Y, and Z coordinates in the global coordinate system of the factory. As described above, the storage unit 53 of the information processing device 5 prestores a reference route Ip, which is a route along which the vehicle 10 should next head. The remote control unit 526 uses the vehicle position information and the reference route Ip to determine a target position to which the vehicle 10 should next head. The reference route Ip is a route along which the vehicle 10 should next head. The route is represented by a node indicating a departure point, a node indicating a passing point, a node indicating a destination, and links connecting the nodes. The remote control unit 526 determines a target position on the reference route Ip ahead of the current location of the vehicle 10. The remote control unit 526 generates a travel control signal for driving the vehicle 10 toward the determined target position. As described above, in this embodiment, the traveling control signal includes the acceleration and steering angle of the vehicle 10 as parameters. The remote control unit 526 calculates the traveling speed of the vehicle 10 from the transition of the position of the vehicle 10, and compares the calculated traveling speed with a predetermined target speed of the vehicle 10. When the traveling speed is lower than the target speed as a whole, the remote control unit 526 determines the acceleration so that the vehicle 10 accelerates, and when the traveling speed is higher than the target speed, the remote control unit 526 determines the acceleration so that the vehicle 10 decelerates. In addition, when the vehicle 10 is located on the reference route Ip, the remote control unit 526 determines the steering angle and acceleration so that the vehicle 10 does not deviate from the reference route Ip, and when the vehicle 10 is not located on the reference route Ip, in other words, when the vehicle 10 deviates from the reference route Ip, the remote control unit 526 determines the steering angle and acceleration so that the vehicle 10 returns to the reference route Ip. In this way, the remote control unit 526 generates a traveling control signal including the acceleration and steering angle of the vehicle 10 as parameters. Then, the remote control unit 526 transmits the generated driving control signal to the vehicle 10. Note that at least a part of the functions of the CPU 52 of the information processing device 5 may be realized by a hardware circuit.

[0035] A-2. Vehicle operation control method: Fig. 2 is a flowchart showing a method for driving the vehicle 10 in the information processing system 1 of the first embodiment. The flow shown in Fig. 2 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts driving in an unmanned manner, for example.

[0036] The calculation unit 525 of the information processing device 5 acquires vehicle position information using a detection result output from the external camera 9 located outside the vehicle 10 (step S110). The remote control unit 526 determines a target position to which the vehicle 10 should next head (step S120). The remote control unit 526 determines a target position to which the vehicle 10 should next head using the vehicle position information and the reference route Ip. The remote control unit 526 determines a target position on the reference route Ip that is ahead of the current location of the vehicle 10.

[0037] The remote control unit 526 generates a travel control signal for driving the vehicle 10 toward the determined target position (step S130). The remote control unit 526 calculates the travel speed of the vehicle 10 from the transition of the position of the vehicle 10, and compares the calculated travel speed with a predetermined target speed of the vehicle 10. When the travel speed is lower than the target speed, the remote control unit 526 determines the acceleration so that the vehicle 10 accelerates, and when the travel speed is higher than the target speed, the remote control unit 526 determines the acceleration so that the vehicle 10 decelerates. When the vehicle 10 is located on the reference route Ip, the remote control unit 526 determines the steering angle so that the vehicle 10 does not deviate from the reference route Ip, and when the vehicle 10 is not located on the reference route Ip, in other words, when the vehicle 10 deviates from the reference route Ip, the remote control unit 526 determines the steering angle so that the vehicle 10 returns to the reference route Ip.

[0038] The remote control unit 526 transmits the generated driving control signal to the vehicle 10 (step S140). The information processing device 5 repeats, at a predetermined cycle, the acquisition of vehicle position information, the determination of a target position, the generation of a driving control signal, and the transmission of the driving control signal.

[0039] The vehicle control device 15 mounted on the vehicle 10 receives a driving control signal from the information processing device 5 (step S150). The operation control unit 153 of the vehicle control device 15 controls the actuator using the received driving control signal to cause the vehicle 10 to drive at the acceleration and steering angle indicated in the driving control signal (step S160). The vehicle control device 15 repeats receiving the driving control signal and controlling the actuator at a predetermined cycle. According to the information processing system 1 of this embodiment, the vehicle 10 can be driven by remote control, and therefore the vehicle 10 can be moved without using transportation equipment such as a crane or a conveyor.

[0040] A-3. Data processing method: Fig. 3 is a flowchart showing a method for relearning the first learning model Md1 in the first embodiment. The flow shown in Fig. 3 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0041] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring a captured image as change information to the external camera 9 (step S211). The external camera 9 that has received the request signal transmits the captured image to the information processing device 5 (step S212).

[0042] After the change information acquisition step, a learning step is executed. In the learning step, the learning unit 522 uses the captured image to determine whether or not the appearance state of the target area has changed since the first learning model Md1 stored in the storage unit 53 was learned. If it is determined that the appearance state of the target area has changed since the first learning model Md1 stored in the storage unit 53 was learned (step S213: Yes), the learning unit 522 re-learns the first learning model Md1 (step S214). On the other hand, if it is determined that the appearance state of the target area has not changed since the first learning model Md1 stored in the storage unit 53 was learned (step S213: No), the learning unit 522 ends the flow without re-learning the first learning model Md1.

[0043] Fig. 4 is a flowchart showing a method for acquiring shape data in the first embodiment. The flow shown in Fig. 4 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0044] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire a captured image as state information (step S311). The external camera 9 that has received the request signal transmits the captured image to the information processing device 5 (step S312). The state information acquisition unit 523 analyzes the captured image to identify the appearance state of the target area (step S313).

[0045] After the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524 acquires shape data by inputting a captured image to the first learning model Md1 that has been trained using a training image including the vehicle 10 and a target area of ​​the appearance state specified by the state information. In detail, when the appearance state specified by the state information is different from the appearance state of the target area in the training image used when the first learning model Md1 stored in the storage unit 53 is trained (step S314: Yes), the control information acquisition unit 524 executes the following process. In this case, the control information acquisition unit 524 inputs the captured image to the first learning model Md1 after re-learning that has been re-learned using a plurality of teacher data in which the training image including the vehicle 10 and the target area of ​​the appearance state specified by the state information is associated with the area answer label. As a result, the control information acquisition unit 524 acquires shape data (step S315). On the other hand, if the appearance state specified by the state information is the same as the appearance state of the target area in the training image used when learning the first learning model Md1 stored in the storage unit 53 (step S314: No), the control information acquisition unit 524 executes the following process. In this case, the control information acquisition unit 524 acquires shape data by inputting the captured image to the first learning model Md1 that has not been re-trained using multiple teacher data that associates the vehicle 10 and the training image including the target area of ​​the appearance state specified by the state information with the area answer label (step S316).

[0046] A-4. Vehicle position calculation method: FIG. 5 is a flowchart showing an example of a method for generating shape data and a method for calculating the position of the vehicle 10 using the shape data. FIG. 6 is a schematic diagram showing examples of various images Im1 to Im6 when the method shown in FIG. 5 is executed. In FIG. 6, step numbers corresponding to the steps in FIG. 5 are assigned. In this embodiment, the positioning point 10e of the vehicle 10 is the rear end on the left side of the vehicle 10. Note that the positioning point 10e of the vehicle 10 may be other than the rear end on the left side of the vehicle 10.

[0047] In an image acquisition step (step S410), the control information acquisition unit 524 acquires an original image Im1 output from the external camera 9. In a distortion correction step (step S420), the control information acquisition unit 524 corrects the distortion of the original image Im1 to generate a corrected image Im2 as a processed image. In a rotation processing step (step S430), the control information acquisition unit 524 rotates the corrected image Im2. As a result, the control information acquisition unit 524 generates a rotated image Im3 as a processed image. In a crop processing step (step S440), the control information acquisition unit 524 deletes from the rotated image Im3 an unnecessary area A2 other than a necessary area A1 constituted by the vehicle 10 and the area surrounding the vehicle 10, among the areas constituting the rotated image Im3. As a result, the control information acquisition unit 524 generates a processed image Im4 as a processed image. The steps shown from step S420 to step S440 are pre-processing steps for improving the detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4. Therefore, at least any of the steps shown from step S420 to step S440 may be omitted.

[0048] In the detection step (step S450), the control information acquisition unit 524 inputs the processed image Im4 to the first learning model Md1. As a result, the control information acquisition unit 524 detects the vehicle 10 from the processed image Im4 and acquires a first mask image Im5 as shape data Da. The first mask image Im5 is an image in which a mask area Ms is added to the processed image Im4 by masking the area indicating the vehicle 10 among the areas constituting the processed image Im4.

[0049] In the perspective transformation step (step S460), the calculation unit 525 performs perspective transformation on the first mask image Im5 to generate a second mask image Im6. The calculation unit 525 performs perspective transformation on the first mask image Im5 into a bird's-eye image seen from a viewpoint above the vehicle 10 that is substantially perpendicular to the road surface Rs, for example, using a predetermined perspective transformation parameter. The perspective transformation parameter is, for example, a parameter related to position information and internal parameters of the external camera 9 obtained by calibration. In this way, the calculation unit 525 generates a second mask image Im6 expressed in the image coordinate system from the first mask image Im5 expressed in the camera coordinate system. The camera coordinate system is a coordinate system having the focal point of the external camera 9 as the origin and having coordinate axes represented by the Xc axis and the Yc axis perpendicular to the Xc axis. The image coordinate system is a coordinate system having a point on the image plane as the origin and having coordinate axes represented by the Xi axis and the Yi axis perpendicular to the Xi axis.

[0050] In the coordinate point calculation step (step S470), the calculation unit 525 calculates an image coordinate point indicating the position of the vehicle 10 in the image coordinate system. FIG. 7 is a diagram for explaining the details of the coordinate point calculation step. In order to calculate the image coordinate point, the calculation unit 525 first calculates a base coordinate point P0 from a first circumscribing rectangle R1 set in the mask area Ms in the first mask image Im5, which is an image before perspective transformation. FIG. 8 is a diagram for explaining a calculation method of the base coordinate point P0. In order to calculate the base coordinate point P0, the calculation unit 525 sets a base circumscribing rectangle R0 for the mask area Ms in the first mask image Im5. Next, the calculation unit 525 rotates the first mask image Im5 by a required rotation amount with the center of gravity C of the mask area Ms as the rotation center so that the direction of the movement vector V of the vehicle 10 corresponding to the mask area Ms in the first mask image Im5 faces a predetermined direction. The predetermined direction is, for example, the upward direction on the screen of the display device that displays the first mask image Im5. Next, the calculation unit 525 sets a first circumscribing rectangle R1 for the mask area Ms of the rotated first mask image Im5 so that the long side is parallel to the movement vector V. Next, the calculation unit 525 rotates the first mask image Im5 to which the first circumscribing rectangle R1 has been added by the above-mentioned rotation amount with the center of gravity C of the mask area Ms as the rotation center. As a result, the calculation unit 525 sets the coordinate point of one of the four vertices of the first circumscribing rectangle R1, which has coordinates that are closest to the positioning point 10e of the vehicle 10, as the base coordinate point P0.

[0051] Next, the calculation unit 525 performs perspective transformation on the first mask image Im5 after reverse rotation, i.e., the first mask image Im5 after calculating the base coordinate point P0, as shown in Fig. 7. As a result, the calculation unit 525 sets the coordinate point corresponding to the base coordinate point P0 as the first coordinate point P1 in the first circumscribed rectangle R1 deformed by the perspective transformation.

[0052] Next, the calculation unit 525 sets a second circumscribing rectangle R2 for the mask area Ms in the second mask image Im6 obtained by performing perspective transformation on the first mask image Im5. Then, the calculation unit 525 sets, among the vertices of the second circumscribing rectangle R2, a vertex indicating the same position as the first coordinate point P1 as the second coordinate point P2. That is, the first coordinate point P1 and the second coordinate point P2 are coordinate points indicating the same position, and therefore have a correlation with each other.

[0053] Next, the calculation unit 525 performs correction to replace the coordinates (Xi1, Yi1) of the first coordinate point P1 with the coordinates (Xi2, Yi2) of the second coordinate point P2 depending on the magnitude relationship between the coordinate values ​​of the first coordinate point P1 and the second coordinate point P2. If the coordinate value Xi1 in the Xi direction of the first coordinate point P1 is greater than the coordinate value Xi2 in the Xi direction of the second coordinate point P2 (Xi1>Xi2), the calculation unit 525 replaces the coordinate value Xi1 in the Xi direction of the first coordinate point P1 with the coordinate value Xi2 in the Xi direction of the second coordinate point P2. When the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is greater than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction (Yi1>Yi2), the calculation unit 525 replaces the coordinate value Yi1 of the first coordinate point P1 in the Yi direction with the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. In the example shown in FIG. 7, the coordinate value Xi1 of the first coordinate point P1 in the Xi direction is greater than the coordinate value Xi2 of the second coordinate point P2 in the Xi direction. Also, the coordinate value Yi1 of the first coordinate point P1 in the Yi direction is smaller than the coordinate value Yi2 of the second coordinate point P2 in the Yi direction. Therefore, the image coordinate point P3 has coordinates (Xi2, Yi1). In this way, the calculation unit 525 calculates the image coordinate point P3 indicating the position of the vehicle 10 in the image coordinate system by correcting the first coordinate point P1 using the second coordinate point P2.

[0054] In the position conversion step (step S480), the calculation unit 525 converts the image coordinate point P3 into a vehicle coordinate point to calculate a vehicle coordinate point indicating the position of the positioning point 10e of the vehicle 10 in the global coordinate system. The calculation unit 525 converts the image coordinate point P3 into the vehicle coordinate point using relational expressions (1) to (3) described later, which use the vehicle coordinate point as a response variable and include the image coordinate point P3, the imaging parameters, and the vehicle parameters as explanatory variables. The imaging parameters are parameters related to the distance of the external camera 9 from a predetermined reference point. In this embodiment, the imaging parameter is the height H of the external camera 9 from the road surface Rs (FIG. 9 described later). The vehicle parameters are parameters related to the distance of the positioning point 10e of the vehicle 10 from the reference point. In this embodiment, the vehicle parameter is the height h of the positioning point 10e of the vehicle 10 from the road surface Rs (FIG. 9 described later).

[0055] FIG. 9 is a first diagram for explaining the details of the position conversion process. FIG. 9 illustrates the state when the vehicle 10 is viewed from the left side. FIG. 10 is a second diagram for explaining the details of the position conversion process. FIG. 10 illustrates the state when the vehicle 10 is viewed from the roof side. The global coordinate system shown in FIG. 9 and FIG. 10 is a coordinate system having a fixed coordinate point Pf indicating an arbitrary reference position of the road surface Rs as the origin, and coordinate axes indicated by the Xg axis and the Yg axis perpendicular to the Xg axis. The imaging coordinate point Pc is a coordinate point indicating the position in the global coordinate system of the external camera 9 that outputs the original image Im1 used in the calculation of the image coordinate point P3. The fixed coordinate point Pf and the imaging coordinate point Pc are stored in advance in, for example, the storage unit 53 of the information processing device 5.

[0056] As shown in Fig. 9, the observation distance on the XgYg plane between the position of the external camera 9 and the position of the vehicle 10 (image coordinate point P3) is Do. The observation error is ΔD. The height [m] of the external camera 9 from the road surface Rs as an imaging parameter is H. The height [m] of the positioning point 10e of the vehicle 10 from the road surface Rs as a vehicle parameter is h. In this case, the observation error ΔD can be expressed by the following equation (1). ΔD=h / H×Do Equation (1) That is, the larger the observation distance Do, the larger the observation error ΔD.

[0057] When the actual distance between the position of the external camera 9 and the positioning point 10e of the vehicle 10 is defined as a first distance D, the first distance D can be expressed by the following formula (2). D=Do×(1-h / H) Equation (2) That is, the first distance D is determined by the observation distance Do, the height H of the external camera 9 as an imaging parameter, and the height h of the positioning point 10e of the vehicle 10 as a vehicle parameter.

[0058] As shown in Figure 10, if the distance between the reference position and the estimated position of the vehicle 10 is the estimated distance Dp, and the actual distance between the reference position and the vehicle 10 is the second distance Dt, the second distance Dt can be expressed by the following equation (3). Dt = Dp × (1-h / H) Equation (3)

[0059] Here, the estimated distance Dp can be calculated using the third distance Dc, which is the actual distance calculated from the fixed coordinate point Pf and the image coordinate point Pc, the image coordinate point P3, and the fixed coordinate point Pf. Therefore, the calculation unit 525 calculates the vehicle coordinate point Pv using the second distance Dt and the fixed coordinate point Pf, which are calculated by correcting the estimated distance Dp using the above formula (3). At this time, the calculated vehicle coordinate point Pv is a coordinate point indicating the position of the positioning point 10e of the vehicle 10 in the global coordinate system, and therefore corresponds to the position of the vehicle 10 in the real space.

[0060] According to the first embodiment, the information processing device 5 can acquire the change information. Then, when the information processing device 5 determines that the appearance state of the target area Ta has changed using the change information, the information processing device 5 can execute the following process. In this case, the information processing device 5 can re-learn the first learning model Md1, which has been learned using a plurality of teacher data in which a training image including the vehicle 10 and the target area Ta before the appearance state changes is associated with an area correct label, using a plurality of teacher data in which a training image including the vehicle 10 and the target area Ta after the appearance state changes is associated with an area correct label. In this way, when the appearance state of the target area Ta changes, it is possible to suppress a decrease in detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0061] Furthermore, according to the first embodiment, the information processing device 5 can acquire, as change information, the captured images Im1-Im4 including the target area Ta and the vehicle 10. Then, the information processing device 5 can determine whether or not the appearance state of the target area Ta has changed since the first learning model Md1 was learned by detecting dirt from the captured images Im1-Im4 through image analysis.

[0062] According to the first embodiment, the information processing device 5 can re-learn the first learning model Md1 during the period when control of unmanned driving is being executed. Note that, in other embodiments, the information processing device 5 may learn the first learning model Md1 during the period when control of manned driving is being executed, for example, without executing control of unmanned driving.

[0063] According to the first embodiment, the vehicle 10 runs in an unmanned manner within the factory. In this manner, the information processing device 5 can re-learn the first learning model Md1 during the process of manufacturing the vehicle 10.

[0064] According to the first embodiment, the information processing device 5 can acquire state information. When the appearance state specified by the state information is different from the appearance state of the target area Ta in the training image used when the first learning model Md1 is learned, the information processing device 5 can acquire the shape data Da by executing the following process. In this case, the information processing device 5 can acquire the shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 that has been re-learned using a plurality of teacher data in which the training image including the target area Ta of the vehicle 10 and the appearance state specified by the state information is associated with the area correct label. In this way, when the appearance state of the target area Ta is different from when the first learning model Md1 is learned, it is possible to suppress a decrease in accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0065] Furthermore, according to the first embodiment, the information processing device 5 can acquire, as state information, the captured images Im1-Im4 including the target area Ta and the vehicle 10. Then, the information processing device 5 can identify the appearance state of the target area Ta by detecting dirt from the captured images Im1-Im4 through image analysis.

[0066] The information processing device 5 may re-learn the first learning model Md1 every time the appearance state of the target area Ta changes, and may store the first learning model Md1 after re-learning in the storage unit 53 as a new first learning model Md1 different from the first learning model Md1 before re-learning. When the first learning model Md1 after re-learning is stored in the storage unit 53 as a new first learning model Md1 different from the first learning model Md1 before re-learning, it is preferable to do as follows. In this case, it is preferable to associate the supplementary information indicating the appearance state of the target area Ta contained in the training image used during re-learning with the first learning model Md1 after re-learning. In this form, the information processing device 5 can acquire at least one of the traveling control signal and the signal generation parameter using the learned first learning model Md1 according to the appearance state of the target area Ta without re-learning every time the appearance state of the target area Ta changes. Furthermore, the information processing device 5 may obtain both the traveling control signal and the signal generation parameters by inputting the captured images Im1 to Im4 to the first learning model Md1.

[0067] B. Second embodiment: FIG. 11 is a diagram showing the configuration of an information processing system 1a in the second embodiment. The information processing system 1a includes one or more vehicles 10, an external camera 9, and an information processing device 5a. In this embodiment, the types of change information and state information are different from those in the first embodiment. The element that affects the change in the appearance state in the target area Ta is dirt that occurs in the target area Ta, as in the first embodiment. The other configurations of the information processing system 1a are the same as those in the first embodiment unless otherwise described. The same symbols are used for the same configurations as in the first embodiment, and descriptions are omitted.

[0068] The information processing device 5a includes a communication unit 51, a CPU 52a, and a storage unit 53a. The storage unit 53a of the information processing device 5a stores various information including various programs P5a for controlling the operation of the information processing device 5a, a reference route Ip, a first learning model Md1, and cleaning information Ic. The cleaning information Ic is information indicating the cleaning status of dirt that may occur in the passage Ro and the surrounding area of ​​the passage Ro for each target area Ta. Specifically, the cleaning information Ic is information indicating the timing of cleaning in association with each imaging area RG of the external camera 9. The cleaning information Ic is updated every time cleaning is performed. Cleaning of dirt is performed at a predetermined cycle.

[0069] The CPU 52a of the information processing device 5a functions as a change information acquisition unit 521a, a learning unit 522a, a status information acquisition unit 523a, a control information acquisition unit 524, a calculation unit 525, and a remote control unit 526 by deploying various programs P5a stored in the memory unit 53a.

[0070] The change information acquisition unit 521a acquires cleaning information Ic as the change information. The learning unit 522a uses the cleaning information Ic to determine whether the appearance state of the target area Ta has changed. For example, when the pre-change state is a state in which no dirt has occurred in the target area Ta and the post-change state is a state in which dirt has occurred in the target area Ta, the learning unit 522a makes the following determination. In this case, when cleaning of the target area Ta is performed within a predetermined period, the learning unit 522a determines that no dirt has occurred in the target area Ta and that the appearance state of the target area Ta has not changed. On the other hand, when cleaning of the target area Ta is not performed within a predetermined period, the learning unit 522a determines that dirt has occurred in the target area Ta and that the appearance state of the target area Ta has changed.

[0071] The state information acquisition unit 523a acquires the state information by generating the state information using the cleaning information Ic. For example, when the pre-change state is a state in which no dirt has occurred in the target area Ta and the post-change state is a state in which dirt has occurred in the target area Ta, the state information acquisition unit 523a generates the state information as follows. In this case, when the target area Ta has been cleaned within a predetermined period, the state information acquisition unit 523a generates state information indicating that no dirt has occurred in the target area Ta. On the other hand, when the target area Ta has not been cleaned within a predetermined period, the state information acquisition unit 523 generates state information indicating that dirt has occurred in the target area Ta.

[0072] According to the second embodiment, the information processing device 5a can acquire the cleaning information Ic as change information. Then, the information processing device 5a can determine whether the appearance state of the target area Ta has changed since the first learning model Md1 was learned by determining whether the target area Ta has been cleaned within a predetermined period of time using the cleaning information Ic.

[0073] Furthermore, according to the second embodiment, the information processing device 5a can acquire state information by using the cleaning information Ic to determine whether the target area Ta has been cleaned within a predetermined period of time.

[0074] C. Third embodiment: FIG. 12 is a diagram showing the configuration of an information processing system 1b in the third embodiment. The information processing system 1b includes one or more vehicles 10, an external camera 9, and an information processing device 5b. In this embodiment, when it is determined that the appearance state of the target area Ta has changed, the information processing device 5b notifies the administrator of learning request information for re-learning the first learning model Md1. Then, the administrator who recognizes the learning request information instructs the first learning model Md1 to be re-learned via the input operation unit 55, so that the first learning model Md1 is re-learned. The other configurations of the information processing system 1b are the same as those of the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0075] The information processing device 5b includes a communication unit 51, a CPU 52b, a storage unit 53b, a notification unit 54, and an input operation unit 55. The storage unit 53b of the information processing device 5b stores various information including various programs P5b that control the operation of the information processing device 5b, a reference path Ip, and a first learning model Md1. The notification unit 54 notifies the administrator of the learning request information. The notification unit 54 is, for example, a display that displays the learning request information as text information. The notification unit 54 may be a speaker that reproduces the learning request information as audio information. The input operation unit 55 is an input device for the administrator who recognizes the learning request information to instruct the administrator to re-learn the first learning model Md1. The input operation unit 55 is, for example, a touch panel, a mouse, or a keyboard that accepts input from the administrator.

[0076] The CPU 52b of the information processing device 5b deploys various programs P5b stored in the storage unit 53b, thereby functioning as a change information acquisition unit 521, a notification control unit 527, and a learning unit 522b. Furthermore, the CPU 52b of the information processing device 5b deploys various programs P5b stored in the storage unit 53b, thereby functioning as a state information acquisition unit 523, a control information acquisition unit 524, a calculation unit 525, and a remote control unit 526.

[0077] When it is determined that the appearance state of the target area Ta has changed using the change information, the notification control unit 527 notifies the administrator of the learning request information. In this embodiment, the notification control unit 527 determines whether the appearance state of the target area Ta has changed using the captured images Im1 to Im4 as the change information. Then, when it is determined that the appearance state of the target area Ta has changed, the notification control unit 527 notifies the administrator of the learning request information via the notification unit 54.

[0078] When the learning unit 522b receives a re-learning instruction from the administrator via the input operation unit 55, the learning unit 522b re-learns the first learning model Md1.

[0079] Fig. 13 is a flowchart showing a method for relearning the first learning model Md1 in the third embodiment. The flow shown in Fig. 13 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0080] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1-Im4 as change information to the external camera 9 (step S221). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5b (step S222).

[0081] After the change information acquisition step, a notification step is executed. In the notification step, the notification control unit 527 uses the captured images Im1 to Im4 to determine whether the appearance state of the target area Ta has changed. When it is determined that the appearance state of the target area Ta has changed (step S223: Yes), the notification control unit 527 notifies the administrator of the learning request information via the notification unit 54 (step S224). After the notification step, a learning step is executed. In the learning step, when a re-learning instruction is received from the administrator via the input operation unit 55 (step S225: Yes), the learning unit 522b re-learns the first learning model Md1 (step S226). On the other hand, when it is determined that the appearance state of the target area Ta has not changed (step S223: No), the information processing device 5b ends the flow without notifying the administrator of the learning request information or re-learning the first learning model Md1.

[0082] According to the third embodiment, when the information processing device 5b determines that the appearance state of the target area Ta has changed using the change information, the information processing device 5b can notify the administrator of the learning request information. In this way, the information processing device 5b can prompt the administrator to take measures to re-learn the first learning model Md1. As a result, the administrator who recognizes the learning request information can quickly re-learn the first learning model Md1 by issuing a re-learning instruction to the information processing device 5b via the input operation unit 55.

[0083] The information processing device 5b may include the notification control unit 527 without including the learning unit 522b. Even in this embodiment, when it is determined that the appearance state of the target area Ta has changed using the change information, the information processing system 1b can re-learn the first learning model Md1 by communicatively connecting another device, such as a machine learning device (not shown) that realizes the function of the learning unit 522b, to the information processing device 5b.

[0084] D. Fourth embodiment: FIG. 14 is a diagram showing a configuration of an information processing system 1c in the fourth embodiment. The information processing system 1c includes one or more vehicles 10, an external camera 9, and an information processing device 5c. In this embodiment, the information processing device 5c learns a second learning model Md2 as a machine learning model Md for each appearance state of the target area Ta. Then, the information processing device 5c inputs the captured images Im1 to Im4 to the second learning model Md2 corresponding to the appearance state of the target area Ta specified by the state information from among a plurality of second learning models Md2 prepared for each appearance state of the target area Ta, thereby acquiring shape data Da as a parameter for signal generation. Unless otherwise described, the other configurations of the information processing system 1c are the same as those of the first embodiment. The same reference numerals are given to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0085] The information processing device 5c includes a communication unit 51, a CPU 52c, and a storage unit 53c. The storage unit 53c of the information processing device 5c stores various information including various programs P5c for controlling the operation of the information processing device 5c, a reference path Ip, and a plurality of second learning models Md2 prepared for each appearance state of the target area Ta.

[0086] When the captured images Im1 to Im4 are input, each of the second learning models Md2 detects the vehicle 10 from the captured images Im1 to Im4 and outputs shape data Da. The second learning model Md2 is learned by supervised learning using a priority learning data set corresponding to the appearance state of the target area Ta among a plurality of priority learning data sets prepared for each appearance state of the target area Ta. Each priority learning data set includes M pieces (M is an integer of 2 or more) of first training data and N pieces (N is an integer of 0 or more and less than M) of second training data. The first training data is training data in which a training image including the target area Ta and the vehicle 10 classified into one appearance state among the plurality of appearance states of the target area Ta is associated with an area correct answer label. The second training data is training data in which a training image including the target area Ta and the vehicle 10 classified into the remaining appearance state other than the one appearance state among the plurality of appearance states of the target area Ta is associated with a correct answer label. The number M of the first training data may be, for example, 1.5 times or more, or may be twice or more, the number N of the second training data. In addition, when there are a plurality of appearance states different from the one appearance state, the number M of the first training data is, for example, more than the total number of the second training data. In other words, the second learning model Md2 is a machine learning model Md that has been made to preferentially learn the feature amounts of the target region Ta classified into one appearance state by having more first training data learn than the second training data.

[0087] The second learning models Md2 prepared for each appearance state of the target area Ta are, for example, a non-stained model Md2a and a stained model Md2b. The non-stained model Md2a is the second learning model Md2 that has been trained with more first training images including the vehicle 10 and the target area Ta in an unstained state than the second training images including the vehicle 10 and the target area Ta in an unstained state. The non-stained model Md2a is used when detecting the vehicle 10 from the captured images Im1 to Im4 including the vehicle 10 and the target area Ta in an unstained state. The stained model Md2b is the second learning model Md2 that has been trained with more first training images including the vehicle 10 and the target area Ta in an unstained state than the second training images including the vehicle 10 and the target area Ta in an unstained state. The stained model Md2b is used when detecting the vehicle 10 from the captured images Im1 to Im4 including the vehicle 10 and the target area Ta in an unstained state.

[0088] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the second learning model Md2. During learning of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the second learning model Md2 and the region correct label.

[0089] The CPU 52c of the information processing device 5c functions as a learning unit 522c, a state information acquisition unit 523, a model acquisition unit 528, a control information acquisition unit 524c, a calculation unit 525, and a remote control unit 526 by developing various programs P5c stored in the storage unit 53c. The learning unit 522c learns a second learning model Md2 for each appearance state of the target area Ta. The model acquisition unit 528 acquires a second learning model Md2 selected according to the appearance state of the target area Ta specified by the state information from among a plurality of second learning models Md2 prepared for each appearance state of the target area Ta. The control information acquisition unit 524c acquires shape data Da by inputting the captured images Im1 to Im4 to the second learning model Md2 acquired by the model acquisition unit 528.

[0090] Fig. 15 is a flowchart showing a learning method of the second learning model Md2. The flow shown in Fig. 15 is executed, for example, before the vehicle 10 starts traveling in an unmanned driving mode.

[0091] In the preparation step, a priority learning data set is prepared for each appearance state of the target region Ta (step S231). After the preparation step, a learning step is executed. In the learning step, the learning unit 522c uses the priority learning data set to train a second learning model Md2 for each appearance state of the target region Ta (step S232).

[0092] Fig. 16 is a flowchart showing a method for acquiring shape data Da using the second learning model Md2. The flow shown in Fig. 16 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0093] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S331). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5c (step S332). The state information acquisition unit 523 analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S333).

[0094] After the state information acquisition step, a model acquisition step is executed. In the model acquisition step, the model acquisition unit 528 acquires a second learning model Md2 selected according to the appearance state of the target area Ta specified by the state information from among a plurality of second learning models Md2 prepared for each appearance state of the target area Ta (step S334).

[0095] After the model acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524c acquires shape data Da by inputting the captured images Im1 to Im4 to the second learning model Md2 acquired by the model acquisition unit 528 (step S335).

[0096] According to the fourth embodiment, the information processing device 5c learns the second learning model Md2 by using more training images including the vehicle 10 and the target area Ta classified into one appearance state than training images including the vehicle 10 and the target area Ta classified into the remaining other appearance states. This allows the information processing device 5c to prepare the second learning model Md2 suitable for detecting the vehicle 10 from the captured images Im1 to Im4 including the vehicle 10 and the target area Ta classified into one appearance state.

[0097] According to the fourth embodiment, the second learning model Md2 is prepared for each appearance state of the target area Ta. Thus, the information processing device 5c can acquire state information and acquire the second learning model Md2 selected according to the appearance state of the target area Ta specified by the state information from among the multiple second learning models Md2 prepared for each appearance state of the target area Ta. Then, the information processing device 5c can accurately detect the vehicle 10 from the captured images Im1 to Im4 by inputting the captured images Im1 to Im4 to the second learning model Md2 according to the appearance state of the target area Ta included in the captured images Im1 to Im4. Thus, the information processing device 5c can improve the detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0098] E. Fifth embodiment: FIG. 17 is a diagram showing a configuration of an information processing system 1d in the fifth embodiment. The information processing system 1d includes one or more vehicles 10, an external camera 9, and an information processing device 5d. In this embodiment, the information processing device 5d associates an appearance state with a training image and learns a third learning model Md3 as a machine learning model Md. The information processing device 5d then inputs the captured images Im1 to Im4 to the third learning model Md3 to obtain shape data Da as a parameter for signal generation. Unless otherwise described, the other configurations of the information processing system 1d are the same as those of the first embodiment. The same reference numerals are given to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0099] The information processing device 5d includes a communication unit 51, a CPU 52d, and a storage unit 53d. The storage unit 53d of the information processing device 5d stores various information including various programs P5d for controlling the operation of the information processing device 5d, a reference path Ip, and a third learning model Md3.

[0100] When the captured images Im1 to Im4 are input, the third learning model Md3 detects the vehicle 10 from the captured images Im1 to Im4 and outputs shape data Da. The third learning model Md3 is trained by supervised learning using a simultaneous learning dataset. The simultaneous learning dataset includes multiple pieces of teacher data that associate a training image including the target area Ta and the vehicle 10 with a region correct label and supplementary information indicating the appearance state of the target area Ta included in the training image.

[0101] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the third learning model Md3. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the third learning model Md3 and the region correct label.

[0102] The CPU 52d of the information processing device 5d functions as a learning unit 522d, a state information acquisition unit 523, a control information acquisition unit 524d, a calculation unit 525, and a remote control unit 526 by deploying various programs P5d stored in the storage unit 53d. The learning unit 522d trains a third learning model Md3 using a simultaneous learning data set. The control information acquisition unit 524d inputs the captured images Im1 to Im4 and the appearance state of the target area Ta specified by the state information to the third learning model Md3 to acquire shape data Da.

[0103] Fig. 18 is a flowchart showing a learning method of the third learning model Md3. The flow shown in Fig. 18 is executed, for example, before the vehicle 10 starts traveling in an unmanned driving mode.

[0104] In the preparation step, a simultaneous learning data set is prepared (step S241). After the preparation step, a learning step is executed. In the learning step, the learning unit 522d trains the third learning model Md3 using the simultaneous learning data set (step S242).

[0105] Fig. 19 is a flowchart showing a method for acquiring shape data Da using the third learning model Md3. The flow shown in Fig. 19 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0106] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S341). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5d (step S342). The state information acquisition unit 523 analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S343).

[0107] After the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524d inputs the captured images Im1 to Im4 and the appearance state of the target area Ta specified by the state information to the third learning model Md3. As a result, the control information acquisition unit 524d acquires shape data Da (step S344).

[0108] According to the fifth embodiment, the information processing device 5d can train the third learning model Md3 using a plurality of teacher data in which a training image including the target area Ta and the vehicle 10, a region correct answer label, and supplementary information indicating the appearance state of the target area Ta included in the training image are associated with each other. This allows the information processing device 5d to prepare the third learning model Md3 trained in association with the appearance state of the target area Ta.

[0109] According to the fifth embodiment, the third learning model Md3 is prepared in advance, which is learned using a plurality of teacher data items that correspond to a training image including the target area Ta and the vehicle 10, a region correct label, and supplementary information indicating the appearance state of the target area Ta included in the training image. As a result, the information processing device 5d can accurately detect the vehicle 10 from the captured images Im1 to Im4 by acquiring state information and inputting the captured images Im1 to Im4 and the appearance state of the target area Ta identified by the state information to the third learning model Md3. Therefore, the information processing device 5d can improve the detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0110] F. Sixth embodiment: FIG. 20 is a diagram showing an example of a method for generating a training image Ti. In this embodiment, the training image Ti used for learning the machine learning model Md is a synthetic image Ci. The synthetic image Ci is a two-dimensional image generated by synthesizing a virtual vehicle image Vi with a background image Bi, which is either a real background image Br or a virtual background image Bv. The background image Bi is an image including a target area Ta without including a vehicle 10. The real background image Br is a background image Bi obtained by actually capturing an image of the target area Ta using an external camera 9. The virtual background image Bv is a background image Bi that virtually reproduces the target area Ta. The virtual background image Bv is, for example, either two-dimensional CAD data or three-dimensional CAD data representing the target area Ta. The virtual vehicle image Vi is an image that virtually reproduces the vehicle 10. The virtual vehicle image Vi is either two-dimensional CAD data or three-dimensional CAD data representing the vehicle 10.

[0111] In this embodiment, the composite image Ci is generated by compositing the virtual vehicle image Vi with the virtual background image Bv. Specifically, the composite image Ci is generated by superimposing the virtual vehicle image Vi on each target area Ta of a production line of a factory reproduced in a virtual space, for example. That is, in this embodiment, the training image Ti is generated by utilizing a digital twin.

[0112] When the virtual background image Bv is three-dimensional data, the virtual background image Bv is converted from three-dimensional data to two-dimensional data in the process of generating the composite image Ci. When the virtual vehicle image Vi is three-dimensional data, the virtual vehicle image Vi is converted from three-dimensional data to two-dimensional data in the process of generating the composite image Ci. In addition, in the process of combining the virtual vehicle image Vi with the background image Bi, the position of the virtual vehicle image Vi with respect to the background image Bi may be adjusted, or the pixel values ​​of each of the images Bi, Ci, and Vi may be adjusted.

[0113] According to the sixth embodiment, the machine learning model Md can be trained by using the composite image Ci generated by combining the virtual vehicle image Vi with the background image Bi as the training image Ti.

[0114] According to the sixth embodiment, when the virtual vehicle image Vi is synthesized with the background image Bi, at least one of the position and the orientation of the vehicle 10 can be varied by adjusting the arrangement of the virtual vehicle image Vi with respect to the background image Bi. This makes it possible to easily generate a plurality of training images Ti in which at least one of the position and the orientation of the vehicle 10 with respect to the passage Ro is different.

[0115] According to the sixth embodiment, in the process of synthesizing the virtual vehicle image Vi with the background image Bi, at least one of the pixel values ​​of the background image Bi, the virtual vehicle image Vi, and the synthetic image Ci can be changed from the initial value. This makes it possible to easily generate a plurality of training images Ti with different imaging conditions such as brightness. Therefore, it is possible to easily generate a plurality of training images Ti assuming a case where pixel values ​​such as brightness change depending on the weather, time of day, season, lighting environment, etc.

[0116] According to the sixth embodiment, the machine learning model Md can be trained by using the composite image Ci generated by combining the virtual vehicle image Vi with the virtual background image Bv as the training image Ti. In this way, the target area Ta with different appearance states can be easily reproduced by changing the configuration of the virtual background image Bv, so that a plurality of training images Ti including the target areas Ta with different appearance states can be easily generated.

[0117] In addition, as in this embodiment, when the training image Ti is a synthetic image Ci, the correct answer label of the machine learning model Md that outputs the shape data Da may be the contour correct answer label Lc instead of the area correct answer label Lr. The contour correct answer label Lc is a correct answer label that represents the contour of the vehicle 10 extracted from the virtual vehicle image Vi. In this form, by extracting the outer shape of the virtual vehicle image Vi, it is possible to easily identify whether each area constituting the training image Ti is an area indicating the vehicle 10 or an area indicating something other than the vehicle 10. This makes it possible to reduce the load required for preparation for learning the machine learning model Md.

[0118] G. Seventh embodiment: FIG. 21 is a diagram showing a configuration of an information processing system 1e in the seventh embodiment. The information processing system 1e includes one or more vehicles 10, an external camera 9, and an information processing device 5e. In this embodiment, the information processing device 5e inputs captured images Im1 to Im4 to a fourth learning model Md4 as a machine learning model Md to obtain rectangular coordinate values ​​as parameters for signal generation. The other configurations of the information processing system 1e are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0119] 22 is a diagram for explaining the rectangular coordinate values ​​Cr1 to Cr4. The rectangular coordinate values ​​Cr1 to Cr4 are the coordinate values ​​of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc. The vehicle circumscribing rectangle Rc is a quadrangle that is set so as to surround the area occupied by the vehicle 10 when the vehicle 10 is projected onto the road surface Rs in the images Ti, Im1 to Im4. The rectangular coordinate values ​​Cr1 to Cr4 are used to calculate at least one of the position and the orientation of the vehicle 10.

[0120] 21, the information processing device 5e includes a communication unit 51, a CPU 52e, and a storage unit 53e. The storage unit 53e of the information processing device 5e stores various information including various programs P5e for controlling the operation of the information processing device 5e, a reference path Ip, a fourth learning model Md4, a rectangle database Db1, and a camera parameter Pa.

[0121] The fourth learning model Md4 outputs rectangular coordinate values ​​Cr1 to Cr4 when the captured images Im1 to Im4 are input. The learning data set used for learning the fourth learning model Md4 includes a plurality of teacher data in which a training image Ti including a target area Ta and a vehicle 10 is associated with a rectangular coordinate correct label. The rectangular coordinate correct label is a correct label indicating the coordinate values ​​Cr1 to Cr4 of the four vertices Vc1 to Vc4 of a vehicle circumscribing rectangle Rc set for the vehicle 10 included in the training image Ti. In the rectangular coordinate correct label, each of the coordinate values ​​Cr1 to Cr4 is associated with additional information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc the coordinate values ​​Cr1 to Cr4 are. In this embodiment, the rectangular coordinate values ​​Cr1 to Cr4 output from the fourth learning model Md4 are coordinate values ​​in an image coordinate system. Therefore, in the rectangular coordinate correct label, each of the coordinate values ​​Cr1 to Cr4 is expressed in the image coordinate system.

[0122] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the fourth learning model Md4. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the fourth learning model Md4 and the rectangular coordinate correct label.

[0123] The rectangle database Db1 is used to calculate the coordinate values ​​of the positioning point 10e of the vehicle 10 from the rectangle coordinate values ​​Cr1 to Cr4. The rectangle database Db1 is a database that indicates the relative positional relationship between each of the vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc and the positioning point 10e of the vehicle 10.

[0124] The camera parameters Pa are parameters related to the external camera 9. In this embodiment, the camera parameters Pa are used to convert the coordinate values ​​of the positioning point 10e of the vehicle 10 expressed in the image coordinate system into the global coordinate system. The camera parameters Pa are, for example, the installation position, installation attitude, and focal length of the external camera 9.

[0125] The CPU 52e of the information processing device 5e functions as a change information acquisition unit 521, a learning unit 522e, a status information acquisition unit 523, a control information acquisition unit 524e, a calculation unit 525e, and a remote control unit 526 by deploying various programs P5e stored in the memory unit 53e.

[0126] The learning unit 522e uses the change information to determine whether the appearance state of the target area Ta has changed since the fourth learning model Md4 was learned. If it is determined that the appearance state of the target area Ta has changed, the learning unit 522e re-learns the fourth learning model Md4 using a re-learning dataset including a plurality of teacher data items that correspond to the vehicle 10 and the training image Ti including the target area Ta after the appearance state has changed, and the rectangular coordinate answer label.

[0127] The control information acquisition unit 524e uses the state information to identify the appearance state of the target area Ta, and acquires rectangular coordinate values ​​Cr1 to Cr4 expressed in the image coordinate system by inputting the captured images Im1 to Im4 to a fourth learning model Md4 corresponding to the appearance state of the target area Ta.

[0128] The calculation unit 525e calculates the coordinate value of the positioning point 10e of the vehicle 10 from the rectangle coordinate values ​​Cr1 to Cr4 using the rectangle database Db1. Then, the calculation unit 525e converts the coordinate value of the positioning point 10e of the vehicle 10 expressed in the image coordinate system into a coordinate value in the global coordinate system using the camera parameter Pa. At this time, in order to acquire the camera parameter Pa, the calculation unit 525e may acquire a camera identifier for identifying the multiple external cameras 9, which indicates the external camera 9 that acquired the captured images Im1 to Im4.

[0129] The calculation unit 525e further calculates the orientation of the vehicle 10. The calculation unit 525e calculates the orientation of the vehicle 10 by using, for example, the shape of the vehicle circumscribing rectangle Rc and supplementary information indicating which of the four vertices Vc1 to Vc4 of the vehicle circumscribing rectangle Rc have coordinate values ​​Cr1 to Cr4.

[0130] Fig. 23 is a flowchart showing a method for relearning the fourth learning model Md4. The flow shown in Fig. 23 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling in an unmanned driving state, for example.

[0131] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1-Im4 as change information to the external camera 9 (step S251). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5e (step S252).

[0132] After the change information acquisition process, a learning process is executed. In the learning process, the learning unit 522e uses the captured images Im1 to Im4 to determine whether the appearance state of the target area Ta has changed since the fourth learning model Md4 was learned. If it is determined that the appearance state of the target area Ta has changed since the fourth learning model Md4 was learned (step S253: Yes), the learning unit 522e re-learns the fourth learning model Md4 (step S254). On the other hand, if it is determined that the appearance state of the target area Ta has not changed since the fourth learning model Md4 was learned (step S253: No), the learning unit 522e ends the flow without re-learning the fourth learning model Md4.

[0133] Fig. 24 is a flowchart showing a method for acquiring the rectangular coordinate values ​​Cr1 to Cr4 using the fourth learning model Md4. The flow shown in Fig. 24 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0134] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S351). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5e (step S352). The state information acquisition unit 523 analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S353).

[0135] After the state information acquiring step, a control information acquiring step is executed. In the control information acquiring step, the control information acquiring unit 524e acquires rectangular coordinate values ​​Cr1 to Cr4 by inputting the captured images Im1 to Im4 to the fourth learning model Md4 (Step S354).

[0136] According to the seventh embodiment, the information processing device 5e can train the fourth learning model Md4 that outputs rectangular coordinate values ​​Cr1-Cr4 when the captured images Im1-Im4 are input. As a result, the information processing device 5e can obtain the rectangular coordinate values ​​Cr1-Cr4 by inputting the captured images Im1-Im4 to the fourth learning model Md4, and generate a driving control signal.

[0137] The rectangular coordinate values ​​Cr1 to Cr4 may be coordinate values ​​in a camera coordinate system or may be coordinate values ​​in a global coordinate system. When the rectangular coordinate values ​​Cr1 to Cr4 are coordinate values ​​expressed in either the camera coordinate system or the global coordinate system, the fourth learning model Md4 is learned, for example, as follows. In this case, the fourth learning model Md4 is learned by associating at least one of a camera identifier indicating the external camera 9 that acquired the training image Ti and a camera parameter Pa of the external camera 9 that acquired the training image Ti with the training image Ti. In addition, in this case, in the rectangular coordinate correct label, each coordinate value Cr1 to Cr4 is expressed, for example, in the same coordinate system as the coordinate system of the output rectangular coordinate values ​​Cr1 to Cr4.

[0138] Furthermore, the information processing device 5e may acquire the coordinate values ​​of the ground contact surface of the wheel as the signal generation parameters instead of the rectangular coordinate values ​​Cr1 to Cr4. Even in this form, the information processing device 5e can acquire the rectangular coordinate values ​​Cr1 to Cr4 and generate the running control signal.

[0139] H. Eighth embodiment: FIG. 25 is a diagram showing the configuration of an information processing system 1f in the eighth embodiment. The information processing system 1f includes one or more vehicles 10, an external camera 9, and an information processing device 5f. In this embodiment, the information processing device 5f inputs captured images Im1 to Im4 to a fifth learning model Md5 as a machine learning model Md to obtain three-dimensional coordinate values ​​as parameters for signal generation. The other configurations of the information processing system 1f are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0140] 26 is a diagram for explaining the three-dimensional coordinate values ​​Cb1 to Cb8. The three-dimensional coordinate values ​​Cb1 to Cb8 are the coordinate values ​​of eight vertices Vb1 to Vb8 of a vehicle circumscribing rectangular solid Rb. The vehicle circumscribing rectangular solid Rb is a rectangular solid set so as to surround the vehicle 10 in the images Ti, Im1 to Im4. The three-dimensional coordinate values ​​Cb1 to Cb8 are used to calculate at least one of the position and the orientation of the vehicle 10.

[0141] 25, the information processing device 5f includes a communication unit 51, a CPU 52f, and a storage unit 53f. The storage unit 53f of the information processing device 5f stores various information including various programs P5f for controlling the operation of the information processing device 5f, a reference path Ip, a fifth learning model Md5, a stereo database Db2, and a camera parameter Pa.

[0142] The fifth learning model Md5 outputs three-dimensional coordinate values ​​Cb1 to Cb8 when the captured images Im1 to Im4 are input. The learning data set used for learning the fifth learning model Md5 includes a plurality of teacher data in which a training image Ti including a target area Ta and a vehicle 10 is associated with a three-dimensional coordinate correct label. The three-dimensional coordinate correct label is a correct label indicating the coordinate values ​​Cb1 to Cb8 of the eight vertices Vb1 to Vb8 of a vehicle circumscribed rectangular parallelepiped Rb set for the vehicle 10 included in the training image Ti. In the three-dimensional coordinate correct label, each of the coordinate values ​​Cb1 to Cb8 is associated with additional information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular parallelepiped Rb the coordinate values ​​Cb1 to Cb8 are. In this embodiment, the three-dimensional coordinate values ​​Cb1 to Cb8 output from the fifth learning model Md5 are coordinate values ​​in the camera coordinate system. Therefore, in the stereoscopic coordinate correct label, each of the coordinate values ​​Cb1 to Cb8 is expressed in the camera coordinate system. The fifth learning model Md5 is learned by associating at least one of the camera parameters Pa and the camera identifier used to acquire the camera parameters Pa with the training images Ti.

[0143] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the fifth learning model Md5. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the fifth learning model Md5 and the 3D coordinate correct label.

[0144] The three-dimensional database Db2 is used to calculate the coordinate values ​​of the positioning point 10e of the vehicle 10 from the three-dimensional coordinate values ​​Cb1 to Cr8. The three-dimensional database Db2 is a database that indicates the relative positional relationship between each of the vertices Vb1 to Vb8 of the vehicle circumscribed rectangular solid Rb and the positioning point 10e of the vehicle 10.

[0145] The CPU 52f of the information processing device 5f functions as a change information acquisition unit 521, a learning unit 522f, a status information acquisition unit 523, a control information acquisition unit 524f, a calculation unit 525f, and a remote control unit 526 by deploying various programs P5f stored in the memory unit 53f.

[0146] The learning unit 522f uses the change information to determine whether the appearance state of the target area Ta has changed since the fifth learning model Md5 was learned. If it is determined that the appearance state of the target area Ta has changed, the learning unit 522f re-learns the fifth learning model Md5 using a re-learning dataset including a plurality of teacher data items that correspond to the vehicle 10 and the training image Ti including the target area Ta after the appearance state has changed, and the 3D coordinate correct label.

[0147] The control information acquisition unit 524f uses the state information to identify the appearance state of the target area Ta, and acquires three-dimensional coordinate values ​​Cb1 to Cb8 expressed in the camera coordinate system by inputting the captured images Im1 to Im4 to a fifth learning model Md5 corresponding to the appearance state of the target area Ta.

[0148] The calculation unit 525f calculates the coordinate value of the positioning point 10e of the vehicle 10 from the three-dimensional coordinate values ​​Cb1 to Cb8 using the three-dimensional database Db2. Then, the calculation unit 525f converts the coordinate value of the positioning point 10e of the vehicle 10 expressed in the image coordinate system into a coordinate value in the camera coordinate system using the camera parameter Pa. At this time, the calculation unit 525f may acquire a camera identifier in order to acquire the camera parameter Pa.

[0149] The calculation unit 525f further calculates the orientation of the vehicle 10. The calculation unit 525f calculates the orientation of the vehicle 10 by using, for example, the shape of the vehicle circumscribed rectangular solid Rb and supplementary information indicating which of the eight vertices Vb1 to Vb8 of the vehicle circumscribed rectangular solid Rb have coordinate values ​​Cb1 to Cb8.

[0150] Fig. 27 is a flowchart showing a method for relearning the fifth learning model Md5. The flow shown in Fig. 27 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0151] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1-Im4 as change information to the external camera 9 (step S261). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5f (step S262).

[0152] After the change information acquisition process, a learning process is executed. In the learning process, the learning unit 522f uses the captured images Im1 to Im4 to determine whether the appearance state of the target area Ta has changed since the fifth learning model Md5 was learned. If it is determined that the appearance state of the target area Ta has changed since the fifth learning model Md5 was learned (step S263: Yes), the learning unit 522f re-learns the fifth learning model Md5 (step S264). On the other hand, if it is determined that the appearance state of the target area Ta has not changed since the fifth learning model Md5 was learned (step S263: No), the learning unit 522f ends the flow without re-learning the fifth learning model Md5.

[0153] Fig. 28 is a flowchart showing a method for acquiring the three-dimensional coordinate values ​​Cb1 to Cb8 using the fifth learning model Md5. The flow shown in Fig. 28 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0154] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S361). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5f (step S362). The state information acquisition unit 523 analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S363).

[0155] After the state information acquiring step, a control information acquiring step is executed. In the control information acquiring step, the control information acquiring unit 524f acquires the three-dimensional coordinate values ​​Cb1 to Cb8 by inputting the captured images Im1 to Im4 to the fifth learning model Md5 (Step S364).

[0156] According to the eighth embodiment, the information processing device 5f can train the fifth learning model Md5 that outputs the three-dimensional coordinate values ​​Cb1-Cb8 when the captured images Im1-Im4 are input. As a result, the information processing device 5f can obtain the three-dimensional coordinate values ​​Cb1-Cb8 by inputting the captured images Im1-Im4 to the fifth learning model Md5, and generate a traveling control signal.

[0157] The stereoscopic coordinate values ​​Cb1 to Cb8 may be coordinate values ​​in an image coordinate system or may be coordinate values ​​in a global coordinate system.

[0158] I. Ninth embodiment: FIG. 29 is a diagram showing a configuration of an information processing system 1g in the ninth embodiment. The information processing system 1g includes one or more vehicles 10, an external camera 9, and an information processing device 5g. In this embodiment, the information processing device 5g inputs captured images Im1 to Im4 to a sixth learning model Md6 as a machine learning model Md, thereby acquiring vehicle position information as a parameter for signal generation. Unless otherwise described, other configurations of the information processing system 1g are similar to those of the first embodiment. The same reference numerals are attached to the same configurations as those of the first embodiment, and description thereof will be omitted.

[0159] The information processing device 5g includes a communication unit 51, a CPU 52g, and a storage unit 53g. The storage unit 53g of the information processing device 5g stores various information including various programs P5g for controlling the operation of the information processing device 5g, a reference route Ip, a sixth learning model Md6, and a camera parameter Pa. In this embodiment, a program for generating a driving control signal is prepared for each external camera 9. Thus, a reference route Ip is prepared for each external camera 9. The target position determined in the process of generating the driving control signal is represented by X, Y, and Z coordinates in the camera coordinate system.

[0160] The sixth learning model Md6 outputs vehicle position information when the captured images Im1 to Im4 are input. In this embodiment, the sixth learning model Md6 outputs the coordinate value of the positioning point 10e of the vehicle 10 and the orientation of the vehicle 10 when the captured images Im1 to Im4 are input. The learning data set used for learning the sixth learning model Md6 includes a training image Ti including the target area Ta and the vehicle 10, a position coordinate correct label, and an orientation correct label in association with each other. The orientation correct label is a correct label indicating the orientation of the vehicle 10 included in the training image Ti. The position coordinate correct label is a correct label indicating the coordinate value of the positioning point 10e of the vehicle 10 included in the training image Ti. In this embodiment, the coordinate value of the positioning point 10e of the vehicle 10 output from the sixth learning model Md6 is a coordinate value in the camera coordinate system. Therefore, in the position coordinate correct label, the coordinate value of the positioning point 10e of the vehicle 10 is expressed in the camera coordinate system. The sixth learning model Md6 is learned by associating at least one of the camera parameters Pa and the camera identifier used to acquire the camera parameters Pa with the training images Ti.

[0161] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the sixth learning model Md6. During learning of the CNN, for example, parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce errors between the output result of the sixth learning model Md6 and the position coordinate correct label and the orientation correct label.

[0162] The CPU 52g of the information processing device 5g functions as a change information acquisition unit 521, a learning unit 522g, a state information acquisition unit 523, a control information acquisition unit 524g, and a remote control unit 526 by deploying various programs P5g stored in the memory unit 53g.

[0163] The learning unit 522g uses the change information to determine whether the appearance state of the target area Ta has changed since the sixth learning model Md6 was learned. If it is determined that the appearance state of the target area Ta has changed, the learning unit 522g re-learns the sixth learning model Md6 using a re-learning dataset including a plurality of training images Ti including the vehicle 10 and the target area Ta after the appearance state has changed, a position coordinate correct label, and a direction correct label associated with each other.

[0164] The control information acquisition unit 524g uses the state information to identify the appearance state of the target area Ta, and inputs the captured images Im1 to Im4 to the sixth learning model Md6 according to the appearance state of the target area Ta. As a result, the control information acquisition unit 524g acquires the coordinate value of the positioning point 10e of the vehicle 10 expressed in the camera coordinate system, and the orientation of the vehicle 10.

[0165] Fig. 30 is a flowchart showing a method for re-learning the sixth learning model Md6. The flow shown in Fig. 30 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling in an unmanned driving state, for example.

[0166] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1-Im4 as change information to the external camera 9 (step S271). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5f (step S272).

[0167] After the change information acquisition process, a learning process is executed. In the learning process, the learning unit 522g uses the captured images Im1 to Im4 to determine whether the appearance state of the target area Ta has changed since the sixth learning model Md6 was learned. If it is determined that the appearance state of the target area Ta has changed since the sixth learning model Md6 was learned (step S273: Yes), the learning unit 522g re-learns the sixth learning model Md6 (step S274). On the other hand, if it is determined that the appearance state of the target area Ta has not changed since the sixth learning model Md6 was learned (step S273: No), the learning unit 522g ends the flow without re-learning the sixth learning model Md6.

[0168] Fig. 31 is a flowchart showing a method for acquiring vehicle position information using the sixth learning model Md6. The flow shown in Fig. 31 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0169] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S371). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5g (step S372). The state information acquisition unit 523 analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S373).

[0170] After the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524g inputs the captured images Im1 to Im4 to the sixth learning model Md6 to acquire the coordinate value of the positioning point 10e indicating the position of the vehicle 10 (Step S374).

[0171] According to the ninth embodiment, when the captured images Im1 to Im4 are input, the information processing device 5g can train the sixth learning model Md6 that outputs the position and orientation of the vehicle 10. As a result, the information processing device 5g can acquire vehicle position information and generate a traveling control signal by inputting the captured images Im1 to Im4 to the sixth learning model Md6.

[0172] Moreover, according to the ninth embodiment, a program for generating a driving control signal is prepared for each external camera 9. This allows the information processing device 5g to acquire vehicle position information and generate a driving control signal without performing conversion processing from the camera coordinate system to the global coordinate system.

[0173] J. Tenth embodiment: FIG. 32 is a diagram showing the configuration of an information processing system 1h in the tenth embodiment. The information processing system 1h includes one or more vehicles 10, an external camera 9, and an information processing device 5h. In this embodiment, the information processing device 5h acquires a driving control signal by inputting captured images Im1 to Im4 and a camera identifier to a seventh learning model Md7 as a machine learning model Md. The other configurations of the information processing system 1h are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0174] The information processing device 5h includes a communication unit 51, a CPU 52h, and a storage unit 53h. The storage unit 53h of the information processing device 5h stores various information including various programs P5h that control the operation of the information processing device 5h and a seventh learning model Md7.

[0175] The seventh learning model Md7 outputs a driving control signal when the captured images Im1 to Im4 and a camera identifier indicating the external camera 9 that captured the captured images Im1 to Im4 are input. The learning dataset used for learning the seventh learning model Md7 includes a plurality of training images Ti including the target area Ta and the vehicle 10, a camera identifier indicating the external camera 9 that captured the training images Ti, and a signal correct answer label that are associated with each other. The signal correct answer label is a correct answer label that indicates the target value of each parameter included in the driving control signal. The signal correct answer label is generated based on the reference path Ip and the target speed in the image capture area RG of the external camera 9 that captured the training image Ti.

[0176] For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the seventh learning model Md7. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the seventh learning model Md7 and the signal correct answer label.

[0177] The CPU 52h of the information processing device 5h deploys various programs P5h stored in the memory unit 53h, and thereby functions as a change information acquisition unit 521, a learning unit 522h, a state information acquisition unit 523, a control information acquisition unit 524h, and a remote control unit 526h.

[0178] The learning unit 522h uses the change information to determine whether the appearance state of the target area Ta has changed since the seventh learning model Md7 was learned. If it is determined that the appearance state of the target area Ta has changed, the learning unit 522h executes the following process. In this case, the learning unit 522h re-learns the seventh learning model Md7 using a re-learning dataset including a training image Ti including the vehicle 10 and the target area Ta after the appearance state has changed, a camera identifier, and a signal correct answer label.

[0179] The control information acquisition unit 524h uses the state information to identify the appearance state of the target area Ta, and acquires a driving control signal by inputting the captured images Im1 to Im4 and the camera identifier to a seventh learning model Md7 corresponding to the appearance state of the target area Ta. The remote control unit 526h transmits the driving control signal acquired by the control information acquisition unit 524h to the vehicle 10.

[0180] Fig. 33 is a flowchart showing a method for re-learning the seventh learning model Md7. The flow shown in Fig. 33 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0181] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S281). Upon receiving the request signal, the external camera 9 transmits the captured images Im1 to Im4 to the information processing device 5h (step S282).

[0182] After the change information acquisition step, a learning step is executed. In the learning step, the learning unit 522h uses the captured images Im1 to Im4 to determine whether the appearance state of the target area Ta has changed since the seventh learning model Md7 was learned. If it is determined that the appearance state of the target area Ta has changed since the seventh learning model Md7 was learned (step S283: Yes), the learning unit 522h re-learns the seventh learning model Md7 (step S284). On the other hand, if it is determined that the appearance state of the target area Ta has not changed since the seventh learning model Md7 was learned (step S283: No), the learning unit 522h ends the flow without re-learning the seventh learning model Md7.

[0183] Fig. 34 is a flowchart showing a method for acquiring a driving control signal using the seventh learning model Md7. The flow shown in Fig. 34 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts driving in an unmanned driving state, for example.

[0184] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S381). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5h (step S382). The state information acquisition unit 523 analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S383).

[0185] After the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524h inputs the captured images Im1 to Im4 and the camera identifier to the seventh learning model Md7 to acquire a driving control signal (Step S384).

[0186] According to the tenth embodiment, the information processing device 5h can train the seventh learning model Md7 that outputs a driving control signal when the captured images Im1 to Im4 and the camera identifier are input. As a result, the information processing device 5h can acquire a driving control signal by inputting the captured images Im1 to Im4 and the camera identifier to the seventh learning model Md7.

[0187] The information input to the seventh learning model Md7 is not limited to the above. The information processing device 5h may acquire a traveling control signal by inputting other information, such as speed information indicating the traveling speed of the vehicle 10, route information indicating the reference route Ip of the vehicle 10, and pinion angle, in addition to the captured images Im1 to Im4 and the camera identifier, to the seventh learning model Md7. In this manner, the information processing device 5h can acquire a more appropriate traveling control signal according to the traveling situation of the vehicle 10 and the reference route Ip.

[0188] K. Eleventh embodiment: FIG. 35 is a diagram showing the configuration of an information processing system 1i in the eleventh embodiment. The information processing system 1i includes one or more vehicles 10, an external camera 9, and an information processing device 5i. In this embodiment, when it is determined that the appearance state of the target area Ta has changed, the information processing device 5i executes transfer learning in which the first learning model Md1 is learned. Transfer learning is an example of additional learning. The other configurations of the information processing system 1i are the same as those of the first embodiment unless otherwise described. The same symbols are used for the same configurations as those of the first embodiment, and descriptions thereof will be omitted.

[0189] The information processing device 5i includes a communication unit 51, a CPU 52i, and a storage unit 53i. The storage unit 53i of the information processing device 5i stores various information including various programs P5i that control the operation of the information processing device 5i and a first learning model Md1.

[0190] The CPU 52i of the information processing device 5i functions as a change information acquisition unit 521, a learning unit 522i, a status information acquisition unit 523, a control information acquisition unit 524i, a calculation unit 525, and a remote control unit 526 by deploying various programs P5i stored in the memory unit 53i.

[0191] The learning unit 522i uses the change information to determine whether the appearance state of the target area Ta has changed since the first learning model Md1 was learned. Then, when it is determined that the appearance state of the target area Ta has changed, the learning unit 522i executes the following process. In this case, the learning unit 522i transfers the first learning model Md1, which has been learned using the initial learning dataset, to the transfer learning dataset. The transfer learning dataset includes a plurality of teacher data items that associate the training image Ti including the vehicle 10 and the target area Ta after the appearance state has changed with the area correct label Lr. In the transfer learning, the learning unit 522i executes, for example, the following process. Specifically, the learning unit 522i adds a layer of the CNN constituting the first learning model Md1. Then, the learning unit 522i uses the transfer learning dataset to update parameters related to the added layer without updating the trained parameters.

[0192] The control information acquisition unit 524i acquires shape data Da by specifying the appearance state of the target area Ta using the state information and inputting the captured images Im1 to Im4 to the first learning model Md1 according to the appearance state of the target area Ta. Specifically, when the appearance state of the target area Ta specified by the state information is different from the appearance state of the target area Ta in the training image Ti used when the first learning model Md1 is learned, the control information acquisition unit 524i executes the following process. In this case, the control information acquisition unit 524i inputs the captured images Im1 to Im4 to the first learning model Md1 after transfer learning, which is performed using a plurality of teacher data in which the vehicle 10 and the training image Ti including the target area Ta in the appearance state specified by the state information are associated with the area correct label Lr. On the other hand, when the appearance state of the target area Ta specified by the state information is the same as the appearance state of the target area Ta in the training image Ti used when the first learning model Md1 is learned, the control information acquisition unit 524i executes the following process. In this case, the control information acquisition unit 524i inputs the captured images Im1 to Im4 to a first learning model Md1 that has not undergone transfer learning using multiple teacher data that associates a training image Ti including a target area Ta of the vehicle 10 and the appearance state identified by the state information with an area correct label Lr.

[0193] Fig. 36 is a flowchart showing a transfer learning method of the first learning model Md1. The flow shown in Fig. 36 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0194] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S291). The external camera 9 that has received the request signal transmits the captured images Im1 to Im4 to the information processing device 5i (step S292).

[0195] After the change information acquisition step, a learning step is executed. In the learning step, the learning unit 522i uses the captured images Im1 to Im4 to determine whether the appearance state of the target region Ta has changed since the first learning model Md1 stored in the storage unit 53i was learned. When it is determined that the appearance state of the target region Ta has changed since the first learning model Md1 stored in the storage unit 53i was learned (step S293: Yes), the learning unit 522i transfer-learns the first learning model Md1 (step S294). On the other hand, when it is determined that the appearance state of the target region Ta has not changed since the first learning model Md1 stored in the storage unit 53i was learned (step S293: No), the learning unit 522i ends the flow without transferring learning the first learning model Md1.

[0196] Fig. 37 is a flowchart showing a method for acquiring shape data Da in the 11th embodiment. The flow shown in Fig. 37 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0197] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S391). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5i (step S392). The state information acquisition unit 523i analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S393).

[0198] After the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524i acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 that has been trained using a training image Ti including the vehicle 10 and the target area Ta of the appearance state specified by the state information. In detail, when the appearance state specified by the state information is different from the appearance state of the target area Ta in the training image Ti used when the first learning model Md1 stored in the storage unit 53i is trained (step S394: Yes), the control information acquisition unit 524i executes the following process. In this case, the control information acquisition unit 524i inputs the captured images Im1 to Im4 to the first learning model Md1 after transfer learning that has been trained using a plurality of teacher data that associates the training image Ti including the target area Ta of the appearance state specified by the vehicle 10 and the state information with the area correct label Lr. As a result, the control information acquisition unit 524i acquires the shape data Da (step S395). On the other hand, if the appearance state specified by the state information is the same as the appearance state of the target area Ta in the training image Ti used when learning the first learning model Md1 stored in the storage unit 53i (step S394: No), the control information acquisition unit 524i executes the following process. In this case, the control information acquisition unit 524i acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 that has not been subjected to transfer learning using multiple teacher data that associates the vehicle 10 and the training image Ti including the target area Ta of the appearance state specified by the state information with the area correct label Lr (step S396).

[0199] According to the eleventh embodiment, the information processing device 5i can execute the following process when it is determined that the appearance state of the target area Ta has changed using the change information. In this case, the information processing device 5i can perform transfer learning on the first learning model Md1, which has been trained using a plurality of teacher data in which the training image Ti including the vehicle 10 and the target area Ta before the appearance state changes is associated with the area correct label Lr, using a plurality of teacher data in which the training image Ti including the vehicle 10 and the target area Ta after the appearance state changes is associated with the area correct label Lr. In this way, when the appearance state of the target area Ta changes, it is possible to suppress a decrease in detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0200] According to the eleventh embodiment, when the appearance state specified by the state information is different from the training image Ti used when the first learning model Md1 is learned, the information processing device 5i can acquire the shape data Da by executing the following process. In this case, the information processing device 5i can acquire the shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 that has been subjected to transfer learning using a plurality of teacher data in which the training image Ti including the target area Ta of the vehicle 10 and the appearance state specified by the state information is associated with the area correct label Lr. In this way, when the appearance state of the target area Ta is different from when the first learning model Md1 is learned, it is possible to suppress a decrease in accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0201] L. Twelfth embodiment: FIG. 38 is a diagram showing the configuration of an information processing system 1j in the twelfth embodiment. The information processing system 1j includes one or more vehicles 10, an external camera 9, and an information processing device 5j. In this embodiment, when it is determined that the appearance state of the target area Ta has changed, the information processing device 5j discards the first learning model Md1 and newly learns an eighth learning model Md8 as a machine learning model Md. The other configurations of the information processing system 1j are the same as those of the first embodiment unless otherwise described. The same reference numerals are given to the same configurations as those of the first embodiment, and descriptions thereof will be omitted.

[0202] The information processing device 5j includes a communication unit 51, a CPU 52j, and a storage unit 53j. The storage unit 53j of the information processing device 5j stores various information including various programs P5j that control the operation of the information processing device 5j and a first learning model Md1.

[0203] The CPU 52j of the information processing device 5j functions as a change information acquisition unit 521, a learning unit 522j, a status information acquisition unit 523, a control information acquisition unit 524j, a calculation unit 525, and a remote control unit 526 by deploying various programs P5j stored in the memory unit 53j.

[0204] The learning unit 522j uses the change information to determine whether the appearance state of the target area Ta has changed since the first learning model Md1 was learned. If it is determined that the appearance state of the target area Ta has changed, the learning unit 522j executes the following process. In this case, the learning unit 522j discards the first learning model Md1 learned using the initial learning dataset. The learning unit 522j then uses a new learning dataset to learn a new eighth learning model Md8. The new learning dataset includes a plurality of teacher data in which a training image Ti including the vehicle 10 and the target area Ta after the appearance state has changed is associated with an area correct label Lr. For example, a CNN that realizes either semantic segmentation or instance segmentation can be used for the eighth learning model Md8. During learning of the CNN, for example, the parameters of the CNN are updated by backpropagation (error backpropagation method) so as to reduce the error between the output result by the eighth learning model Md8 and the area correct label Lr. The learning unit 522j stores the created eighth learning model Md8 in the memory unit 53j.

[0205] The control information acquisition unit 524j uses the state information to specify the appearance state of the target region Ta, and inputs the captured images Im1 to Im4 to the machine learning model Md according to the appearance state of the target region Ta to acquire the shape data Da. Specifically, when the appearance state of the target region Ta specified by the state information is different from the appearance state of the target region Ta in the training image Ti used when the first learning model Md1 is learned, the control information acquisition unit 524j executes the following process. In this case, the control information acquisition unit 524j inputs the captured images Im1 to Im4 to the eighth learning model Md8 to acquire the shape data Da. On the other hand, when the appearance state of the target region Ta specified by the state information is the same as the appearance state of the target region Ta in the training image Ti used when the first learning model Md1 is learned, the control information acquisition unit 524j executes the following process. In this case, the control information acquisition unit 524j inputs the captured images Im1 to Im4 to the first learning model Md1 to acquire the shape data Da.

[0206] Fig. 39 is a flowchart showing a learning method of the learning models Md1 and Md8 in the twelfth embodiment. The flow shown in Fig. 39 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0207] In the change information acquisition step, the change information acquisition unit 521 transmits a request signal for acquiring the captured images Im1 to Im4 as change information to the external camera 9 (step S301). The external camera 9 that has received the request signal transmits the captured images Im1 to Im4 to the information processing device 5i (step S302).

[0208] After the change information acquisition step, a learning step is executed. In the learning step, the learning unit 522j uses the captured images Im1 to Im4 to determine whether the appearance state of the target area Ta has changed since the first learning model Md1 stored in the storage unit 53j was learned. If it is determined that the appearance state of the target area Ta has changed since the first learning model Md1 stored in the storage unit 53j was learned (step S303: Yes), the learning unit 522j discards the first learning model Md1 (step S304). Then, the learning unit 522j newly learns the eighth learning model Md8 (step S305). On the other hand, if it is determined that the appearance state of the target area Ta has not changed since the first learning model Md1 stored in the storage unit 53j was learned (step S303: No), the learning unit 522j discards the first learning model Md1 and ends the flow without newly learning the eighth learning model Md8.

[0209] Fig. 40 is a flowchart showing a method for acquiring shape data Da in the twelfth embodiment. The flow shown in Fig. 40 is repeatedly executed at predetermined time intervals from the time when the vehicle 10 starts traveling by unmanned driving, for example.

[0210] In the state information acquisition step, the state information acquisition unit 523 transmits a request signal to the external camera 9 to acquire the captured images Im1-Im4 as state information (step S401). The external camera 9 that has received the request signal transmits the captured images Im1-Im4 to the information processing device 5j (step S402). The state information acquisition unit 523j analyzes the captured images Im1-Im4 to identify the appearance state of the target area Ta (step S403).

[0211] After the state information acquisition step, a control information acquisition step is executed. In the control information acquisition step, the control information acquisition unit 524j acquires shape data Da by inputting the captured images Im1 to Im4 to the machine learning model Md according to the appearance state of the target region Ta. In detail, when the appearance state specified by the state information is different from the appearance state of the target region Ta in the training image Ti used when learning the first learning model Md1 stored in the storage unit 53j (step S404: Yes), the control information acquisition unit 524j executes the following process. In this case, the control information acquisition unit 524j acquires shape data Da by inputting the captured images Im1 to Im4 to the eighth learning model Md8 (step S405). On the other hand, when the appearance state specified by the state information is the same as the appearance state of the target region Ta in the training image Ti used when learning the first learning model Md1 stored in the storage unit 53j (step S404: No), the control information acquisition unit 524j executes the following process. In this case, the control information acquisition unit 524j acquires the shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1 (Step S406).

[0212] According to the twelfth embodiment, the information processing device 5j can execute the following process when it is determined that the appearance state of the target area Ta has changed using the change information. In this case, the information processing device 5j can discard the first learning model Md1 trained using a plurality of teacher data in which the training image Ti including the vehicle 10 and the target area Ta after the appearance state has changed is associated with the area correct label Lr, and can newly train the eighth learning model Md8 using a plurality of teacher data in which the training image Ti including the vehicle 10 and the target area Ta after the appearance state has changed is associated with the area correct label Lr. In this way, when the appearance state of the target area Ta has changed, it is possible to suppress a decrease in detection accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0213] According to the twelfth embodiment, when the appearance state specified by the state information is different from the training image Ti used when the first learning model Md1 is learned, the information processing device 5j can acquire the shape data Da by executing the following process. In this case, the information processing device 5j can acquire the shape data Da by inputting the captured images Im1 to Im4 to the eighth learning model Md8 trained using a plurality of teacher data in which the training image Ti including the target area Ta of the vehicle 10 and the appearance state specified by the state information is associated with the area correct label Lr. In this way, when the appearance state of the target area Ta is different from when the first learning model Md1 is learned, it is possible to suppress a decrease in accuracy when detecting the vehicle 10 from the captured images Im1 to Im4.

[0214] M. Thirteenth embodiment: FIG. 41 is a diagram showing the configuration of an information processing system 1v in the thirteenth embodiment. The information processing system 1v includes one or more vehicles 10v, an external camera 9, and an information processing device 5v. In this embodiment, a vehicle control device 15v mounted on the vehicle 10v acquires vehicle position information using the detection result output from the external camera 300, and generates a driving control signal. The other configurations of the information processing system 1v are the same as those of the first embodiment unless otherwise described. The same reference numerals are used for the same configurations as those of the first embodiment, and description thereof will be omitted.

[0215] The information processing device 5v includes a communication unit 51, a CPU 52v, and a storage unit 53v. The storage unit 53v of the information processing device 5v stores various information including various programs P5v that control the operation of the information processing device 5v and a first learning model Md1.

[0216] The CPU 52v of the information processing device 5v deploys various programs P5v stored in the storage unit 53v, thereby functioning as a change information acquisition unit 521, a learning unit 522, a state information acquisition unit 523, and an update unit 529. The update unit 529 updates the first learning model Md1 stored in the vehicle control device 15v every time the first learning model Md1 is re-learned, so that the first learning model Md1 stored in the vehicle control device 15v is in the latest state.

[0217] The vehicle control device 15v includes a CPU 150v, a storage unit 157v, and an input / output interface 159. The storage unit 157v of the vehicle control device 15v stores various information including various programs P15v that control the operation of the vehicle control device 15v and a reference route Ip.

[0218] The CPU 150v of the vehicle control device 15v functions as a control information acquisition unit 154, a calculation unit 155, a signal generation unit 156, and an operation control unit 153v by expanding various programs P15v stored in the storage unit 157v. The control information acquisition unit 154 acquires shape data Da by inputting the captured images Im1 to Im4 to the first learning model Md1. The calculation unit 155 acquires vehicle position information using the shape data Da. The signal generation unit 156 determines a target position to which the vehicle 10 should next head using the vehicle position information and the reference route Ip. Then, the signal generation unit 156 generates a driving control signal for driving the vehicle 10v toward the determined target position. The operation control unit 153v controls the actuator using the generated driving control signal to drive the vehicle 10.

[0219] Fig. 42 is a flowchart showing a method for driving the vehicle 10v in the information processing system 1v according to the thirteenth embodiment. The flow shown in Fig. 42 is repeatedly executed at predetermined time intervals from the time when the vehicle 10v starts driving by unmanned driving, for example.

[0220] The calculation unit 155 of the vehicle control device 15v mounted on the vehicle 10v acquires vehicle position information using the shape data Da (step S510). The signal generation unit 156 determines a target position to which the vehicle 10v should next move (step S520). The signal generation unit 156 generates a driving control signal for driving the vehicle 10v toward the determined target position (step S530). The operation control unit 153 controls the actuator using the generated driving control signal to drive the vehicle 10v at the acceleration and steering angle represented by the driving control signal (step S540). The vehicle control device 15v repeats the acquisition of vehicle position information, the determination of the target position, the generation of the driving control signal, and the control of the actuator at a predetermined cycle.

[0221] According to the thirteenth embodiment, the information processing system 1v can make the vehicle 10v run by autonomous control without remotely controlling the vehicle 10v by the information processing device 5v.

[0222] N. Other embodiments: N-1. Other embodiment 1: The element affecting the change in the appearance state of the target area Ta may be other than dirt occurring in the target area Ta. The element affecting the change in the appearance state of the target area Ta may be, for example, unevenness occurring in the target area Ta. The unevenness occurring in the target area Ta is, for example, a depression occurring in the passage Ro due to the road surface Rs being scraped due to an increase in the number of times the vehicle 10, 10v travels. The unevenness occurring in the target area Ta may be a protrusion occurring in the passage Ro due to the road surface Rs rising. In this form, the learning unit 522, 522a to 522j can train the machine learning model Md according to the occurrence of unevenness in the target area Ta. This can suppress a decrease in the accuracy of the control information output from the machine learning model Md when the shape of the road surface Rs changes. Furthermore, the control information acquisition unit 154, 524, 524c to 524j can cause the vehicle 10, 10v to travel in an unmanned driving manner by acquiring the control information using the machine learning model Md according to the occurrence of unevenness in the target area Ta.

[0223] N-2. Other embodiment 2: The element affecting the change in the appearance state of the target area Ta may be other than dirt occurring in the target area Ta. The element affecting the change in the appearance state of the target area Ta may be, for example, an object other than the vehicle 10 that is movably arranged in the target area Ta. That is, the element affecting the change in the appearance state of the target area Ta may be at least one of an object that is other than the vehicle 10 and is movably arranged in the target area Ta and an object that has been removed from the target area Ta. The object that is movably arranged in the target area Ta is, for example, a manufacturing facility, a road cone, a signboard, or other equipment whose arrangement can be appropriately changed. In this form, the information processing device 5, 5a to 5j, 5v can train the machine learning model Md according to the arrangement situation of the object in the target area Ta. This makes it possible to suppress a decrease in the accuracy of the control information output from the machine learning model Md when the arrangement situation of the object in the target area Ta changes. Furthermore, the information processing device 5, 5a to 5j, 5v can cause the vehicle 10, 10v to run in an unmanned driving manner by acquiring the control information using the machine learning model Md according to the arrangement situation of the object in the target area Ta.

[0224] N-3. Other embodiment 3: The factor affecting the change in the appearance state of the target area Ta may be other than dirt occurring in the target area Ta. The factor affecting the change in the appearance state of the target area Ta may be, for example, the road surface color of the passage Ro. The road surface color of the passage Ro may be, for example, the color of the demarcation line such as a white line, the paint color of the floor including the passage Ro, or the color of the paved road surface Rs. In this form, the information processing device 5, 5a to 5j, 5v can train the machine learning model Md according to the road surface color of the passage Ro. Thereby, for example, when the color of the demarcation line changes with the passage of time and the maintenance status of the demarcation line, such as becoming lighter due to aging deterioration or becoming darker due to repainting, the following can be realized. In this case, the accuracy of the driving control signal output from the machine learning model Md and the signal generation parameters can be suppressed from decreasing. In addition, when the paint color of the floor surface including the passage Ro changes over time and depending on the maintenance status of the floor surface, such as becoming lighter due to rubbing caused by an increase in the number of times the vehicle 10, 10v runs, becoming darker due to repainting, or changing the hue compared to before repainting, the following can be realized. In this case, it is possible to suppress a decrease in the accuracy of the driving control signal output from the machine learning model Md and the parameter for signal generation. In addition, when the color of the paved road surface Rs changes depending on the paving condition of the passage Ro, such as becoming darker due to repaving, it is possible to suppress a decrease in the accuracy of the driving control signal output from the machine learning model Md and the parameter for signal generation. Furthermore, the information processing device 5, 5a to 5j, 5v can make the vehicle 10, 10v run in an unmanned driving manner by acquiring control information using the machine learning model Md according to the road surface color of the passage Ro.

[0225] N-4. Other embodiment 4: At least some of the functions of the information processing devices 5, 5a to 5j, 5v may be one function of the vehicle control device 15, 15v, or one function of the external camera 9. For example, when all the functions of the information processing devices 5, 5a to 5j, 5v are realized by the external camera 9, the information processing devices 5, 5a to 5j, 5v can acquire the coordinate values ​​Cr1 to Cr4, Cb1 to Cb8 in the camera coordinate system without using a camera identifier.

[0226] N-5. Other embodiment 5: In each of the first to twelfth embodiments, the information processing devices 5, 5a to 5j execute the processes from obtaining the vehicle position information to generating the driving control signal. In contrast, at least a part of the processes from obtaining the vehicle position information to generating the driving control signal may be executed by the vehicle 10. For example, the following forms (1) to (3) may be used.

[0227] (1) The information processing devices 5, 5a to 5j may acquire vehicle position information, determine a target position to which the vehicle 10 should next head, and generate a route from the current location of the vehicle 10 represented in the acquired position information to the target position. The information processing devices 5, 5a to 5j may generate a route to a target position between the current location and the destination, or may generate a route to the destination. The information processing devices 5, 5a to 5j may transmit the generated route to the vehicle 10. The vehicle 10 may generate a driving control signal so that the vehicle 10 drives on the route received from the information processing devices 5, 5a to 5j, and control an actuator using the generated driving control signal.

[0228] (2) The information processing devices 5, 5a to 5j may acquire vehicle position information and transmit the acquired position information to the vehicle 10. The vehicle 10 may determine a target position to which the vehicle 10 should next head, generate a route from the current position of the vehicle 10 represented in the received position information to the target position, generate a driving control signal so that the vehicle 10 travels on the generated route, and control an actuator using the generated driving control signal.

[0229] (3) In the above-mentioned (1) and (2) modes, the vehicle 10 may be equipped with an internal sensor, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. The internal sensor is a sensor equipped in the vehicle 10. Specifically, the internal sensor may include, for example, a camera, LiDAR, a millimeter wave radar, an ultrasonic sensor, a GPS sensor, an acceleration sensor, a gyro sensor, and the like. For example, in the above-mentioned (1) mode, the information processing device 5, 5a to 5j may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor in the route when generating a route. In the above-mentioned (1) mode, the vehicle 10 may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor in the driving control signal when generating a driving control signal. In the above-mentioned (2) mode, the vehicle 10 may acquire the detection result of the internal sensor, and may reflect the detection result of the internal sensor in the route when generating a route ... driving control signal when generating a route.

[0230] (4) In the above thirteenth embodiment, an internal sensor may be mounted on the vehicle 10v, and a detection result output from the internal sensor may be used for at least one of generating a route and generating a driving control signal. For example, the vehicle 10v may acquire a detection result from the internal sensor, and when generating a route, may reflect the detection result from the internal sensor on the route. The vehicle 10v may acquire a detection result from the internal sensor, and when generating a driving control signal, may reflect the detection result from the internal sensor on the driving control signal.

[0231] (5) In the above thirteenth embodiment, the vehicle 10v acquires vehicle position information using the detection result of an external sensor provided at a location different from the vehicle 10v. In contrast, the vehicle 10v may be equipped with an internal sensor, and the vehicle 10v may acquire vehicle position information using the detection result of the internal sensor, determine a target position to which the vehicle 10v should next head, generate a route from the current location of the vehicle 10v represented in the acquired vehicle position information to the target position, generate a travel control signal for traveling along the generated route, and control the actuator using the generated travel control signal. In this case, the vehicle 10v can travel without using any detection result of the external sensor. The vehicle 10v may acquire a target arrival time or traffic congestion information from outside the vehicle 10v, and reflect the target arrival time or traffic congestion information in at least one of the route and the travel control signal.

[0232] (6) In each of the first to twelfth embodiments, the information processing devices 5, 5a to 5j automatically generate a driving control signal to be transmitted to the vehicle 10. In contrast, the information processing devices 5, 5a to 5j may generate a driving control signal to be transmitted to the vehicle 10 in accordance with the operation of an external operator located outside the vehicle 10. For example, an operator may operate a steering device including a display for displaying an image output from an external sensor, a steering wheel for remotely operating the vehicle 10, an accelerator pedal, a brake pedal, and a communication device for communicating with the information processing devices 5, 5a to 5j by wired communication or wireless communication, and the information processing devices 5, 5a to 5j may generate a driving control signal in accordance with the operation applied to the steering device.

[0233] N-6. Other embodiment 6: In each of the above embodiments, the vehicle 10, 10v may have a configuration capable of moving by unmanned driving, and may be in the form of a platform having the configuration described below, for example. Specifically, the vehicle 10, 10v may have at least the vehicle control device 15, 15v, the drive device 11, the steering device 12, and the braking device 13 in order to perform the three functions of "running", "turning", and "stopping" by unmanned driving. When the vehicle 10, 10v acquires information from the outside for unmanned driving, the vehicle 10, 10v may further have a communication device 14. That is, the vehicle 10, 10v capable of moving by unmanned driving may not be equipped with at least a part of interior parts such as a driver's seat and a dashboard, may not be equipped with at least a part of exterior parts such as a bumper and a fender, and may not be equipped with a body shell. In this case, the remaining parts such as the body shell may be attached to the vehicle 10, 10v before the vehicle 10, 10v is shipped from the factory, or the remaining parts such as the body shell may be attached to the vehicle 10, 10v after the vehicle 10, 10v is shipped from the factory without the remaining parts such as the body shell being attached to the vehicle 10, 10v. Each part may be attached from any direction such as the upper side, lower side, front side, rear side, right side, or left side of the vehicle 10, 10v, and may be attached from the same direction or from different directions. Note that the position of the platform shape may be determined in the same manner as the vehicle 10, 10v in each of the above embodiments.

[0234] N-7. Other embodiment 7: The vehicle 10, 10v may be manufactured by combining a plurality of modules. The module means a unit composed of a plurality of parts grouped according to the part or function of the vehicle 10, 10v. For example, the platform of the vehicle 10, 10v may be manufactured by combining a front module constituting the front part of the platform, a central module constituting the central part of the platform, and a rear module constituting the rear part of the platform. The number of modules constituting the platform is not limited to three, and may be two or less or four or more. In addition to or instead of the parts constituting the platform, parts constituting parts of the vehicle 10, 10v other than the platform may be modularized. The various modules may include any exterior parts such as a bumper or a grille, or any interior parts such as a seat or a console. In addition, not limited to the vehicle 10, 10v, any type of moving body may be manufactured by combining a plurality of modules. Such a module may be manufactured, for example, by joining a plurality of parts by welding or a fastener, or may be manufactured by integrally molding at least a part of the parts constituting the module as one part by casting. The molding method for integrally molding a single component, particularly a relatively large component, is also called gigacast or megacast. For example, the front module, the center module, and the rear module may be manufactured using gigacast.

[0235] N-8. Other embodiment 8: Transporting the vehicles 10, 10v by using the unmanned driving of the vehicles 10, 10v is also called "self-propelled transport." Also, the configuration for realizing self-propelled transport is also called a "vehicle remote-controlled autonomous driving transport system." Also, the production method for producing the vehicles 10, 10v by using self-propelled transport is also called "self-propelled production." In self-propelled production, for example, at a factory where the vehicles 10, 10v are manufactured, at least a portion of the transport of the vehicles 10, 10v is realized by self-propelled transport.

[0236] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0237] 1, 1a to 1j, 1v...information processing system, 5, 5a to 5j, 5v...information processing device, 9...external camera, 10, 10v...vehicle, 10e...positioning point, 11...driving device, 12...steering device, 13...braking device, 14...communication device, 15, 15v...vehicle control device, 51...communication unit, 52, 52a to 52j, 52v...CPU of information processing device, 53, 53a to 53j, 53v...storage unit of information processing device, 54...notification unit, 55...input operation unit, 150, 150v...CPU of vehicle control device, 153, 153v...operation control unit, 154, 524, 524c to 524j...control information acquisition unit, 155, 525, 525e, 525f...calculation unit, 156...signal generation unit, 157, 157v...storage unit of vehicle control device, 159...input / output interface, 521, 521a...change information acquisition unit, 522, 522a to 522j...learning unit, 523, 523a...state information acquisition unit, 526...remote control unit, 527...notification control unit, 528...model acquisition unit, 529...update unit, A1...necessary area, A2...unnecessary area, Bi...background image, Br...real background image, Bv...virtual background image, C ...center of gravity, Cb1 to Cb8...3D coordinate values, Ci...synthetic image, Cr1 to Cr4...rectangular coordinate values, D...first distance, ΔD...observation error, Da...shape data, Db1...rectangular database, Db2...3D database, Dc...third distance, Do...observation distance, Dp...estimated distance, Dt...second distance, h...height of vehicle positioning point from road surface, H...height of external camera from road surface, Ic...cleaning information, Im1...original image, Im2...corrected image, Im3...rotated image, Im4...processed image, Im1 to Im4...captured image, Im5...first mask image, Im6: second mask image, Ip: reference path, Lc: contour correct label, Lr: area correct label, Md: machine learning model, Md1: first learning model, Md2, Md2a, Md2b: second learning model, Md3: third learning model, Md4: fourth learning model, Md5: fifth learning model, Md6: sixth learning model, Md7: seventh learning model, Md8: eighth learning model, Ms: mask area, P0: base coordinate point, P1: first coordinate point, P2: second coordinate point, P3: image coordinate point, P5, P5a to P5j, P5v, P15,P15v...program, Pa...camera parameters, Pc...imaging coordinate point, Pf...fixed coordinate point, Pv...vehicle coordinate point, R0...base circumscribing rectangle, R1...first circumscribing rectangle, R2...second circumscribing rectangle, RG...imaging area, Rb...vehicle circumscribing rectangular parallelepiped, Rc...vehicle circumscribing rectangle, Ro...passage, Rs...road surface, Ta...target area, Ti...training image, V...movement vector, Vb1~Vb8, Vc1~Vc4...vertices, Vi...virtual vehicle image,

Claims

1. An information processing device used to move a movable body that can be moved by unmanned operation, a change information acquisition unit that acquires change information regarding a change in an appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; When it is determined that the appearance state has changed using the change information, a learning unit that, when an image acquired by imaging the target area and the moving body with the external camera is input, learns a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body moving by the unmanned driving and a parameter used to generate the control signal, using multiple teacher data that associates training images including the target area and the moving body after the appearance state has changed with a correct answer label; At least one of a notification control unit that notifies an administrator of learning request information for learning the machine learning model; and An information processing device comprising:

2. An information processing device used to move a movable body that can be moved by unmanned operation, a learning unit that trains a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal when a target area included in an image capturing area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway and an image captured by the external camera of the moving body are input; The target region is classified into a plurality of appearance states according to changes in appearance; The learning unit trains the machine learning model for each of the appearance states using M (M is an integer of 2 or more) pieces of teacher data in which training images including the target region and the moving object classified into one of the appearance states are associated with correct labels, and N (N is an integer of 0 or more and less than M) pieces of teacher data in which training images including the target region and the moving object classified into the remaining other appearance states are associated with correct labels.

3. An information processing device used to move a movable body that can be moved by unmanned operation, a learning unit that trains a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used when generating the control signal when a target area included in an image capturing area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway, an image captured by capturing an image of the moving body by the external camera, and an external appearance state of the target area are input; The target region is classified into a plurality of the appearance states according to a change in appearance; The information processing device, wherein the learning unit trains the machine learning model using multiple teacher data that correspond to training images including the target region and the moving object, a correct answer label, and additional information indicating the appearance state of the target region contained in the training images.

4. An information processing device used to move a movable body that can be moved by unmanned operation, a state information acquisition unit that acquires state information regarding an appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a control information acquisition unit that, when an image acquired by imaging the target area and the moving body with the external camera is input, acquires at least one of a control signal that specifies the operation of the moving body moving by unmanned driving and a parameter used when generating the control signal by inputting the captured image to a trained machine learning model that outputs at least one of the control signal and a parameter, When the appearance state specified by the state information is different from the appearance state when the machine learning model was trained, the control information acquisition unit inputs the captured image to the machine learning model trained using a plurality of teacher data that associates training images including the target area and the moving object in the appearance state specified by the state information with a correct answer label.

5. An information processing device used to move a movable body that can be moved by unmanned operation, a state information acquisition unit that acquires state information regarding an appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a model acquisition unit that acquires a machine learning model that outputs at least one of a control signal that specifies the operation of the moving object moving by the unmanned driving and a parameter used when generating the control signal when an image acquired by imaging the target area and the moving object by the external camera is input; a control information acquisition unit that acquires at least one of the control signal and the parameter by inputting the captured image to the machine learning model acquired by the model acquisition unit; The target region is classified into a plurality of the appearance states according to a change in appearance; The machine learning model is prepared for each of the appearance states; Each of the multiple machine learning models prepared for each of the appearance states has been trained in advance using M pieces of teacher data (M is an integer of 2 or more) in which a training image including the target region and the moving object classified into one of the appearance states is associated with a correct label, and N pieces of teacher data (N is an integer of 0 or more and less than M) in which a training image including the target region and the moving object classified into the remaining other appearance states is associated with a correct label, The model acquisition unit acquires the machine learning model selected according to the appearance state from among a plurality of machine learning models prepared for each appearance state.

6. An information processing device used to move a movable body that can be moved by unmanned operation, a state information acquisition unit that acquires state information regarding an appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a control information acquisition unit that acquires at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used to generate the control signal when an image acquired by capturing an image of the target area and the moving body with the external camera and the appearance state specified by the state information are input to a trained machine learning model that outputs at least one of the control signal and a parameter used to generate the control signal by inputting the captured image and the appearance state specified by the state information; The target region is classified into a plurality of the appearance states according to a change in appearance; An information processing device, wherein the machine learning model has been pre-trained using a plurality of teacher data items that correspond to training images including the target region and the moving object, a correct answer label, and supplementary information indicating the appearance state of the target region contained in the training images.

7. The information processing device according to any one of claims 1 to 6, An information processing device, wherein the factors affecting the change in the appearance state are at least one of dirt occurring in the target area, unevenness occurring in the target area, an object other than the moving body that is movably arranged in the target area, and the road surface color of the passage.

8. The information processing device according to any one of claims 1 to 6, The machine learning model is learned during a period in which control of the unmanned driving is being performed.

9. The information processing device according to any one of claims 1 to 6, The training image is at least one of a synthetic image generated by synthesizing a virtual moving body image that virtually reproduces the moving body with either a real background image obtained by capturing an image of the target area using the external camera or a virtual background image that virtually reproduces the target area, and the captured image.

10. An information processing system used to move a movable body that can be moved by unmanned operation, The moving body; An external camera installed at a location different from the moving object; An information processing system comprising: the information processing device according to any one of claims 1 to 6.

11. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: a change information acquisition process for acquiring change information regarding a change in an appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; When it is determined that the appearance state has changed using the change information, a learning process for learning a machine learning model that outputs, when an image obtained by imaging the target area and the moving body with the external camera is input, at least one of a control signal that specifies the operation of the moving body moving by the unmanned driving and a parameter used in generating the control signal, using a plurality of teacher data that associates training images including the target area and the moving body after the appearance state has changed with a correct answer label; At least one of a notification step of notifying an administrator of learning request information for learning the machine learning model; An information processing method comprising:

12. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: a preparation step of preparing a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway, and a plurality of training images including the moving body; A learning process for training a machine learning model that outputs at least one of a control signal that specifies the operation of the moving object moving by the unmanned driving and a parameter used when generating the control signal when an image acquired by imaging the target area and the moving object by the external camera is input, The target region is classified into a plurality of appearance states according to changes in appearance; This information processing method comprises: in the learning step, using M pieces of teacher data (M is an integer of 2 or more) in which the training images including the target area and the moving object classified into one of the appearance states are associated with a correct label; and N pieces of teacher data (N is an integer of 0 or more and less than M) in which the training images including the target area and the moving object classified into the remaining other appearance states are associated with a correct label, the machine learning model is trained for each of the appearance states.

13. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: a preparation step of preparing a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway, and a plurality of training images including the moving body; a learning process for learning a machine learning model that outputs at least one of a control signal that specifies the operation of the moving body that moves by the unmanned driving and a parameter used when generating the control signal when a captured image obtained by capturing an image of the target area and the moving body by the external camera and an external appearance state of the target area are input; The target region is classified into a plurality of the appearance states according to a change in appearance; An information processing method in which, in the learning process, the machine learning model is trained using a plurality of teacher data items that correspond to the training image including the target region and the moving object, a correct answer label, and additional information indicating the appearance state of the target region contained in the training image.

14. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: a state information acquisition step of acquiring state information relating to an external appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a control information acquisition process for acquiring at least one of a control signal that specifies the operation of the moving body moving by the unmanned driving and a parameter used in generating the control signal, by inputting the captured image to a trained machine learning model that outputs at least one of the control signal and a parameter used in generating the control signal when a captured image obtained by capturing an image of the target area and the moving body by the external camera is input; In the information processing method, when the appearance state specified by the state information differs from the appearance state of the target region in a training image input when the machine learning model was trained, the control information acquisition process inputs the captured image to the machine learning model trained using a plurality of teacher data that associates training images including the target region and the moving object in the appearance state specified by the state information with correct labels.

15. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: a state information acquisition step of acquiring state information relating to an external appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a model acquisition process for acquiring a machine learning model that outputs, when an image acquired by imaging the target area and the moving body by the external camera is input, at least one of a control signal that specifies the operation of the moving body that moves by the unmanned driving and a parameter used when generating the control signal; and a control information acquisition step of acquiring at least one of the control signal and the parameter by inputting the captured image to the machine learning model acquired in the model acquisition step, The target region is classified into a plurality of the appearance states according to a change in appearance; The machine learning model is prepared for each of the appearance states; Each of the multiple machine learning models prepared for each of the appearance states has been trained in advance using M pieces of teacher data (M is an integer of 2 or more) in which a training image including the target region and the moving object classified into one of the appearance states is associated with a correct label, and N pieces of teacher data (N is an integer of 0 or more and less than M) in which a training image including the target region and the moving object classified into the remaining other appearance states is associated with a correct label, In the model acquisition step, the machine learning model selected according to the appearance state is acquired from a plurality of machine learning models prepared for each appearance state.

16. An information processing method used to move a moving body that can be moved by unmanned driving, comprising: a state information acquisition step of acquiring state information relating to an external appearance state of a target area included in an imaging area of ​​an external camera installed at a location different from the moving body, the target area including a passageway along which the moving body moves and a peripheral area of ​​the passageway; a control information acquisition process for acquiring at least one of a control signal that specifies the operation of the moving body that moves by unmanned driving and a parameter used in generating the control signal when an image acquired by imaging the target area and the moving body with the external camera and the appearance state specified by the state information are input to a trained machine learning model that outputs at least one of the control signal and a parameter used in generating the control signal by inputting the captured image and the appearance state specified by the state information; The target region is classified into a plurality of the appearance states according to a change in appearance; An information processing method, wherein the machine learning model is pre-trained using multiple training data that associates training images including the target region and the moving object, a correct answer label, and additional information indicating the appearance state of the target region contained in the training images.

Images