Information processing device and program

Abstract

To provide an information processing apparatus and a program that can support a learned model in which data acquired under a variety of environments is processed.SOLUTION: An information processing apparatus 14 acquires image data which is input data. The information processing apparatus 14 acquires a first learned model and a second learned model. The information processing apparatus 14 acquires first output data output from the first learned model by inputting the image data into the first learned model. The information processing apparatus 14 acquires second output data output from the second learned model by inputting the image data into the second learned model. The information processing apparatus 14 generates change data for changing the first output data on the basis of the second output data. The information processing apparatus 14 controls a camera 12 which is a control object, on the basis of the generated change data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] Conventionally, a technique for providing data optimal for subsequent signal processing for identifying an object and improving the accuracy of identifying the object is known (see, for example, Patent Document 1). The technique of Patent Document 1 changes the parameters applied to the sensor when acquiring the image data when outputting the identification result of the image data output from the sensor, and sets the parameters so that the sensor acquires the image data based on the changed parameters.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, for example, consider a case where a trained model that outputs output data for input data is mounted on a signal processing apparatus when the input data is input. In this case, the trained model mounted on the signal processing apparatus is given, and it may not be allowed to change the trained model.

[0005] For example, consider a case where image data captured by a camera mounted on a vehicle is input data. In this case, the image in front of the vehicle captured by the camera can vary depending on the environment. Therefore, for example, it is difficult to process both image data captured at night and image data captured during the day by one object identification process or a trained model.

[0006] This invention has been made in view of the above problems, and aims to provide an information processing device and program that can support a trained model that processes data acquired under various environments. [Means for solving the problem]

[0007] To achieve the above objective, the information processing device according to the present invention includes: a data acquisition unit that acquires input data; a model acquisition unit that acquires a first trained model and a second trained model from a storage unit; a first control unit that acquires first output data output from the first trained model by inputting the input data acquired by the data acquisition unit into the first trained model acquired by the model acquisition unit; a second control unit that acquires second output data output from the second trained model by inputting the input data acquired by the data acquisition unit into the second trained model acquired by the model acquisition unit; and a second control unit that generates modification data to modify the first output data acquired by the first control unit based on the second output data acquired by the second control unit. The information processing device comprises a modification unit and a third control unit that controls an object to be controlled based on the modification data generated by the modification unit, wherein the first trained model is a trained model that has been pre-trained based on first training data which is a combination of the input data input to the first trained model and the first correct data of the first output data output from the first trained model, and the second trained model is a trained model that has been pre-trained based on second training data which is a combination of the input data input to the first trained model and the second correct data of related data related to the first output data output from the first trained model, so that the first output data output by the first trained model approaches the first correct data.

[0008] The object to be controlled according to the present invention is a camera, the input data is image data captured by the camera, the second output data output from the second trained model is related data representing identification data for camera parameters or a method for calculating camera parameters, and the third control unit can change the output data output from the first trained model by setting the camera parameters corresponding to the related data for the camera, which is the object to be controlled, and by controlling the camera to change the characteristics of the image data captured by the camera.

[0009] The second output data output from the second trained model according to the present invention is of the same type as the first output data output from the first trained model, and the third control unit can be controlled to modify the first output data output from the first trained model, which is the object to be controlled, based on the second output data output from the second trained model.

[0010] The data acquisition unit according to the present invention acquires first input data acquired by a first sensor and second input data acquired by a second sensor as input data; the first control unit acquires first output data output from the first trained model by inputting the first input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit; the second control unit acquires second output data output from the second trained model by inputting the second input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit; and the third control unit can control the system to modify the first output data output from the first trained model, which is the object to be controlled, based on the second output data output from the second trained model.

[0011] The program of the present invention comprises a computer, a data acquisition unit that acquires input data, a model acquisition unit that acquires a first trained model and a second trained model from a storage unit, a first control unit that acquires first output data output from the first trained model by inputting the input data acquired by the data acquisition unit into the first trained model acquired by the model acquisition unit, a second control unit that acquires second output data output from the second trained model by inputting the input data acquired by the data acquisition unit into the second trained model acquired by the model acquisition unit, a modification unit that generates modification data to modify the first output data acquired by the first control unit based on the second output data acquired by the second control unit, and the modification unit A program for causing a third control unit to control an object based on the generated modified data, wherein the first trained model is a trained model that has been pre-trained based on first training data which is a combination of the input data input to the first trained model and the first correct data of the first output data output from the first trained model, and the second trained model is a trained model that has been pre-trained based on second training data which is a combination of the input data input to the first trained model and the second correct data of related data associated with the first output data output from the first trained model, so that the first output data output by the first trained model approaches the first correct data. [Effects of the Invention]

[0012] According to the present invention, the effect is obtained that it is possible to support a trained model that processes data acquired under various environmental conditions. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic diagram showing an example of the configuration of an information processing device according to the first embodiment. [Figure 2]It is a diagram showing a configuration example of a computer of an information processing apparatus. [Figure 3] It is a diagram for explaining the outline of an information processing system. [Figure 4] It is a diagram for explaining image data and camera parameters. [Figure 5] It is a diagram showing an example of an information processing routine of the first embodiment. [Figure 6] It is a schematic diagram showing an example of the configuration of an information processing apparatus according to the second embodiment. [Figure 7] It is a diagram for explaining the outline of the first embodiment. [Figure 8] It is a diagram for explaining the outline of the first embodiment. [Figure 9] It is a diagram for explaining the outline of the second embodiment. [Figure 10] It is a diagram for explaining the outline of the second embodiment. [Figure 11] It is a diagram showing an example of an information processing routine of the second embodiment.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] In the present embodiment, a case where a camera is attached to a vehicle and the camera mounted on the vehicle sequentially images the front of the vehicle will be described as an example.

[0016] <First Embodiment> Referring to FIG. 1, an information processing system according to the first embodiment will be described. FIG. 1 shows a schematic diagram of an information processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the information processing system 10 includes a camera 12 and an information processing apparatus 14.

[0018] Camera 12 is installed on a vehicle, which is an example of a moving object. Camera 12 captures an image of the area in front of the vehicle. Camera 12 includes a sensor device 12A which includes an optical system such as a lens and an image sensor that outputs a signal according to the intensity of the received light, and a preprocessor 12B which is a so-called image engine that converts the signal output by sensor device 12A into an image file.

[0019] As shown in Figure 2, the information processing device 14 is implemented by a computer comprising a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer of the information processing device 14 also includes an input / output interface (I / F) 54 to which external devices and output devices are connected, and a read / write (R / W) unit 55 that controls the reading and writing of data to the recording medium 59. The computer also includes a network I / F 56 that connects to a network such as the Internet. The CPU 51, memory 52, storage unit 53, input / output I / F 54, R / W unit 55, and network I / F 56 are connected to each other via a bus 57.

[0020] The storage unit 53 can be implemented using a hard disk drive (HDD), solid state drive (SSD), flash memory, etc. The storage unit 53, as a storage medium, stores programs that enable the computer to function. The CPU 51 reads the programs from the storage unit 53, loads them into memory 52, and sequentially executes the processes contained within the programs.

[0021] When describing this information processing device 14 in terms of functional blocks divided according to the functional implementation means determined based on hardware and software, as shown in Figure 1, it comprises a data acquisition unit 140, a learned model storage unit 141, a data storage unit 142, a model acquisition unit 143, a first control unit 144, a second control unit 145, a modification unit 146, and a third control unit 147.

[0022] The data acquisition unit 140 acquires image data for each time point captured by the camera 12. The image data is an example of the input data of the present invention.

[0023] The trained model memory unit 141 stores the first trained model and the second trained model.

[0024] Figure 3 shows a diagram illustrating the overview of the information processing system 10 of the first embodiment. As shown in Figure 3, in the information processing system 10 of this embodiment, the camera parameters of the camera 12 are set. Specifically, in the information processing system 10 of this embodiment, when image data captured by the camera 12 is input to the first trained model M1, the first trained model M1 outputs an object recognition result as an example of first output data. The object recognition result is the recognition result of the object captured in the image data. For example, the first trained model M1 outputs an object recognition result as shown in Figure 4.

[0025] Figure 4 shows a diagram illustrating the image data and camera parameters. As shown in Figure 4, in Figure 4(A), no objects are recognized, and the number of detections is 0, and the confidence meter is 0. Note that the exposure time of camera 12 in Figure 4(A) is "Exposure Time 1," which represents a predetermined exposure time. In Figure 4(B), objects are recognized, and the number of detections is 2, and the confidence meter is 168. Note that the exposure time of camera 12 in Figure 4(B) is "Exposure Time 2." In Figure 4(C), objects are recognized, and the number of detections is 2, and the confidence meter is 193. Note that the exposure time of camera 12 in Figure 4(C) is "Exposure Time 3." In Figure 4(D), objects are recognized, and the number of detections is 1, and the confidence meter is 93. Note that the exposure time of camera 12 in Figure 4(D) is "Exposure Time 4."

[0026] As shown in Figure 4, if the exposure time, which is a camera parameter of camera 12, is different, the image data captured by camera 12 will be different, and the object recognition results obtained by the first trained model M1 will also be different. Of Figures 4(A) to (D), the most preferable recognition result is shown in Figure 4(C), and it can be said that the exposure time of 3 in Figure 4(C) is an appropriate camera parameter.

[0027] Therefore, the information processing system 10 of this embodiment sets the camera parameters of the camera 12 using the second trained model M2 so that the object recognition result output from the first trained model M1 approaches the correct answer. This allows the first trained model M1 to appropriately support the processing of image data acquired under various environments. A detailed explanation follows below.

[0028] The first pre-trained model M1 is pre-machine-trained based on the first training data, which is a combination of image data input to the first pre-trained model M1 and the first output data output from the first pre-trained model M1, which is the ground truth data of the object recognition result (hereinafter also simply referred to as "first ground truth data").

[0029] The second trained model M2 is pre-trained using second training data, which is a combination of image data input to the first trained model M1 and the ground truth data of related data associated with the object recognition result output from the first trained model M1 (hereinafter also simply referred to as "second ground truth data"). The second trained model M2 is pre-trained so that the object recognition result output by the first trained model M1 approaches the first ground truth data. The related data associated with the object recognition result is either the camera parameters themselves or the identification data of the camera parameter calculation method. In this embodiment, the case where the related data is the identification data of the camera parameter calculation method will be explained as an example.

[0030] In this embodiment, the second correct answer data of the related data included in the second training data is the correct answer data of the related data related to the object recognition result output from the first trained model M1.

[0031] For example, in Figures 4(A) to (D), (A) has a confidence meter of 0, (B) has a confidence meter of 168, (C) has a confidence meter of 193, and (D) has a confidence meter of 93. When the image data of (B) and (C) are input to the first trained model M1, the vehicle in front is detected, while when the image data of (A) and (D) are input to the first trained model M1, detection of the vehicle in front fails. In this embodiment, identification data of the camera parameters themselves or the method for calculating the camera parameters when the image data of Figures 4(A) to (D) is captured is included as related data in the second training data. Since the second training data includes identification data of the camera parameters themselves or the method for calculating the camera parameters when the image data of (A) and (D) is captured, the second trained model M2 will output identification data of the camera parameters themselves or the method for calculating the camera parameters that improves the camera parameters when image data that the first trained model M1 struggles with is captured.

[0032] Various types of data are stored in the data storage unit 142. Specifically, the data storage unit 142 stores image data captured by the camera 12, object recognition results output from the first trained model M1, identification data of the camera parameters themselves or the camera parameter calculation method output from the second trained model M2, and camera parameters calculated using the selected camera parameter calculation method. These various types of data are used in the parts described later.

[0033] The model acquisition unit 143 reads out the first trained model M1 and the second trained model M2 stored in the trained model storage unit 141.

[0034] The first control unit 144 inputs the image data acquired by the data acquisition unit 140 to the first trained model M1 acquired by the model acquisition unit 143, thereby acquiring the object identification result output from the first trained model M1. The object identification result acquired by the first control unit 144 is used for various vehicle control purposes.

[0035] The second control unit 145 inputs the image data acquired by the data acquisition unit 140 to the second trained model M2 acquired by the model acquisition unit 143, thereby acquiring identification data for the camera parameter calculation method output from the second trained model M2.

[0036] The modification unit 146 generates camera parameters, which are modification data for modifying the object identification result acquired by the first control unit 144, based on the identification data of the camera parameter calculation method acquired by the second control unit 145.

[0037] Specifically, if there are three camera parameter calculation methods, A, B, and C, the modification unit 146 selects a camera parameter calculation method based on which camera parameter calculation method the identification data output from the second trained model M2 indicates. For example, if the identification data output from the second trained model M2 represents A:0.8, B:0.1, and C:0.1, the modification unit 146 selects camera parameter calculation method A, which has the highest probability. Then, the modification unit 146 uses camera parameter calculation method A to calculate the camera parameter P A Calculate.

[0038] The third control unit 147 controls the camera 12, which is the object to be controlled, based on the camera parameters generated by the modification unit 146.

[0039] Specifically, the camera parameter P generated by the modification unit 146. A Set this for camera 12. Note that camera parameter P A This can be a parameter of either the sensor device 12A of the camera 12 or the preprocessor 12B. This changes the characteristics of the image data captured by the camera 12, making it possible to control the identification result output from the first trained model M1 to change.

[0040] <Operation of Information Processing System 10>

[0041] Next, the operation of the information processing system 10 according to this embodiment will be described. When the information processing system 10 mounted on the vehicle is started up and the camera 12 starts capturing images of the area in front of the vehicle, the information processing device 14 executes the information processing routine shown in Figure 5. Note that the information processing routine shown in Figure 5 is executed each time new image data is captured.

[0042] In step S100, the data acquisition unit 140 acquires image data captured by the camera 12.

[0043] In step S102, the model acquisition unit 143 acquires the first trained model M1 and the second trained model M2 stored in the trained model storage unit 141.

[0044] In step S104, the first control unit 144 inputs the image data acquired in step S100 to the first trained model M1 acquired in step S102, thereby obtaining the object recognition result output from the first trained model M1.

[0045] In step S106, the second control unit 145 inputs the image data acquired in step S100 to the second trained model M2 acquired in step S102, thereby obtaining identification data for the camera parameter calculation method output from the second trained model M2.

[0046] In step S108, the modification unit 146 identifies the camera parameter calculation method indicated by the identification data obtained in step S106. Then, the modification unit 146 generates the camera parameters using the identified camera parameter calculation method.

[0047] In step S110, the third control unit 147 sets the camera parameters generated in step S108 to the camera 12 and terminates the information processing routine.

[0048] As is clear from the above description, the information processing device 14 of this embodiment acquires image data, which is an example of input data. The information processing device 14 also acquires the first trained model M1 and the second trained model M2 from the trained model storage unit 141. The information processing device 14 acquires the object recognition result output from the first trained model M1 by inputting the image data into the first trained model M1. The information processing device 14 acquires the identification data of the camera parameter calculation method output from the second trained model M2 by inputting the image data into the second trained model M2. Based on the identification data of the camera parameter calculation method, the information processing device 14 generates camera parameters, which are change data for changing the object recognition result. The information processing device 14 then controls the camera to set the camera parameters based on the generated camera parameters. The first trained model M1 is a trained model that has been pre-trained based on first training data, which is a combination of image data input to the first trained model M1 and first correct answer data, which is the correct answer of the object recognition result output from the first trained model M1. Furthermore, the second pre-trained model M2 is a pre-trained model that is pre-trained so that the object recognition result output by the first pre-trained model M1 approaches the first ground truth data, based on second training data which is a combination of image data input to the first pre-trained model M1 and second ground truth data which is related data related to the object recognition result output from the first pre-trained model M1, specifically the camera parameters of the camera 12 that maximize the number of objects detected in the object recognition result of the first pre-trained model M1. This allows the pre-trained model to process data acquired under various environments. Specifically, if the first pre-trained model M1 is an object detector, the accuracy of the object recognition result output from the object detector can be improved by setting camera parameters for camera 12 that maximize the number of objects detected in the object recognition result of the first pre-trained model M1.

[0049] Furthermore, in order to mount a system that outputs object recognition results from image data in a vehicle, the system needs to be high-speed. In this regard, the technology disclosed in Patent Document 1 (for example, Figure 11 of Patent Document 1) has a simple parameter controller 340 that sets sensor parameters when detection results are obtained by the object classifier 332. Since the technology disclosed in Patent Document 1 controls the sensor by the output from this simple parameter controller 340, it is necessary to train the simple parameter controller 340, which is a machine learning model, with all kinds of scenes. For this reason, it is difficult to lighten the simple parameter controller 340, which is a machine learning model, and shorten the inference time to speed up processing in the technology disclosed in Patent Document 1. Moreover, it is practically difficult to collect training data with different sensor parameters for all kinds of scenes.

[0050] Therefore, it is possible to train the second pre-trained model M2 using only training data that improves the camera parameters when image data that the first pre-trained model M1 struggles with is captured. In this case, for example, data where the number of object detections in the object recognition result is already large, and including such data in the training data would not improve the object recognition result of the first pre-trained model M1, is not included in the training data. This makes it possible to lighten the load of the second pre-trained model M2. Furthermore, since only such data is included in the training data, the cost of collecting training data can be reduced.

[0051] Furthermore, even if the first pre-trained model M1 is given and cannot be changed, the accuracy of the object recognition results output from the first pre-trained model M1 can be improved by changing the camera parameters for camera 12. In particular, pre-trained models as object detectors are often developed individually in various places, making it difficult for third parties to modify or improve them. Also, making changes to a pre-trained model as an object detector would necessitate re-evaluating the signal processing device itself, so there may be circumstances where such changes should be avoided as much as possible. In contrast, according to the information processing system 10 of this embodiment, even if the first pre-trained model M1 is given and cannot be changed, the accuracy of the object recognition results output from the first pre-trained model M1 can be improved by changing the camera parameters for camera 12.

[0052] <Second Embodiment> Next, a second embodiment will be described. Note that in the configuration of the second embodiment, parts that are the same as those in the first embodiment are denoted by the same reference numerals and their description is omitted. In the information processing system 10 of the second embodiment, the second output data output from the second trained model M2 is of the same type as the first output data output from the first trained model M1. Furthermore, the information processing system 10 of the second embodiment differs from the first embodiment in that it controls the system to modify the first output data output from the first trained model M1 based on the second output data output from the second trained model M2.

[0053] Referring to Figure 6, the information processing system according to the second embodiment will be described. Figure 6 shows a schematic diagram of the information processing system 210 according to the second embodiment.

[0054] As shown in Figure 6, the information processing system 210 is comprised of a visible light camera 212, which is an example of a first sensor, a far-infrared camera 13, which is an example of a second sensor, and an information processing device 214.

[0055] The visible light camera 212 has the same configuration as the camera 12 of the first embodiment and acquires visible image data, which is image data that visualizes the reflection intensity in the visible light wavelength band. The visible image data is an example of the first input data.

[0056] The far-infrared camera 13 is a sensor that detects synchrotron radiation in the far-infrared region emitted by a subject and acquires far-infrared image data. The far-infrared image data is an example of second input data. The far-infrared camera 13 includes a sensor device 13A that contains an image sensor, and a preprocessor 13B, which is a so-called image engine that converts the signal output by the sensor device 13A into an image file.

[0057] The information processing device 214 of the second embodiment can be described in terms of functional blocks divided according to the functional implementation means determined based on hardware and software. As shown in Figure 6, it includes a data acquisition unit 240, a learned model storage unit 241, a data storage unit 142, a model acquisition unit 243, a first control unit 144, a second control unit 245, a modification unit 246, and a third control unit 247.

[0058] The data acquisition unit 240 acquires visible image data acquired by the visible light camera 212 and far-infrared image data acquired by the far-infrared camera 13 as input data.

[0059] The trained model storage unit 241 stores a first trained model M1 and a second trained model M22, similar to those in the first embodiment.

[0060] Figures 7 to 10 show diagrams illustrating the overview of the information processing system 10 of the second embodiment. Figures 7 and 8 show diagrams illustrating the overview of the information processing system 10 of the first embodiment described above. Figure 7 is a diagram illustrating the learning phase in the information processing system 10 of the first embodiment, and Figure 8 is a diagram illustrating the operation phase in the information processing system 10 of the first embodiment.

[0061] As shown in Figure 7, in the learning phase of the first embodiment, the second trained model M2, which is undergoing machine learning, outputs identification data for the camera parameter calculation method. Then, machine learning is performed on the second trained model M2 so as to minimize the error between the identification data output from the second trained model M2 during learning and the identification data for the camera parameter calculation method that yields the highest confidence in the object recognition result (e.g., number of detected objects).

[0062] In this process, the image data ImmA captured when the camera parameters calculated by camera parameter calculation method A are set for camera 12, the image data ImB captured when the camera parameters calculated by camera parameter calculation method B are set for camera 12, and the image data ImC captured when the camera parameters calculated by camera parameter calculation method C are set for camera 12 are input to the first trained model M1. Then, from the object recognition results output from the first trained model M1, the identification data of the camera parameter calculation method that yields the highest confidence in the object recognition result (e.g., number of objects detected) is set as the second ground truth data.

[0063] Then, as shown in Figure 8, in the operation phase of the first embodiment, the second trained model M2 outputs identification data for the camera parameter calculation method. For example, as shown in Figure 8, if calculation method B is selected, the camera parameters calculated by calculation method B are set to camera 12. As a result, image data ImB captured with those parameters is input to the first trained model M1.

[0064] On the other hand, as shown in Figure 9, in the learning phase of the second embodiment, visible image data is input to the first trained model M1, and a first object identification result is output as the object identification result. At this time, as shown in Figure 9, the first object identification result is compared with the object's location, which is the correct first ground truth data, and weight data is output with weights added to the objects that were not detected by the first trained model M1. In addition, far-infrared image data is input to the second trained model M22, and a second object identification result is output as the object identification result. Next, as shown in Figure 9, the second object identification result is compared with the object's location, which is the correct second ground truth data, and the error is calculated. Then, further weight data is added to this error. As a result, the second trained model M22 is trained to detect objects that were not detected by the first trained model M1.

[0065] As shown in Figure 10, in the operation phase of the second embodiment, visible image data is input to the first trained model M1, and the first trained model M1 outputs a first object identification result. Far-infrared image data is input to the second trained model M22, and the second trained model M22 outputs a second object identification result. The first object identification result output from the first trained model M1 and the second object identification result output from the second trained model M22 are then integrated to obtain the final object identification result. In this regard, since the second trained model M22 is configured to preferentially detect objects that the first trained model M1 has difficulty detecting, the final object identification result may be a result in overcoming the detection of objects that the first trained model M1 has difficulty detecting. This will be explained in detail below.

[0066] The model acquisition unit 243 acquires the first trained model M1 and the second trained model M22 from the trained model storage unit 241.

[0067] The first control unit 144, similar to the first embodiment, inputs the visible image data acquired by the data acquisition unit 240 to the first trained model M1 acquired by the model acquisition unit 243, thereby acquiring the first object identification result output from the first trained model M1. The first object identification result is an example of the first output data.

[0068] The second control unit 245 inputs the far-infrared image data acquired by the data acquisition unit 240 to the second trained model M22 acquired by the model acquisition unit 243, thereby obtaining the second object identification result output from the second trained model M22. The second object identification result is an example of the second output data.

[0069] The modification unit 246 generates modification data to modify the first object identification result obtained by the first control unit 144, based on the second object identification result obtained by the second control unit 245. The modification data here is data in which the second object identification result has been modified so that it can be integrated with the first object identification result. For example, if the format of the first object identification result and the format of the second object identification result are different, integration will be difficult, so the modification data is generated by changing the format of the second object identification result to facilitate integration.

[0070] The third control unit 247 controls the first object identification result output from the first trained model M1, which is the object being controlled, to be modified based on the second object identification result output from the second trained model M22. For example, the third control unit 247 integrates the first object identification result and the second object identification result. Specifically, the third control unit 247 generates the final object identification result by summing the probabilities of each object region in the image, which are the first object identification result, and the probabilities of each object region in the image, which are the second object identification result.

[0071] <Operation of Information Processing System 210>

[0072] Next, the operation of the information processing system 210 according to the second embodiment will be described. When the information processing system 210 mounted on the vehicle is started up and the visible light camera 212 and the far-infrared camera 13 begin capturing images of the area in front of the vehicle, the information processing device 214 executes the information processing routine shown in Figure 11. Note that the information processing routine shown in Figure 11 is executed each time new image data is captured.

[0073] In step S200, the data acquisition unit 240 acquires visible image data captured by the visible light camera 212 and far-infrared image data captured by the far-infrared camera 13.

[0074] In step S202, the model acquisition unit 243 acquires the first trained model M1 and the second trained model M22 stored in the trained model storage unit 241.

[0075] In step S204, the first control unit 144 inputs the visible image data acquired in step S200 to the first trained model M1 acquired in step S202, thereby obtaining the first object recognition result output from the first trained model M1.

[0076] In step S206, the second control unit 245 inputs the far-infrared image data acquired in step S200 to the second trained model M22 acquired in step S202, thereby obtaining the second object identification result output from the second trained model M22.

[0077] In step S208, the modification unit 246 generates modification data to modify the first object identification result obtained in step S204, based on the second object identification result obtained in step S206.

[0078] In step S210, the third control unit 247 modifies the first object identification result output from the first trained model M1 based on the second object identification result output from the second trained model obtained in step S206. Specifically, the third control unit 247 generates the final object identification result by integrating the probabilities of each object region in the image, which are the first object identification result, with the modified data generated from the probabilities of each object region in the image, which are the second object identification result.

[0079] In step S210, the third control unit 247 outputs the final object identification result generated in step S210 as a result and terminates the information processing routine.

[0080] Furthermore, the other configurations and operations of the information processing system 210 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described.

[0081] As is clear from the above explanation, in the information processing device 214 of the second embodiment, the data output from the second trained model M22 is of the same type as the data output from the first trained model M1. The information processing device 214 acquires visible image data acquired by the visible light camera 212 and far-infrared image data acquired by the far-infrared camera 13 as input data. The information processing device 214 inputs the visible image data to the first trained model M1 and obtains the first object identification result output from the first trained model M1. The information processing device 214 inputs the far-infrared image data to the second trained model M22 and obtains the second object identification result output from the second trained model M22. Then, the information processing device 214 controls the first object identification result output from the first trained model M1, which is the object to be controlled, to correct it based on the second object identification result output from the second trained model M22. This allows us to support trained models that process data acquired under various environmental conditions, thereby improving the accuracy of object recognition results using the trained models.

[0082] Furthermore, according to the second embodiment, a second trained model M22 is generated that accurately detects objects that the first trained model M1 would falsely detect. Therefore, by integrating the first object identification result from the first trained model M1 with the second object identification result from the second trained model M22, the accuracy of the object identification result can be improved.

[0083] Furthermore, even if the first trained model M1 is given and cannot be modified, the accuracy of the object identification results output from the first trained model M1 can be improved by generating a second trained model M22 that accurately detects objects that the first trained model M1 misidentifies.

[0084] Furthermore, the second pre-trained model M22 is machine-trained using training data related to objects that the first pre-trained model M1 misidentifies, resulting in a lighter model (i.e., a relatively simpler network structure) compared to the first pre-trained model M1.

[0085] It should be noted that the present invention is not limited to the embodiments described above, and various modifications and applications are possible without departing from the spirit of the invention.

[0086] In the first embodiment described above, the case where the sensor is a camera 12 was used as an example, but the invention is not limited to this. Similarly, in the second embodiment described above, the case where the first sensor is a visible light camera 212 and the second sensor is a far-infrared camera 13 was used as an example, but the invention is not limited to this. The sensor can be of any type; for example, the sensor may be a laser radar.

[0087] Furthermore, although the second embodiment described above uses the example where the first sensor is a visible light camera 212 and the second sensor is a far-infrared camera 13, the invention is not limited to this. For example, the first sensor and the second sensor may be of the same type.

[0088] Furthermore, although this embodiment describes the case where the first trained model M1 is an object detector, it is not limited to this. The first trained model M1 can be anything that performs some kind of processing on the sensor data, which is the output of the sensor. For example, if the first trained model M1 is an object detector, the input data can be image data, and the output data can be object identification results (for example, at least one of the number of objects detected, the probability (or likelihood) of object detection, the position of the object, and the type of object). In this case, as described above, the number of objects detected can be used as the confidence level, which is an arbitrary index.

[0089] Furthermore, if the first trained model M1 is an object evaluator, when human image data is input, the output data may be a 5-point rating from "not beautiful" to "beautiful". In this case, the 5-point rating can be used as a confidence score, which is an arbitrary metric. For example, the data corresponding to the one with the highest 5-point rating may be used as the ground truth data.

[0090] Furthermore, if the first trained model M1 is a speech recognition system, the output data may be the transcript of the speech when audio signal data is input. In this case, the transcription error rate can be used as an arbitrary metric, such as confidence. For example, the data corresponding to the transcription with the smallest error rate may be used as the ground truth data.

[0091] The program of the present invention can be provided by storing it on a recording medium. [Explanation of symbols]

[0092] 10,210 Information Processing Systems 12 cameras 13. Far-infrared camera 14,214 Information Processing Devices 140,240 Data Acquisition Unit 141,241 Pre-trained model memory 142 Data Storage Unit 143,243 Model Acquisition Section 144 First Control Unit 145,245 Second Control Unit 146,246 Changes 147,247 Third Control Unit 212 Visible light camera

Claims

1. A data acquisition unit that acquires input data, A model acquisition unit that retrieves the first trained model and the second trained model from the memory unit, A first control unit inputs the input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, thereby acquiring the first output data output from the first trained model. A second control unit inputs the input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit, thereby acquiring the second output data output from the second trained model. A modification unit generates modification data to modify the first output data acquired by the first control unit, based on the second output data acquired by the second control unit, The system includes a third control unit that controls the object to be controlled based on the modified data generated by the modification unit, The first pre-trained model is a pre-trained model that has been trained in advance based on first training data, which is a combination of the input data input to the first pre-trained model and the first correct data of the first output data output from the first pre-trained model. The second pre-trained model is a pre-trained model that is pre-trained so that the first output data output by the first pre-trained model approaches the first ground truth data, based on second training data which is a combination of the input data input to the first pre-trained model and second ground truth data of related data associated with the first output data output from the first pre-trained model. The second output data output from the second trained model is of the same type as the first output data output from the first trained model. The third control unit controls the first output data output from the first trained model, which is the object to be controlled, to modify the second output data output from the second trained model based on the second output data output from the second trained model. Information processing device.

2. The data acquisition unit acquires the first input data acquired by the first sensor and the second input data acquired by the second sensor as the input data. The first control unit inputs the first input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, thereby acquiring the first output data output from the first trained model. The second control unit inputs the second input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit, thereby acquiring the second output data output from the second trained model. The third control unit controls the first output data output from the first trained model, which is the object to be controlled, to modify the second output data output from the second trained model based on the second output data output from the second trained model. The information processing apparatus according to claim 1.

3. Computers, Data acquisition unit that acquires input data, A model acquisition unit that retrieves the first trained model and the second trained model from the memory unit. A first control unit inputs the input data acquired by the data acquisition unit to the first trained model acquired by the model acquisition unit, thereby acquiring the first output data output from the first trained model. A second control unit inputs the input data acquired by the data acquisition unit to the second trained model acquired by the model acquisition unit, thereby acquiring the second output data output from the second trained model. A modification unit that generates modification data to modify the first output data acquired by the first control unit based on the second output data acquired by the second control unit, and A third control unit controls the object to be controlled based on the modified data generated by the modification unit. It is a program designed to function as such. The first pre-trained model is a pre-trained model that has been trained in advance based on first training data, which is a combination of the input data input to the first pre-trained model and the first correct data of the first output data output from the first pre-trained model. The second pre-trained model is a pre-trained model that is pre-trained so that the first output data output by the first pre-trained model approaches the first ground truth data, based on second training data which is a combination of the input data input to the first pre-trained model and second ground truth data of related data associated with the first output data output from the first pre-trained model. The second output data output from the second trained model is of the same type as the first output data output from the first trained model. The third control unit controls the first output data output from the first trained model, which is the object to be controlled, to modify the second output data output from the second trained model based on the second output data output from the second trained model. program.

Images