Information processing device, information processing method, and information processing program
The information processing device uses a data converter and trained model to maintain object detection accuracy across sensors with varying specifications by minimizing errors, eliminating the need for retraining.
Patent Information
- Application Number
- JP2023075266
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-08
AI Technical Summary
Existing trained models for object detection on visible light image data face accuracy decreases when data from sensors with different specifications are used, making retraining costly and time-consuming.
An information processing device that includes a data converter and a trained model to convert data from one or more sensors, reducing errors by training the converter to handle data from sensors with different numbers, specifications, and types, thereby maintaining accuracy.
The solution effectively suppresses the decrease in accuracy of object detection processes when using data from different sensors, without the need for retraining the model.
Smart Images

Figure 2024159333000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]
[0002] Patent Document 1 discloses a technique for performing object identification processing based on the results of prior learning using learning data on far-infrared image data and converted image data obtained by tone-converting the far-infrared image data. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2021-163222 A Summary of the Invention [Problem to be solved by the invention]
[0004] In recent years, trained models that perform a predetermined process on input data, such as trained models that perform object detection processing on visible light image data captured by a visible light camera, have been developed. If such existing trained models can be used, development costs can be reduced. However, with existing trained models, for example, when using data obtained from a sensor with specifications different from the sensor used during learning, the accuracy of the execution result of the predetermined process may decrease.
[0005] In this case, re-training the existing trained model in order to suppress a decrease in accuracy of the execution results using the existing trained model may be difficult in terms of development costs, development time, etc.
[0006] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device, an information processing method, and an information processing program that can suppress a decrease in accuracy of the execution results of a specified process executed using an existing trained model. [Means for solving the problem]
[0007] An information processing device of a first aspect includes an acquisition unit that acquires first data output from one or more sensors that measure physical quantities of the surrounding environment, a generation unit that generates second data by converting the first data by inputting the first data to a data converter that is a trained model that outputs data converted from input data, and an execution unit that executes a predetermined process on the input data and outputs the execution result, and executes the predetermined process by inputting the second data to a data processor that is trained using data output from a sensor that differs from the one or more sensors in at least one of number, specifications, setting parameters, and type, and the data converter is trained so as to reduce an error derived based on the output of the data processor and ground truth data.
[0008] According to the information processing device of the first aspect, the data input to the data converter and the input data used in learning the data processor are obtained by sensors that differ in at least one of the number, specifications, setting parameters, and type. Also, according to the information processing device of the first aspect, the data converter is trained so that the error derived based on the output of the data processor and the correct answer data is reduced. Therefore, it is possible to suppress a decrease in the accuracy of the execution result of a specified process executed using an existing trained model.
[0009] A second aspect of the information processing device is the information processing device of the first aspect, wherein the first data is two-dimensional first image data captured by a first camera, and the data processor is a trained model that performs object detection processing as the defined processing on two-dimensional second image data captured by a second camera that differs from the first camera in at least one of number, specifications, setting parameters, and type.
[0010] According to the information processing device of the second aspect, it is possible to suppress a decrease in accuracy of a result of executing an object detection process on image data.
[0011] In a third aspect of the information processing device, the first image data is two-dimensional far-infrared image data captured by a far-infrared camera, and the second image data is two-dimensional visible light image data captured by a visible light camera.
[0012] According to the information processing device of the third aspect, it is possible to suppress a decrease in accuracy of a result of executing an object detection process on image data.
[0013] A fourth aspect of the information processing method is an information processing method in which a computer executes a process of acquiring first data output from one or more sensors that measure physical quantities of the surrounding environment, inputting the first data into a data converter that is a trained model that outputs data obtained by converting the input data to generate second data by converting the first data, executing a predetermined process on the input data, and outputting the execution result, the information processing method being a trained model in which a computer executes a process of executing the predetermined process by inputting the second data into a data processor that has been trained using data output from a sensor that differs from the one or more sensors in at least one of number, specifications, setting parameters, and type, the data converter being trained so as to reduce an error derived based on the output of the data processor and ground truth data.
[0014] An information processing program of a fifth aspect is a trained model that acquires first data output from one or more sensors that measure physical quantities of the surrounding environment, generates second data by converting the first data by inputting the first data into a data converter that is a trained model that outputs data obtained by converting the input data, performs a predetermined process on the input data, and outputs the execution result, and is an information processing program for causing a computer to execute a process of inputting the second data into a data processor that has been trained using data output from a sensor that differs from the one or more sensors in at least one of number, specifications, setting parameters, and type as input data, thereby executing the predetermined process, and the data converter has been trained so as to reduce an error derived based on the output of the data processor and ground truth data. Effect of the Invention
[0015] According to the present disclosure, it is possible to suppress a decrease in accuracy of the execution results of a specified process executed using an existing trained model. [Brief description of the drawings]
[0016] [Figure 1] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 13 is a diagram for explaining a data converter. [Diagram 3] FIG. 2 is a diagram for explaining an object detector. [Figure 4] 10 is a block diagram showing an example of a functional configuration of an information processing device in a learning phase. FIG. [Diagram 5] FIG. 11 is a diagram for explaining a learning process. [Figure 6] 13 is a flowchart illustrating an example of a learning process. [Figure 7] 1 is a block diagram showing an example of a functional configuration of an information processing device in an operation phase. FIG. [Figure 8] FIG. 11 is a diagram for explaining an object detection process. [Figure 9] 13 is a flowchart illustrating an example of an object detection process. [Figure 10] 13A and 13B are diagrams for explaining an object detection process according to a modified example. [Figure 11] 13A and 13B are diagrams for explaining an object detection process according to a modified example. [Figure 12] 13A and 13B are diagrams for explaining an object detection process according to a modified example. [Figure 13] FIG. 13 is a diagram for explaining a data converter according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Hereinafter, examples of embodiments for carrying out the technology of the present disclosure will be described in detail with reference to the drawings.
[0018] A hardware configuration of an information processing device 10 according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the information processing device 10 includes a central processing unit (CPU) 20, a read only memory (ROM) 21, a random access memory (RAM) 22, and a communication interface (I / F) 23. The CPU 20, the ROM 21, the RAM 22, and the communication interface (I / F) 23 are connected to each other via a bus 27 so as to be able to communicate with each other. An example of the information processing device 10 in the operation phase described below is a computer such as an in-vehicle ECU (Electronic Control Unit). Also, an example of the information processing device 10 in the learning phase described below is a computer such as a personal computer or a server computer.
[0019] The CPU 20 is an example of a processor, and executes various programs and controls various parts. That is, the CPU 20 reads out a program from the ROM 21 and executes the program using the RAM 22 as a working area.
[0020] The ROM 21 serving as a storage unit stores an information processing program 30, a data converter 32, and an object detector 34. The RAM 22 serves as a working area for temporarily storing programs and data. The communication I / F 23 is an interface for communicating with an external device.
[0021] The camera 11 is connected to the communication I / F 23. The camera 11 is, for example, an in-vehicle camera. The camera 11 is, for example, installed facing forward at the position of a rearview mirror inside the vehicle, and captures an image of the area ahead of the vehicle as an example of the surrounding environment of the vehicle. In a learning phase to be described later, the camera 11 does not need to be connected to the communication I / F 23.
[0022] The camera 11 is a far-infrared camera equipped with an image sensor that detects far-infrared rays, as an example of a sensor that measures a physical quantity of the surrounding environment and outputs two-dimensional data. The camera 11 outputs two-dimensional far-infrared image data obtained by capturing an image of the area ahead of the vehicle to the CPU 20 via the communication I / F 23. The two-dimensional far-infrared image data output from the camera 11 is an example of the first data and the first image data according to the disclosed technology.
[0023] The data converter 32 is a trained model that outputs data obtained by converting input data. As shown in Fig. 2, the data converter 32 according to this embodiment receives two-dimensional far-infrared image data as input, and converts the input far-infrared image data into two-dimensional image data to be input to the object detector 34. The data converter 32 is configured, for example, by a deep neural network. The data converter 32 is trained in a training phase to be described later.
[0024] The object detector 34 is a trained model that executes a predetermined process on input data and outputs the execution result. As shown in FIG. 3, the object detector 34 according to this embodiment is a trained model that has been trained in advance using visible light image data captured by a visible light camera as input data. The visible light image data is an example of data output from a sensor of a different type from the sensor included in the camera 11. An example of this sensor is an image sensor that detects visible light. In other words, the visible light image data is an example of two-dimensional second image data captured by a second camera of a different type from the first camera 11.
[0025] The object detector 34 is configured, for example, by a deep neural network. The object detector 34 performs object detection processing as a predetermined process on the input visible light image data, and outputs an object detection result as the execution result. Examples of the object detection result include the number of detected objects, the object position, the object size, type, and reliability. The dashed rectangle in the example of FIG. 3 indicates a rectangle that is tangent to the contour of the object detected by the object detector 34. The object detector 34 has internal parameters preset by machine learning using learning data.
[0026] Next, a functional configuration of the information processing device 10 in the learning phase according to this embodiment will be described with reference to Fig. 4 and Fig. 5. As shown in Fig. 4, the information processing device 10 in the learning phase includes an acquisition unit 40, a generation unit 42, an execution unit 44, and a learning unit 46. The CPU 20 executes the information processing program 30, thereby functioning as the acquisition unit 40, the generation unit 42, the execution unit 44, and the learning unit 46.
[0027] The acquisition unit 40 acquires learning data. The learning data according to the present embodiment is a set of far-infrared image data and answer data representing the position, size, type, etc. of an object present in the far-infrared image data. The acquisition unit 40 may acquire the learning data from an external system or from a storage device such as the ROM 21.
[0028] As shown in Fig. 5, the generation unit 42 generates image data by converting the far-infrared image data acquired by the acquisition unit 40 by inputting the far-infrared image data to the data converter 32. In the example of Fig. 5, this image data is indicated as "converted image".
[0029] As shown in FIG. 5, the execution unit 44 inputs the image data generated by the generation unit 42 to the object detector 34 to execute the object detection process.
[0030] 5, the learning unit 46 trains the data converter 32 so as to reduce an error derived based on the object detection result output from the object detector 34 and the correct answer data acquired by the acquisition unit 40 by the execution unit 44 executing the object detection process. Specifically, the learning unit 46 derives an error by inputting the object detection result output from the object detector 34 and the correct answer data acquired by the acquisition unit 40 into an error function. Then, the learning unit 46 trains the data converter 32 so as to minimize the derived error. Examples of the error function include a cross-entropy error function, a mean squared error function, and a mean absolute error function.
[0031] Next, the operation of the information processing device 10 in the learning phase according to the present embodiment will be described with reference to Fig. 6. The learning process shown in Fig. 6 is executed, for example, when a user inputs an instruction to start execution.
[0032] 6, the acquisition unit 40 acquires learning data. In step S12, the generation unit 42 inputs the far-infrared image data acquired in step S10 to the data converter 32, thereby generating image data by converting the far-infrared image data.
[0033] In step S14, the execution unit 44 executes the object detection process by inputting the image data generated in step S12 to the object detector 34. In step S16, the learning unit 46 updates the parameters of the data converter 32 so that the error derived based on the object detection result output from the object detector 34 by executing step S14 and the correct answer data acquired in step S10 is reduced, as described above. When the process of step S16 ends, the data learning process is completed. The above learning process is executed using a plurality of different learning data, and after the learning process using all the learning data is completed, the learning process is repeated a predetermined number of times using the same learning data. The CPU 20 executes the above learning process in a state where the internal parameters of the object detector 34 are fixed, i.e., without updating the internal parameters of the object detector 34.
[0034] Next, a functional configuration of the information processing device 10 in the operation phase according to this embodiment will be described with reference to Fig. 7 and Fig. 8. As shown in Fig. 7, the information processing device 10 in the operation phase includes an acquisition unit 50, a generation unit 52, an execution unit 54, and an output unit 56. The CPU 20 executes the information processing program 30 to function as the acquisition unit 50, the generation unit 52, the execution unit 54, and the output unit 56.
[0035] The acquisition unit 50 acquires far-infrared image data captured by the camera 11 from the camera 11 via the communication I / F 23.
[0036] As shown in Fig. 8, the generating unit 52 generates image data by converting the far-infrared image data acquired by the acquiring unit 50 by inputting the far-infrared image data to the data converter 32. In the example of Fig. 8, this image data is indicated as "converted image". This converted image data is an example of the second data and the second image data according to the disclosed technology.
[0037] As shown in FIG. 8, the execution unit 54 inputs the image data generated by the generation unit 52 to the object detector 34 to execute the object detection process.
[0038] The output unit 56 outputs the object detection result output from the object detector 34 as a result of the execution unit 54 executing the object detection process to an external device via the communication I / F 23. The object detection result is used, for example, for controlling automatic driving of the vehicle.
[0039] Next, the action of the information processing device 10 in the operation phase according to the present embodiment will be described with reference to Fig. 9. The object detection process shown in Fig. 9 is executed, for example, when the ignition switch of the vehicle is turned on or when autonomous driving starts.
[0040] 9, the acquisition unit 50 acquires far-infrared image data captured by the camera 11 from the camera 11 via the communication I / F 23. In step S22, the generation unit 52 inputs the far-infrared image data acquired in step S20 to the data converter 32, thereby generating image data by converting the far-infrared image data.
[0041] In step S24, execution unit 54 executes the object detection process by inputting the image data generated in step S22 to object detector 34. In step S26, output unit 56 outputs the object detection result output from object detector 34 by executing step S24 to an external device via communication I / F 23. When the process of step S26 ends, the object detection process ends. CPU 20 executes the above object detection process with the internal parameters of object detector 34 fixed, i.e., without updating the internal parameters of object detector 34.
[0042] As described above, according to this embodiment, the data converter 32 converts the far-infrared image data, and the converted image data is input to the object detector 34 to execute the object detection process. Therefore, even when using far-infrared image data captured by a camera of a different type from the camera that captured the visible light image data used for learning by the object detector 34, it is possible to suppress a decrease in the accuracy of the object detection result.
[0043] In the above embodiment, an example has been described in which visible light image data output from a sensor of a type different from that of the camera 11 is applied as input data for the object detector 34, but the disclosed technology is not limited to this aspect. The input data for the object detector 34 may be data output from a sensor that is different from the sensor that the acquisition unit 50 is to acquire in at least one of the number, specifications, setting parameters, and type.
[0044] 10, the input data of object detector 34 may be visible light image data, and camera 11 may be a grayscale camera that outputs grayscale image data. Also, for example, when camera 11 is a visible light camera, camera 11 and the camera that captured the visible light image data used for learning of object detector 34 may have different specifications, such as the focal length of the attached lens, or different setting parameters, such as resolution or sensitivity.
[0045] In the above embodiment, a LiDAR (Light Detection and Ranging) may be used instead of the camera 11. In this case, the LiDAR outputs a two-dimensional distance map, a reflectance map, or light and dark image data, or the like.
[0046] 11, the number of input data of the data converter 32 and the number of input data of the object detector 34 may be different. That is, the number of sensors from which the acquisition unit 50 acquires data may be different from the number of sensors that acquire input data used for training the object detector 34. FIG. 11 shows an example in which the camera 11A is a visible light camera and the camera 11B is a far-infrared camera. In this case, when a far-infrared camera is added to a system equipped with a visible light camera, the accuracy of the object detection process can be improved without re-training the object detector 34.
[0047] Also, as shown in Fig. 12, the number of input data of the object detector 34 may be greater than the number of input data of the data converter 32. Fig. 12 shows an example in which the camera 11A is a visible light camera and the camera 11B is a far-infrared camera. Fig. 12 also shows an example in which the input data of the object detector 34 is data output by three different types of sensors, namely, visible light image data, far-infrared image data, and a distance map. In this case, in a system including a visible light camera, a far-infrared camera, and a LiDAR, when the LiDAR becomes unavailable, it is possible to suppress a decrease in the accuracy of the object detection process without re-learning the object detector 34.
[0048] In the above embodiment, an example has been described in which the object detector 34 that performs object detection processing is applied as the data processor that performs a process determined for the input data, but the disclosed technology is not limited to this aspect. For example, the data processor may be an object identifier that identifies an object present in the visible light image data, which is the input data, from the image data. Also, for example, the data processor may be an image captioner that adds a caption to the visible light image data, which is the input data. For example, the data processor may be a segmentation device that performs segmentation processing on the visible light image data, which is the input data.
[0049] In the above embodiment, the input data of the data converter 32 is two-dimensional data, but the disclosed technology is not limited to this. For example, as shown in FIG. 13, the input data of the data converter 32 may be one-dimensional data such as voice data. In this case, instead of the camera 11, a voice input device 12 such as a microphone that outputs one-dimensional voice data is used. In addition, in this case, instead of the object detector 34, a voice recognizer 36 that executes voice recognition processing on the input data and outputs the execution result is used. In this case, even if the voice input device 12 is changed to a different type, for example, it is possible to suppress a decrease in the accuracy of the voice recognition processing without re-learning the voice recognizer 36.
[0050] In the above embodiment, the functional units that execute various processes of the information processing device 10 may be realized by a processor other than the CPU. Examples of the processor in this case include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) that is a processor having a circuit configuration designed exclusively for executing a specific process. Each functional unit of the information processing device 10 may be realized by one of these various processors, or may be realized by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements.
[0051] In the above embodiment, the information processing program 30 is stored (installed) in advance in the ROM 21, but the present invention is not limited to this. The information processing program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 30 may be downloaded from an external device via a network. [Explanation of symbols]
[0052] 10. Information processing device 11 Camera 12 Audio input device 20 CPU 30 Information Processing Program 32 Data Converter 34 Object Detector 36 Voice Recognizer 40, 50 acquisition department 42, 52 generation section 44, 54 Executive Division 46 Learning Department 56 Output section
Claims
1. an acquisition unit that acquires first data output from one or more sensors that measure physical quantities of the surrounding environment; A generation unit that generates second data by converting the first data by inputting the first data to a data converter that is a trained model that outputs data obtained by converting input data; a trained model that executes a predetermined process on input data and outputs an execution result, the trained model executing the predetermined process by inputting second data to a data processor that has been trained using data output from a sensor that is different from the one or more sensors in at least one of number, specifications, setting parameters, and type; Including, The data converter is trained so as to reduce an error derived based on the output of the data processor and the correct answer data. Information processing device.
2. the first data is two-dimensional first image data captured by a first camera; The data processor is a trained model that executes an object detection process as the predetermined process on two-dimensional second image data captured by a second camera that is different from the first camera in at least one of the number, specifications, setting parameters, and type. The information processing device according to claim 1 .
3. The first image data is two-dimensional far-infrared image data captured by a far-infrared camera, The second image data is two-dimensional visible light image data captured by a visible light camera. The information processing device according to claim 2 .
4. Acquire first data output from one or more sensors that measure physical quantities of the surrounding environment; A data converter is a trained model that outputs data obtained by converting input data, and generates second data by converting the first data. A trained model that executes a predetermined process on input data and outputs a result of the execution, the trained model executing the predetermined process by inputting second data to a data processor that has been trained using data output from a sensor that is different from the one or more sensors in at least one of number, specifications, setting parameters, and type. An information processing method in which a process is executed by a computer, comprising the steps of: The data converter is trained so as to reduce an error derived based on the output of the data processor and the correct answer data. Information processing methods.
5. Acquire first data output from one or more sensors that measure physical quantities of the surrounding environment; A data converter is a trained model that outputs data obtained by converting input data, and generates second data by converting the first data. A trained model that executes a predetermined process on input data and outputs a result of the execution, the trained model executing the predetermined process by inputting second data to a data processor that has been trained using data output from a sensor that is different from the one or more sensors in at least one of number, specifications, setting parameters, and type. An information processing program for causing a computer to execute a process, The data converter is trained so as to reduce an error derived based on the output of the data processor and the correct answer data. Information processing program.
Citation Information
Patent Citations
Sensor domain adaptation
CN115406449A
Machine learning device and image processing device
JP2022038287A
Training image generation method
JP2023018519A
Method and system for tuning a camera image signal processor for computer vision tasks
US20200389588A1
End to end differentiable machine vision systems, methods, and media
US20220301123A1
