Information processing device, information processing method, and program

Abstract

This information processing device includes an ISP control calculation unit, a data manipulation unit, and an inference unit. The ISP control calculation unit adjusts an ISP parameter for each frame. The data manipulation unit adjusts a RAW data manipulation method for each frame on the basis of the ISP parameter, and generates manipulated data for inference. The inference unit performs inference processing on the manipulated data. In an information processing method according to the present disclosure, the processing of the information processing device is executed by a computer. A program according to the present disclosure causes a computer to perform the processing of the information processing device.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] It is known that inference processing using AI (Artificial Intelligence), accuracy can be improved by inputting multiple different image processing results and taking a majority vote (data expansion during inference).

[0003] Special table 2022-504713 publication Special table 2022-533519 publication

[0004] In conventional systems, processed RGB images from a camera are transferred to a downstream GPU server where they are then expanded. However, expansion is limited to processes such as cropping, resizing, and flipping, making it impossible to adjust noise or fine-tune brightness.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that are capable of performing a wide variety of image processing.

[0006] According to the present disclosure, there is provided an information processing device having an ISP control calculation unit that adjusts ISP parameters for each frame, a data processing unit that adjusts a processing method for RAW data for each frame based on the ISP parameters and generates processed data to be inferred, and an inference unit that performs inference processing on the processed data. Also according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer, and a program that causes a computer to realize the information processing of the information processing device.

[0007] FIG. 1 is an explanatory diagram of time-of-inference data augmentation (TTA). FIG. 2 is a diagram illustrating an example configuration of an object detection system of the present disclosure. FIG. 3 is a diagram illustrating an example of ISP processing. FIG. 4 is a diagram illustrating an example of data augmentation correction. FIG. 5 is a diagram illustrating an example of filtering processing. FIG. 6 is a diagram illustrating an example of filtering processing. FIG. 7 is a diagram illustrating an example of ISP parameter control based on score entropy. FIG. 8 is a diagram illustrating an example of ISP parameter control based on score entropy. FIG. 9 is a diagram illustrating another example of ISP parameter control. FIG. 10 is a diagram illustrating a flow of object detection processing. FIG. 11 is a diagram illustrating setting of a data augmentation method using a license. FIG. 12 is a diagram illustrating system variations. FIG. 13 is a diagram illustrating system variations.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The explanation will be given in the following order: [1. Data augmentation during inference] [2. Configuration of object detection system] [3. Flow of object detection processing] [4. Setting of data augmentation method by license] [5. System variations] [6. Effects]

[0010] 1. Data Augmentation at Inference Time FIG. 1 is an explanatory diagram of data augmentation at inference time (TTA).

[0011] TTA performs data augmentation (rotation, flipping, cropping, etc.) to the original image I ０ From multiple images I １ , I ２ , …, I ＮThis technology improves inference accuracy by generating a set of metadata and integrating the inference results for each image. In the example shown in Figure 1, the inference results (metadata) for each image are integrated by majority vote, and the integrated inference result is output as the final data. TTA can easily improve inference accuracy, but it has low flexibility in data expansion because it performs data expansion on developed RGB images. In addition, since RGB images are transferred to a server via a network, issues such as increased data traffic and privacy risks arise.

[0012] The present disclosure has been made in consideration of the above-mentioned problems. In this disclosure, AI for inference processing is installed in a camera that serves as an edge device. RAW data is processed within the edge device while adjusting ISP (Image Signal Processor) parameters for each frame. This allows for flexible expansion. Since data expansion and inference processing are completed within the edge device, only the metadata, which is the final data, needs to be transferred to the server. This reduces data communication volume and avoids privacy risks. The inference system of the present disclosure will be described in detail below.

[0013] In the following description, an object detection system is described as an inference system, but the techniques disclosed herein are broadly applicable not only to object detection but also to other tasks such as image classification, segmentation, position correction, and filtering processing.

[0014] 2. Configuration of Object Detection System FIG. 2 is a diagram showing an example configuration of an object detection system DS according to the present disclosure.

[0015] The object detection system DS includes a camera 100 and a server 400. The camera 100 is an intelligent vision sensor equipped with AI for object detection. The camera 100 uploads the recognition results (metadata 505) of objects detected by the AI ​​to the server 400. The camera 100 utilizes the processing power of a DCPU (Data Center Processing Unit) 330 to perform data expansion in the time series direction, and uploads the metadata 505 with improved inference accuracy to the server 400.

[0016] "Performing data extension in the chronological direction" refers to a process of adjusting the processing method of RAW data 501 for each frame and sequentially generating similar image data 502. Multiple pieces of RAW data 501 captured at similar times have similar content. By performing data extension in the chronological direction, multiple pieces of image data 502 showing similar content are sequentially generated.

[0017] The AI ​​sequentially performs inference processing on each image data 502. The camera 100 integrates the inference results (metadata 504) acquired in a unit period to generate final metadata 505. The unit period is set based on the upload rate of the metadata 505. For example, if the upload rate is 1 fps, the unit period is 1 second.

[0018] The camera 100 is driven at a drive rate that is faster than the upload rate. For example, if the drive rate of the camera 100 is 30 fps, 30 processes (capture, develop, and expand data) are performed during one upload. By integrating the 30 inference results (metadata 504), highly accurate metadata 505 is generated. Note that, because the unit period is relatively short, there are no significant changes in the subject. Therefore, the metadata 505 can be considered to be an integration of the results of repeated inferences on the same subject.

[0019] In normal data extension, multiple similar images are generated from the same original image using different processing methods. In this case, there is only one image (original image) that is the source of processing. When data extension is performed in the chronological direction, the source data is multiple pieces of RAW data 501 that were taken at different times. Therefore, the processing content of "data extension in the chronological direction" is slightly different from that of normal data extension.

[0020] However, because the multiple pieces of RAW data 501 to be processed were acquired within a short period of time, they are data (data with the same content) without significant changes in the subject. Therefore, in the present disclosure, the process of generating multiple pieces of similar image data 502 (processed data) by adjusting the processing method of the RAW data 501 for each frame is referred to as "data expansion," in the sense that various similar data are generated using data with the same content as the original data.

[0021] The camera 100 does not upload the image data 502 itself, but rather uploads metadata 505 inferred from the image data 502. Uploading only the necessary data reduces data transfer delays, power consumption, and communication costs. Privacy risks are also avoided because personally identifiable information is not output externally.

[0022] The camera 100 has a CIS (Contact Image Sensor) 200 and an information processing device 300. The information processing device 300 performs development processing, data enhancement processing, and inference processing on RAW data 501 acquired by the CIS 200. The information processing device 300 outputs the object recognition results obtained by inference as metadata 505. The server 400 presents the metadata 505 or data obtained by processing the metadata 505 to the licensee US (see FIG. 13 ).

[0023] The information processing device 300 includes an ISP 310 , an inference unit 320 , a DCPU 330 , a memory 340 , a buffer 350 , and a communication unit 360 .

[0024] The ISP 310 performs processes such as development, processing, cropping, and noise removal on the RAW data 501 based on the ISP parameters 503. The development, processing, and cropping of the RAW data 501 can be adjusted for each frame based on the ISP parameters 503. The ISP 310 outputs the RAW data 501 that has been developed, etc., to the memory 340.

[0025] 3 is a diagram showing an example of processing by the ISP 310. In the example of FIG. 3, black level processing, defect value correction, white balance, demosaic, color reproduction, gamma correction, shading correction, distortion correction, cropping, and scaling are shown as examples of ISP processing. Parameters used in each process (see FIG. 3) are included in the ISP parameters 503. The various processes shown in FIG. 3 can be performed using known techniques.

[0026] The DCPU 330 acquires RAW data developed by the ISP 310 from the memory 340. The DCPU 330 performs data expansion processing on the data acquired from the memory 340. Algorithms and parameters related to the data expansion processing are included in the ISP parameters 503. The DCPU 330 adjusts the image processing method for each frame through data expansion. As a result, the DCPU 330 sequentially generates multiple pieces of image data 502 (processed data) in which the image content differs slightly for each frame. The DCPU 330 sequentially outputs the generated image data 502 to the memory 340.

[0027] The inference unit 320 sequentially acquires the image data 502 buffered in the memory 340. The inference unit 320 performs inference processing on the acquired image data 502 (processed data). In the case of the present disclosure, the inference processing is an object detection processing. The inference unit 320 recognizes objects included in the image data 502 using AI for object detection. The inference unit 320 outputs metadata 504 related to the object as the object recognition result (inference result).

[0028] The DCPU 330 includes a data expansion unit 331 , a data expansion correction unit 332 , a filtering unit 333 , and an ISP control calculation unit 334 .

[0029] The data expansion unit 331 acquires RAW data developed by the ISP 310 from the memory 340. The data expansion unit 331 performs processing (data expansion processing) such as movement, rotation, enlargement, and reduction on the acquired data based on the ISP parameters 503. Processing such as movement, rotation, enlargement, and reduction can be adjusted for each frame based on the ISP parameters 503. The data expansion unit 331 outputs image data 502 obtained by the data expansion processing to the memory 340.

[0030] The data expansion unit 331 and ISP 310 function as a data processing unit 390 that processes RAW data 501. The data processing unit 390 adjusts the processing method of the RAW data 501 for each frame based on ISP parameters 503, and generates image data 502 (processed data) of the object to be inferred. Examples of the ISP parameters 503 to be adjusted include a crop position, an NR (Noise Reduction) amount, a gamma value, a linear matrix value, and a scaling amount.

[0031] The data processing unit 390 adjusts the processing method for the RAW data 501 for each frame based on the ISP parameters 503. As a result, the data processing unit 390 performs data extension in the time series direction. Processing of the RAW data 501 can be performed by either the data extension unit 331 or the ISP 310. In the present disclosure, cropping is performed as the data extension process. The cropping is performed by the ISP 310.

[0032] The data expansion correction unit 332 performs a process to restore coordinate information that has been changed by data expansion. For example, when an image is cropped or enlarged under the control of the ISP 310, the data expansion correction unit 332 converts metadata of location information into the coordinates of the original image. By correcting the location information, it is possible to ensure consistency of location when the data is aggregated in the subsequent filtering unit 333.

[0033] FIG. 4 is a diagram illustrating an example of data expansion correction.

[0034] In the example of Fig. 4, the original image is subjected to cropping and scaling during the inference process. The data expansion correction unit 332 determines the vertical and horizontal image size (X, Y) of the original image, the offset coordinates of the crop position (x ｏｆｆｓｅｔ , y ｏｆｆｓｅｔ ), the horizontal and vertical image size after cropping (X ｃｒｏｐ , Y ｃｒｏｐ ), vertical and horizontal magnification ratio (scale_x, scale_y), input size to AI (X', Y'), and object detection position (x' ｍｉｎ , y′ ｍｉｎ , x′ ｍａｘ , y′ ｍａｘ ), the object position (x ｍｉｎ , y ｍｉｎ , x ｍａｘ , y ｍａｘ ) is calculated.

[0035] The above is an example of conversion when the inference unit 320 performs object detection. Similar conversion is also performed for other AI tasks involving position information (such as segmentation).

[0036] The filtering unit 333 aggregates the inference results (metadata 504) of objects for multiple frames most recently acquired. The filtering unit 333 acquires statistical values ​​of the inference results for multiple frames acquired by aggregation as metadata 505. In this way, the filtering unit 333 removes instability and outliers in the inference results and acquires optimized inference results.

[0037] 5 to 7 are diagrams showing an example of the filtering process.

[0038] In this example, the category, score, and position information of the object are acquired as the inference result (metadata 504) for each frame. The filtering unit 333 receives the inference result value of the current frame (time t) after data augmentation correction from the data augmentation correction unit 332. ｔ Get.

[0039] The buffer 350 holds past inference results for one frame or multiple frames. The filtering unit 333 filters past inference results for n frames {value ｔ－１－ｎ, ...value ｔ－１ The filtering unit 333 aggregates the inference results of the most recent (n+1) frames including the current frame, and outputs the aggregated results as an inference result value' ｔ Obtain as.

[0040] The aggregation method can be selected from arithmetic average, weighted average, maximum value, etc. The arithmetic method can be a known method shown in FIG. 6. In the example of FIG. 7, the filtering unit 333 averages the scores for the cat category for the past three frames. The filtering unit 333 obtains the center of gravity, which is the coordinates of the bounding box inferred for each frame weighted by the score, as the position information of the object (cat).

[0041] The ISP control calculation unit 334 controls and updates the ISP parameters 503 based on a certain index. As a result, the ISP control calculation unit 334 adjusts the ISP parameters 503 for each frame. The control index can be set arbitrarily by the system developer. The ISP parameters 503 may be controlled based on the entropy of the score (score entropy) or according to a simple arithmetic rule. Multiple predefined configurable values ​​for the ISP parameters 503 may be switched for each frame.

[0042] 8 to 10 are diagrams for explaining examples of control of the ISP parameter 503 based on the score entropy.

[0043] The ISP control calculation unit 334 adjusts the ISP parameters 503 in a direction that increases the reliability of the inference result. For example, the ISP control calculation unit 334 detects a direction in which the entropy of the image classification score acquired as the inference result (metadata 504) decreases as a direction in which the reliability of the inference result increases. Entropy is an index that represents the uncertainty of an event. The smaller the entropy, the more likely the event is to occur (the higher the probability of occurrence).

[0044] The entropy is calculated by the method shown in Fig. 9. In Fig. 9, "H" indicates entropy, and "p" indicates the category score. In the example of Fig. 9, a classification example of three classes ("person", "cat", and "dog") is shown.

[0045] In "case 1", the score of "person" is 0.1, the score of "cat" is 0.8, and the score of "dog" is 0.1. In "case 2", the score of "person" is 0.3, the score of "cat" is 0.4, and the score of "dog" is 0.3. In this case, the entropy H ｃａｓｅ１ is about 0.64, and the entropy H ｃａｓｅ２ is approximately 1.09. Therefore, the ISP control calculation unit 334 determines that "case 1" provides better results and adopts the set values ​​of "case 1."

[0046] For example, the ISP control calculation unit 334 calculates the score entropy of the current frame from the inference result of the current frame. The ISP control calculation unit 334 compares the calculation result with the score entropy of the previous frame. If the score entropy has increased compared to the previous frame, the ISP control calculation unit 334 selects a configurable value for the next ISP parameter 503 based on the setting information 335 and updates the ISP parameter 503 with the selected configurable value.

[0047] The setting information 335 defines a plurality of setting values ​​that can be set as the ISP parameters 503. If the change in entropy is greater than a reference value, the ISP control calculation unit 334 switches the current setting value of the ISP parameters 503 to the next setting value. If the change in entropy is equal to or less than the reference value, the setting value of the ISP parameters 503 is maintained. If the same setting value is used N times in a row, the ISP control calculation unit 334 changes the setting value of the ISP parameters 503 to the next setting value. The number of times "N" can be set arbitrarily by the system developer.

[0048] In the example of Figure 10, the crop position is adjusted for each frame based on the score entropy. In the example on the left side of Figure 10, the entropy is low, so the ISP parameters 503 (crop position) are not changed. When the entropy increases, as in the example in the center of Figure 10, the ISP parameters 503 are changed. The crop size is set to "+50" and the cropped area is zoomed. When the entropy decreases again, as in the example on the right side of Figure 10, the ISP parameters 503 are maintained from the previous frame.

[0049] FIG. 11 is a diagram illustrating another example of control of the ISP parameters 503. In FIG.

[0050] In the example of Fig. 11, a plurality of settable values ​​are adopted as the setting value of the ISP parameter 503 in turn for each frame. "N" indicates the frame number. When the setting value reaches the last settable value, the first settable value is adopted as the setting value for the next frame. The ISP parameter 503 to be controlled is arbitrary. In the example of Fig. 11, a gamma value or a crop position is shown as an example of the ISP parameter 503 to be controlled.

[0051] 3. Flow of Object Detection Processing FIG. 12 is a diagram showing the flow of object detection processing.

[0052] In the example of Fig. 12, the functions of the information processing device 300 are shared among a CPU (Central Processing Unit) 370, an ISP 310, a DSP (Digital Signal Processor) 380, and a DCPU 330. When the CPU 370 issues a start instruction, the ISP 310 performs processing such as development (including data processing related to data expansion) based on ISP parameters 503. The DSP 380 performs inference processing (object detection) using AI for object detection and outputs the inference results as metadata 504. The DCPU 330 performs data expansion correction and filtering processing on the metadata 504.

[0053] The DSP 380 adjusts the setting values ​​of the ISP parameters 503 based on the metadata 504. The DSP 380 updates the ISP parameters 503 with the adjusted setting values. The adjustment process of the ISP parameters 503, the data processing process (data expansion), the inference process, and the filtering process are repeated a preset number of times (specified number of times).

[0054] The specified number of times is determined by the upload rate. For example, if the upload rate is 1 fps and the drive rate of the camera 100 is 30 fps, the above process is repeated 30 times. The CPU 370 determines whether to output the metadata 505 based on whether the number of times the process has been repeated has reached the specified number.

[0055] When the number of times of processing reaches a predetermined number, the CPU 370 determines to output the filtered metadata 505. The communication unit 360 outputs the metadata 505 obtained after repeating the inference process a number of times according to the upload rate to the server 400 as upload data.

[0056] 4. Setting of Data Expansion Method by License FIG. 13 is a diagram for explaining setting of a data expansion method by license.

[0057] Because the camera 100 has a cloud communication function, various data extension methods can be set according to the license of the user (licensee US). For example, the following settings can be adjusted for each licensee US based on the license. The communication unit 360 outputs upload data at the upload rate specified in the license.

[0058] (i) ISP parameters 503 to be controlled and the granularity of adjustment, (ii) frame rate of data expansion (sensor drive rate), and (iii) selection of control algorithm for ISP parameters 503.

[0059] The server 400 has a license confirmation unit 410 (see FIG. 2). The license confirmation unit 410 confirms the license agreed upon with the licensee US and transmits instruction data 509 corresponding to the license to the camera 100. The instruction data 509 includes the settings for (i) to (iii) above. The ISP control calculation unit 334 adjusts the ISP parameters 503 of the type specified in the license based on the instruction data 509, using the control algorithm and granularity specified in the license.

[0060] For example, the information processing device 300 adjusts the ISP parameters 503 shown in (i) above for each frame using the control algorithm shown in (iii) above, and performs data extension in the time series direction. The information processing device 300 drives the camera 100 at the sensor drive rate shown in (ii) above, and outputs the metadata 505 obtained after repeating the inference process a number of times according to the sensor drive rate as upload data.

[0061] 5. System Variations FIGS. 14 to 16 are diagrams showing system variations.

[0062] The camera 100 in Fig. 14 has the same structure as that shown in Fig. 2. In the example in Fig. 14, the data expansion unit 331 shown in Fig. 2 is omitted. Processing related to data expansion is performed by the ISP 310. The sensor drive rate is 30 fps, and the upload rate is 1 fps. The filtering method uses the maximum value of the inference results for 30 frames as the metadata 505 for upload. The ISP parameters 503 to be controlled are the crop position and the noise reduction amount.

[0063] In the camera 500 of FIG. 15 , processing related to data expansion is performed by a data expansion unit 531 of the DCPU 530. The ISP 510 only performs development processing. The ISP parameters 503 are fixed and are not adjusted for each frame. The function of the ISP control calculation unit 334 of FIG. 14 is performed by an expansion control unit 534 of the DCPU 530. The expansion control unit 534 adjusts parameters related to data expansion (expansion parameters) for each frame. The data expansion unit 531 performs data expansion on images of the same frame based on the expansion parameters.

[0064] 15, the drive rate (sensor drive rate) of the CIS 200 and ISP 510 is 1 fps, and the drive rate of the inference unit 320 and DCPU 530 is 30 fps. The upload rate, filtering method, and extended parameters to be controlled are the same as in the example of FIG. 14.

[0065] The camera 500 in Fig. 16 has the same structure as the camera 500 in Fig. 15. In the example of Fig. 16, the CIS 200 and the ISP 510 are implemented on a sensor chip 551. The inference unit 320, the data expansion unit 531, the data expansion correction unit 332, the filtering unit 333, the expansion control unit 534, the buffer 350, and the communication unit 360 are implemented on a subsequent-stage SoC (System on Chip) 552.

[0066] The drive rate (sensor drive rate) of the sensor chip 551 is 1 fps, and the drive rate of the subsequent stage SoC 552 is 30 fps. The upload rate, filtering method, and expansion parameters to be controlled are the same as in the example of Fig. 15. In the example of Fig. 16, data expansion is also performed on images of the same frame.

[0067] [6. Effects] The information processing device 300 has an ISP control calculation unit 334, a data processing unit 390, and an inference unit 320. The ISP control calculation unit 334 adjusts the ISP parameters 503 for each frame. The data processing unit 390 adjusts the processing method of the RAW data 501 for each frame based on the ISP parameters 503, and generates image data 502 (processed data) of an object to be inferred. The inference unit 320 performs inference processing on the image data 502. In the information processing method disclosed herein, the processing of the information processing device 300 is executed by a computer. A program disclosed herein causes a computer to realize the processing of the information processing device 300.

[0068] According to this configuration, the raw data 501 is directly processed to generate image data 502 for inference, allowing for a wide variety of image processing to be performed.

[0069] The data processing unit 390 adjusts the processing method of the RAW data 501 for each frame based on the ISP parameters 503, thereby expanding the data in the time series direction.

[0070] According to this configuration, a wide variety of data extensions can be implemented.

[0071] The information processing device 300 includes a filtering unit 333. The filtering unit 333 aggregates inference results (metadata 504) for multiple frames most recently acquired. The filtering unit 333 acquires, as metadata 505, statistical values ​​of the inference results for multiple frames acquired by aggregation.

[0072] This configuration reduces variations in inference results.

[0073] The ISP control calculation unit 334 adjusts the ISP parameters 503 in a direction that increases the reliability of the inference results.

[0074] This configuration allows high-quality metadata 505 to be obtained.

[0075] The ISP control calculation unit 334 detects the direction in which the entropy of the image classification score acquired as the inference result (metadata 504) decreases as the direction in which the reliability of the inference result increases.

[0076] According to this configuration, the ISP parameters 503 are appropriately adjusted based on the entropy.

[0077] The information processing device 300 includes a communication unit 360. The communication unit 360 outputs, as upload data, metadata 505 obtained after repeating the inference process a number of times according to the upload rate.

[0078] This configuration allows high-quality metadata 505 to be obtained.

[0079] The communication unit 360 outputs the upload data at the upload rate specified in the license.

[0080] According to this configuration, the quality of the uploaded data can be adjusted by the license.

[0081] The ISP control calculation unit 334 adjusts the ISP parameters 503 of the type specified in the license with the control algorithm and granularity specified in the license.

[0082] According to this configuration, the quality of the uploaded data can be adjusted by the license.

[0083] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0084] [Additional Notes] The present technology may also be configured as follows. (1) An information processing device having an ISP control calculation unit that adjusts ISP parameters for each frame; a data processing unit that adjusts a processing method of RAW data for each frame based on the ISP parameters and generates processed data to be inferred; and an inference unit that performs inference processing on the processed data. (2) The information processing device described in (1) above, in which the data processing unit performs data extension in a time series direction by adjusting the processing method of the RAW data for each frame based on the ISP parameters. (3) The information processing device described in (2) above, in which the information processing device has a filtering unit that aggregates inference results for multiple frames most recently acquired and acquires, as metadata, statistical values ​​of the inference results for the multiple frames obtained by the aggregation. (4) The information processing device described in (3) above, in which the ISP control calculation unit adjusts the ISP parameters in a direction that increases the reliability of the inference results. (5) The information processing device according to (4), wherein the ISP control calculation unit detects a direction in which the entropy of the image classification score obtained as the inference result decreases as a direction in which the reliability of the inference result increases. (6) The information processing device according to any one of (3) to (5), wherein the information processing device has a communication unit that outputs the metadata obtained after repeating the inference process a number of times according to an upload rate as upload data. (7) The information processing device according to (6), wherein the communication unit outputs the upload data at the upload rate specified in a license. (8) The information processing device according to any one of (1) to (7), wherein the ISP control calculation unit adjusts the ISP parameters of a type specified in a license with a control algorithm and granularity specified in the license. (9) An information processing method executed by a computer, comprising: adjusting ISP parameters for each frame; adjusting a processing method for processing RAW data for each frame based on the ISP parameters, generating processed data to be inferred, and performing inference processing on the processed data.(10) A program that causes a computer to adjust ISP parameters for each frame, adjust a method for processing RAW data for each frame based on the ISP parameters, generate processed data to be inferred, and perform inference processing on the processed data.

[0085] 300 Information processing device 320 Inference unit 333 Filtering unit 334 ISP control calculation unit 360 Communication unit 390 Data processing unit 501 RAW data 502 Image data (processed data) 503 ISP parameters 505 Metadata

Claims

1. An information processing device having an ISP control calculation unit that adjusts ISP parameters for each frame, a data processing unit that adjusts the processing method of RAW data for each frame based on the ISP parameters and generates processed data to be inferred, and an inference unit that performs inference processing on the processed data.

2. The information processing device according to claim 1, wherein the data processing unit adjusts a processing method for the RAW data for each frame based on the ISP parameters, thereby performing data extension in the time series direction.

3. The information processing device according to claim 2, further comprising a filtering unit that aggregates inference results for a plurality of frames most recently acquired, and acquires statistical values ​​of the inference results for the plurality of frames acquired by the aggregation as metadata.

4. The information processing device according to claim 3, wherein the ISP control calculation unit adjusts the ISP parameters in a direction that increases the reliability of the inference result.

5. The information processing device according to claim 4, wherein the ISP control calculation unit detects a direction in which the entropy of the image classification score obtained as the inference result decreases as a direction in which the reliability of the inference result increases.

6. The information processing device according to claim 3, further comprising a communication unit that outputs the metadata obtained after repeating the inference process a number of times according to an upload rate as upload data.

7. The information processing device according to claim 6, wherein the communication unit outputs the upload data at the upload rate specified in a license.

8. The information processing device according to claim 1, wherein the ISP control calculation unit adjusts the ISP parameters of the type specified in the license using a control algorithm and granularity specified in the license.

9. An information processing method executed by a computer, comprising: adjusting ISP parameters for each frame; adjusting a method for processing RAW data for each frame based on the ISP parameters; generating processed data to be inferred; and performing inference processing on the processed data.

10. A program that causes a computer to adjust ISP parameters for each frame, adjust the processing method of RAW data for each frame based on the ISP parameters, generate processed data to be inferred, and perform inference processing on the processed data.

Images