Information processing apparatus and method, and computer readable storage medium

Abstract

The invention relates to an information processing apparatus and method, and a computer readable storage medium. The information processing apparatus includes a processing circuit configured to generate a saliency map of a sample picture based on a predetermined model for performing task processing on the sample picture, the saliency map reflecting a degree of attention to objects at different positions in the sample picture when the predetermined model performs the task processing, and generate a saliency map of the sample picture based on the saliency map. And adjusting parameters of an image signal processor for generating the sample image based on the saliency map and the annotation area in the sample image, so that the difference between the task processing result and the annotation value of the sample image meets a preset condition.

Description

Information processing device and method, and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 22, 2022, with application number 202210870640.2 and invention name “Information processing device and method, computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of information processing technology, and more particularly to tuning an image signal processor, and more particularly to an information processing device and method, and a computer-readable storage medium. Background Art

[0003] An image signal processor (ISP) is a low-level hardware image processing device that converts raw light signals captured by optical sensors into images that can be viewed by the human eye on various devices. It is widely used in current digital cameras, mobile phone cameras, and other devices, significantly impacting the quality of the final image. ISPs typically offer a large number of adjustable configuration parameters, and ISP manufacturers typically employ experts to fine-tune these parameters. ISP tuning typically targets human visual perception, such as texture clarity and visual noise. With the advancement of machine learning, a large number of images are being used in computer vision tasks. Consequently, a large number of ISP tuning methods have emerged for advanced computer vision tasks such as autonomous driving. Optimizing ISPs is currently a hot topic of research.

[0004] Summary of the Invention

[0005] A brief overview of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important aspects of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is simply to present certain concepts in a simplified form as a prelude to the more detailed description discussed later.

[0006] According to one aspect of the present disclosure, an information processing device is provided, which includes a processing circuit, which is configured to: generate a saliency map of a sample image based on a predetermined model that performs task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs task processing; and adjust parameters of an image signal processor that generates the sample image based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

[0007] In the information processing device according to the embodiment of the present disclosure, the saliency map does not require an expert but can be automatically generated based on a predetermined model and sample images. In addition, the saliency maps of different sample images are different, so the flexibility and pertinence are stronger. In the information processing device 100 according to the embodiment of the present disclosure, the ISP is not simplified, so it is closer to real-world applications. The saliency map contains decision-making process information in the predetermined model performing task processing (for example, the degree of importance attached to objects at different positions in the sample image when the predetermined model performs task processing). In the process of adjusting the parameters of the ISP, since the quality of the sample image changes, the saliency map changes more evenly and frequently than the task evaluation index. Therefore, the use of the saliency map can be more conducive to tuning the ISP parameters, and can achieve better results than directly tuning the ISP based on the task evaluation index, thereby improving the accuracy of task processing.

[0008] According to another aspect of the present disclosure, an information processing method is provided, including: generating a saliency map of a sample image based on a predetermined model for performing task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs task processing, and adjusting parameters of an image signal processor for generating the sample image based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

[0009] According to other aspects of the present invention, there are also provided computer program codes and computer program products for implementing the above-mentioned information processing method, as well as a computer-readable storage medium having recorded thereon the computer program codes for implementing the above-mentioned information processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to further illustrate the above and other advantages and features of the present invention, the following is a further detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings. The accompanying drawings, together with the detailed description below, are included in this specification and form a part of this specification. Elements with the same function and structure are represented by the same reference numerals. It should be understood that these drawings only depict typical examples of the present invention and should not be regarded as limiting the scope of the present invention. In the drawings:

[0011] FIG1 shows a functional module block diagram of an information processing device according to an embodiment of the present disclosure.

[0012] FIG2 is a schematic diagram illustrating an application scenario according to an embodiment of the present disclosure.

[0013] FIG3 is a schematic flowchart illustrating image signal processor tuning according to an embodiment of the present disclosure.

[0014] FIG4 shows the prediction results of the predetermined model for the sample image using the existing method and after the image signal processor is tuned based on the saliency score.

[0015] FIG5 shows a graph comparing the tuning based on the saliency score and the existing methods when the size of the test dataset varies.

[0016] FIG6 shows a graph comparing the saliency score-based fine-tuning with existing methods in difficult scenarios.

[0017] 7A to 7D are schematic diagrams showing the optimization of calculating saliency scores based on different objects and the optimization of existing methods.

[0018] FIG8 is a schematic diagram illustrating the effect of image signal processor optimization based on saliency scores calculated according to different importance masks according to an embodiment of the present disclosure.

[0019] FIG9 is a flowchart illustrating an example of the flow of an information processing method according to an embodiment of the present disclosure.

[0020] FIG. 10 is a block diagram showing an example structure of a personal computer that can be employed in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of an actual implementation are described in this specification. However, it should be understood that in the process of developing any such actual implementation, many implementation-specific decisions must be made in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is a routine task for those skilled in the art who benefit from the contents of this disclosure.

[0022] It is also necessary to explain here that, in order to avoid obscuring the present disclosure due to unnecessary details, the accompanying drawings only show the device structure and / or processing steps closely related to the scheme according to the present disclosure, while other details that are not closely related to the present disclosure are omitted.

[0023] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 shows a functional module block diagram of an information processing device 100 according to an embodiment of the present disclosure. As shown in Figure 1, the information processing device 100 includes: a processing unit 102, which can be configured to generate a saliency map of a sample image based on a predetermined model for performing task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs task processing; and an adjustment unit 104, which can be configured to adjust the parameters of an image signal processor for generating a sample image based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

[0025] The processing unit 102 and the adjustment unit 104 may be implemented by one or more processing circuits, which may be implemented as a chip, for example.

[0026] As an example, the input of the ISP is the original image captured by the optical sensor. For example, the original image can be a 24-bit Bayer image. However, the above is only an example, and those skilled in the art will understand that the original image can be an image in other forms. The output of the ISP (i.e., the image after ISP processing) is the sample image mentioned above. For example, the sample image can be a 3-channel 8-bit RBG image. However, the above is only an example, and those skilled in the art will understand that the sample image can be an image in other forms.

[0027] ISP can be implemented by an ISP simulator, for example. The input of the ISP simulator is the original image samples in the test data set. For example, the ISP acts as a black box, and the ISP simulator itself can also be a black box, without knowing its internal structure. Preferably, the ISP simulator can correspond to the hardware ISP of the Sony FUJI sensor, and its functions include at least one of demosaicing, white balance, noise reduction, sharpening, tone mapping, and bit length compression. The ISP simulator can simulate a specific hardware ISP, and its parameters correspond one-to-one with the hardware ISP. The effect obtained by a set of parameters on the ISP simulator is consistent with the effect on the corresponding hardware ISP.

[0028] For example, the annotated region in the sample image may correspond to the annotated region in the original image or the original image sample.

[0029] In the following text, unless otherwise specified, no distinction is made between an ISP and an ISP simulator, and no distinction is made between an original image and an original image sample.

[0030] As an example, the predetermined model can be a computer vision task model. The input of the computer vision task model is the sample image, and the output is the result of processing the corresponding task. The computer vision task model can be, for example, a deep learning model, such as a convolutional neural network (CNN). For example, a CNN can be a neural network trained to perform a specific task.

[0031] Task processing may include classification tasks, object detection tasks (ie, object detection tasks), etc. Other examples of task processing can be conceived in the art and will not be described here.

[0032] For different task processing, the same or different computer vision task models can be used.

[0033] In this disclosure, a saliency map is used to quantify information about the decision-making process during task processing by a predetermined model (for example, the degree of emphasis placed on objects at different locations in a sample image when the predetermined model performs task processing). Based on the saliency map, the results of task processing can be quantitatively optimized. By adjusting the parameters of the ISP based on the saliency map and the annotated regions in the sample image, and invoking the ISP to process the original image based on the new parameters, an updated sample image can be obtained. This means that the quality of the sample image output by the adjusted ISP changes. During the iterative adjustment of the ISP parameters, the quality of the sample image continuously changes, which will ensure that important regions in the saliency map (for example, regions corresponding to the objects to be detected) coincide with the annotated regions as closely as possible. Saliency maps change more evenly and frequently than task evaluation metrics. Therefore, using saliency maps can facilitate tuning ISP parameters, achieving better results than directly tuning based on task evaluation metrics. For example, for classification tasks, task evaluation metrics may include accuracy or F1 value; for object detection tasks, task evaluation metrics may include mAP (mean average precision). For example, based on the annotation information contained in the sample image, such as the annotation area, a task evaluation index corresponding to the task processing performed by the predetermined model on the sample image can be calculated.

[0034] For example, the information processing device 100 can call an optimizer to adjust the parameters of the ISP. The optimizer can be set in the information processing device 100 as a part of the information processing device 100, or the optimizer can be set outside the information processing device 100. The optimization goal of the optimizer is to tune the parameters of the ISP so that the difference between the result of the task processing and the labeled value of the sample image meets the predetermined conditions. For computer vision tasks, the goal of optimization is to enable the sample image output by the ISP to obtain a higher task evaluation index in the current computer vision task. The optimizer can be, for example, a CMA-ES (Covariance Matrix Adaptation Evolution Strategy) optimizer. For example, the information processing device 100 can randomly extract a set of numbers equal to the number of parameters inside the optimizer as the initial value of the optimizer. As an example, the information processing device 100 can call an optimizer, which can generate a set of multiple continuous numerical values ​​equivalent to the number of ISP parameters. That is, the optimizer generates numerical values ​​for the ISP parameters, and these numerical values ​​have a one-to-one correspondence with the ISP parameters. The generated multiple continuous numerical values ​​are processed according to the range and numerical type of the actual ISP parameters to make them meet the ISP parameter requirements. The processing performed on the multiple continuous numerical values ​​includes: truncating, scaling, or reflecting the numerical values ​​that exceed the corresponding parameter range to make them meet the parameter range requirements; if the parameter type is discrete, the continuous numerical values ​​are converted into discrete values ​​by rounding. Then, the processed numerical values ​​are used to set the ISP parameters, and the ISP is called to process the original image based on the new parameters to obtain an updated sample image.

[0035] As an example, those skilled in the art may predetermine the above-mentioned predetermined conditions based on experience or application scenarios. For example, the predetermined conditions include that the difference between the result of task processing and the annotation value of the sample image is less than a predetermined threshold (or the task evaluation index of the currently processed sample image in the corresponding task processing is higher than the preset index threshold), or the number of times the ISP parameters are iteratively adjusted (or the number of times the task processing is iteratively performed) reaches a predetermined number, or until the result of the task processing is not improved after a continuous number of iterations, etc.

[0036] In the prior art, methods for optimizing ISP are mainly divided into the following ideas and have corresponding problems.

[0037] In the existing technology, approach 1 automatically tunes the ISP based on expert understanding. For example, based on expert understanding, it is concluded that image sharpness and contrast are helpful for computer vision tasks such as registration and pedestrian detection. Therefore, the ISP is used to enhance image features that the experts deem effective. Although this approach does not require manual expert adjustment, it still requires domain knowledge. Furthermore, the expert's conclusions are often only superficially related to the task evaluation, which affects the final results.

[0038] In idea 2 of the prior art, task evaluation indicators are used as optimization targets, but in order to simplify the optimization, the ISP is approximated or partially optimized. For example, in one implementation, the ISP function is abstracted into several CNNs and trained as a whole for downstream tasks; the problem with this idea is that there is no accurate correspondence between its CNN and ISP parameters, and the processing results cannot be applied to the tuning of ISP parameters, which means that existing hardware cannot be utilized. In addition, in another implementation, the independent modules in the ISP are optimized separately. Although this reduces the difficulty of optimization, it also ignores the mutual influence of the modules, resulting in only suboptimal solutions. In addition, in yet another implementation, a proxy neural network is trained to simulate the ISP as a whole. However, since the noise caused by the simulation cannot be ignored, the results obtained based on the proxy neural network are also suboptimal.

[0039] In the existing technology, approach 3 treats the ISP as a black box, uses the task evaluation metric as the optimization target, and directly tunes the parameters using a black box optimizer. The main difficulty lies in the large number of ISP parameters, and the fact that the task evaluation metric only contains the model's final prediction results, with a limited amount of information, making optimization difficult. For example, in one implementation, a CMA-ES optimizer is used to perform black box tuning on the ISP parameters. This allows for explicit tuning of individual parameters while treating the ISP as a whole. The main problem is that the tuning relies solely on the evaluation score of the computer vision task, resulting in low tuning efficiency and poor optimization results.

[0040] However, in the information processing device 100 according to the embodiment of the present disclosure, the parameters of the ISP are adjusted based on the saliency map, that is, the ISP is tuned to optimize the result of the predetermined model performing task processing. Compared with idea 1, the saliency map does not require an expert but can be automatically generated based on the predetermined model and sample images. In addition, the saliency maps of different sample images are different, so it is more flexible and targeted. Compared with idea 2, the ISP is not simplified in the information processing device 100 according to the embodiment of the present disclosure, so it is closer to real-world applications. Compared with idea 3, the saliency map contains decision-making process information in the predetermined model performing task processing (for example, the degree of importance attached to objects at different positions in the sample image when the predetermined model performs task processing). In the process of iteratively adjusting the parameters of the ISP, since the quality of the sample image changes, the saliency map changes more evenly and frequently than the task evaluation index. Therefore, using the saliency map can be more conducive to tuning the ISP parameters, and can achieve better results than directly tuning the ISP based on the task evaluation index, thereby improving the accuracy of task processing.

[0041] As an example, the information processing device 100 can take data in the form of a data stream as input, invoke the ISP simulator for processing, and output the processed sample images in the form of a data stream to a predetermined model. Because the test dataset contains multiple samples, this helps improve the operating efficiency of the information processing device 100 (i.e., increase the speed of ISP automatic tuning) and conserve system resources.

[0042] As an example, the processing unit 102 may be configured to obtain at least one heat map using a machine learning interpretation tool based on an output obtained after inputting a sample image into a predetermined model, and generate a saliency map based on the at least one heat map.

[0043] For example, heat maps may be generated for objects (subjects) of different types and sizes in a sample image, thereby generating multiple heat maps.

[0044] FIG2 is a schematic diagram illustrating an application scenario according to an embodiment of the present disclosure.

[0045] As shown in Figure 2, the original image is captured by an optical sensor and input into the ISP. The ISP processes the original image and outputs a sample image. For example, the functions of the ISP include at least one of demosaicing, white balance, noise reduction, sharpening, tone correction, and bit length compression. The predetermined model processes the sample image to obtain the model output, and at the same time records the intermediate activation value results processed by the predetermined model. The machine learning interpretation tool can, for example, use the above-mentioned model output and intermediate activation value results to generate a heat map, and generate a saliency map based on the heat map. The parameters of the ISP are adjusted based on the saliency map and the annotated area in the sample image so that the difference between the output result of the predetermined model and the annotated value of the sample image meets the predetermined conditions.

[0046] As an example, the machine learning explanation tool may be a deep learning explanation tool. As an example, the machine learning explanation tool includes at least one of Grad-CAM (Gradient-weighted Class Activation Mapping), Grad-CAM++, XGrad-CAM (improved class activation heatmap), Ablation-CAM (ablated class activation heatmap), Score-CAM (score-based class activation heatmap), and guided backpropagation.

[0047] As an example, the size of the saliency map is the same as the size of the sample image, and the pixel value of a pixel in the saliency map reflects the contribution of the pixel to the task processing. For example, when the task processing is a classification task, the pixel value of a pixel in the saliency map reflects the contribution of the pixel to the classification task. For example, when the task processing is an object detection task, the pixel value of a pixel in the saliency map reflects the contribution of the pixel to the object detection task.

[0048] As an example, when the at least one heat map includes only one heat map, pixel values ​​of pixels in the one heat map are normalized to generate a saliency map.

[0049] For example, when there is only one heat map, the following Equation 1 is used to generate a saliency map.

[0050] Assume that there are n pixels in the heat map. In formula 1, x i Represents the pixel value of the i-th pixel in the heat map, Max() means taking the maximum value, x′ i Represents the saliency value of the i-th pixel in the saliency map, scaled to between 0 and 1.

[0051] As an example, in a case where at least one heat map includes multiple heat maps obtained for different objects included in a sample image, the pixel values ​​of pixels in the multiple heat maps are normalized, and the pixel values ​​of pixels at the same position in the multiple normalized heat maps are averaged to generate a saliency map.

[0052] For example, when there are multiple heat maps, the pixel values ​​of the pixels in each heat map are first normalized according to Formula 1, and then the saliency map is generated using the following Formula 2.

[0053] Assuming there are K heat maps, in Equation 2, x′ ki Represents the normalized value of the i-th pixel in the k-th heat map, y i Represents the pixel value of the i-th pixel in the generated saliency map. In the following, for convenience, y i Represents the pixel value of the i-th pixel in the saliency map.

[0054] As an example, the processing unit 102 may be configured to calculate a saliency score corresponding to the sample image based on the saliency map and the annotated region, and the adjustment unit 104 may be configured to perform the above adjustment based on the saliency score.

[0055] The saliency score calculated based on the saliency map and annotated regions can more directly reflect the effectiveness of the predefined model in processing the task on the sample image. Because the saliency score changes directly based on the quality of the sample image, it can more quickly guide ISP tuning and improve the speed of ISP automatic tuning. Furthermore, the saliency score provides information about the sample image's quality to guide ISP tuning even when the task evaluation metric remains unchanged, thereby improving task processing accuracy.

[0056] For example, when the task is a classification task, the labeled region in the sample image is the region used to determine the type of the sample image. For example, the labeled region can be the image region based on which the expert determines the type of the image.

[0057] As an example, when the task processing is an object detection task, the annotated area in the sample image is a bounding box annotated as an object in the sample image.

[0058] As an example, the adjustment unit 104 may be configured to iteratively adjust the parameters of the ISP with the goal of increasing the significance score. During the iterative adjustment of the ISP parameters, the quality of the sample image output by the adjusted ISP continuously changes until the difference between the result of the task processing and the labeled value of the sample image satisfies the predetermined condition.

[0059] As an example, the processing unit 102 may be configured to calculate the sum of the pixel values ​​of the pixels located within the marked area in the saliency map as a first value and calculate the sum of the pixel values ​​of all pixels in the saliency map as a second value, and calculate a ratio between the first value and the second value as a saliency score.

[0060] For example, the significance score s may be calculated using the following Equation 3.

[0061] In formula 3, y i Represents the pixel value of the i-th pixel in the saliency map. If the i-th pixel is in the marked area, then r i =1, otherwise, r i =0.

[0062] As an example, the processing unit 102 may be configured to generate an importance mask reflecting the importance of pixels in the saliency map based on the saliency map, and calculate a saliency score based on the importance mask and the annotated region.

[0063] Hereinafter, pixel values ​​in a saliency map are sometimes referred to as saliency values.

[0064] Compared with the saliency map, the importance mask can eliminate the influence caused by the accumulation of a large number of extremely small saliency values ​​due to noise.

[0065] As an example, the processing unit 102 may be configured to retain pixels in the saliency map whose pixel values ​​are greater than a first predetermined threshold, and set the pixel values ​​of other pixels in the saliency map to 0, thereby generating an importance mask.

[0066] For example, those skilled in the art may pre-set the first predetermined threshold based on experience or application scenarios.

[0067] For example, given a first predetermined threshold, the importance mask only retains pixels in the saliency map whose pixel values ​​are greater than the first predetermined threshold, and masks pixels whose pixel values ​​are less than or equal to the first predetermined threshold (setting their pixel values ​​to 0).

[0068] As an example, in the importance mask, the pixel value of the retained pixel is the pixel value of the pixel at the corresponding position in the saliency map, or a value calculated based on the pixel value of the pixel at the corresponding position in the saliency map.

[0069] For example, in the importance mask, the pixel value of the retained pixel can be the original value of the saliency value, or the saliency value can be subjected to a certain mathematical transformation, such as squaring, taking a square root, or directly setting it to 1.

[0070] As an example, the processing unit 102 can be configured to calculate a first number of pixels in the importance mask located in the annotated area whose pixel values ​​are greater than a predetermined second threshold and calculate a second number of pixels in the importance mask whose pixel values ​​are greater than the predetermined second threshold, and calculate the ratio between the first number and the second number as a significance score.

[0071] For example, those skilled in the art may pre-set the second predetermined threshold based on experience or application scenarios.

[0072] As an example, the annotated region is at least a portion of the region used to annotate multiple objects in the sample image that corresponds to at least a portion of the multiple objects. This allows for calculating saliency scores based on different objects in the sample image, for example, based on objects of a specific category and / or size in the sample image.

[0073] For example, assume that the sample image includes objects of different categories such as vehicles, pedestrians, and riders. For example, when at least a portion of the above-mentioned area corresponds to any object among vehicles, pedestrians, and riders, the saliency score can be calculated based on the above-mentioned any object. For example, when at least a portion of the above-mentioned area corresponds to one object among vehicles, pedestrians, and riders, the saliency score can be calculated based on the one object; when at least a portion of the above-mentioned area corresponds to any two objects among vehicles, pedestrians, and riders, the saliency score can be calculated based on the any two objects; when at least a portion of the above-mentioned area corresponds to vehicles, pedestrians, and riders, the saliency score can be calculated based on the three of vehicles, pedestrians, and riders.

[0074] As an example, the processing unit 102 may be configured to perform the above adjustment based on the saliency score and the evaluation index of the task processing performed by the predetermined model, thereby further improving the accuracy of the task processing.

[0075] For example, for classification tasks, task evaluation metrics may include accuracy or F1 value; for object detection tasks, task evaluation metrics may include mAP. After the predetermined model performs task processing on the sample image, the corresponding evaluation metrics are calculated.

[0076] As an example, the processing unit 102 may be configured to perform the above adjustment based on a sum of a first value obtained by multiplying the significance score by a first predetermined weight and a second value obtained by multiplying the evaluation index by a second predetermined weight.

[0077] As an example, the first predetermined weight is a ratio between the number of pixels in the saliency map that are located in the marked area and the number of all pixels in the saliency map.

[0078] For example, the first predetermined weight w may be expressed using the following formula 4.

[0079] In Equation 4, if the i-th pixel in the saliency map is within the labeled area, then r i =1, otherwise, r i =0.

[0080] For example, those skilled in the art may also pre-set the first predetermined weight w based on experience or application scenarios.

[0081] For example, those skilled in the art may pre-set the second predetermined weight based on experience or application scenarios.

[0082] The processing unit 102 may also perform the aforementioned adjustments based on the saliency score in combination with other metrics besides the evaluation metric. For example, the other metric may be the mean and variance of batch normalization (BN) of a computer vision model, which evaluates the difference in data distribution of sample images before and after optimization.

[0083] For example, the optimizer can perform ISP tuning based on the evaluation scores of the values ​​it generates for the ISP parameters. For example, the optimizer can use the significance score as the evaluation score of the value generated by the current optimizer, or can use the sum of the first value obtained by multiplying the significance score by the first predetermined weight and the second value obtained by multiplying the evaluation index by the second predetermined weight as the evaluation score of the value generated by the current optimizer. For example, in order to perform optimization more stably, it is preferred that a specific number of repeated processes be performed for the current optimizer to obtain multiple sets of values ​​generated by the current optimizer for the ISP parameters and the evaluation scores corresponding to each set of values. For example, the above-mentioned specific number of times can be 16 times. The information processing device 100 can input the above-mentioned multiple sets of values ​​and the corresponding evaluation scores into the optimizer to update the state of the optimizer. The optimizer compares the evaluation scores and updates the internal state so that the newly generated values ​​for the ISP parameters are more likely to be close to the values ​​with high evaluation scores and more likely to be away from the values ​​with low evaluation scores. For example, the optimizer can update the internal state based on the feedback evaluation scores received from the information processing device 100. The optimizer first sorts the evaluation scores from high to low, assigning the values ​​generated by the optimizer for the ISP parameters corresponding to the highest-ranked evaluation scores as positive values, and the values ​​generated by the optimizer for the lowest-ranked evaluation scores as negative values. The optimizer updates its internal state based on the positive and negative values, so that the average of the newly generated values ​​for the ISP parameters is closer to the positive values ​​and more likely to be away from the negative values.

[0084] To achieve better ISP tuning results, the optimizer state is preferably updated repeatedly until the evaluation score does not improve over the previous iteration for a consecutive times, or until a preset number of times, b, is reached. Preferably, a can be 50 times, and b can be 500 times.

[0085] FIG3 is a schematic flowchart illustrating ISP tuning according to an embodiment of the present disclosure.

[0086] In step S31, the original image in the test data set is read. In step S32, the ISP simulator is called to process the original image to obtain a sample image. In step S33, the sample image is processed using a predetermined model. In step S34, the evaluation index obtained after the predetermined model processes the sample image is determined. In step S35, the significance score is calculated. In steps S36 and S37, the evaluation index and the significance score are fed back to the optimizer. In step S38, the state of the optimizer is updated based on the evaluation index and the significance score. In step S39, the parameters of the ISP are updated based on the updated state of the optimizer. Steps S32 to S39 are iteratively performed until the difference between the result of the predetermined model performing task processing on the sample image and the labeled value of the sample image meets the predetermined condition.

[0087] Hereinafter, application examples according to the information processing apparatus 100 are introduced.

[0088] (1) Application Example 1

[0089] In application example 1, the modified Grad-CAM and Yolov3 object detection models are used to calculate the saliency map and saliency score of the sample image, thereby visualizing its judgment process.

[0090] For example, the classic Grad-CAM technique is used to explain image classification models, which are characterized by the final layer outputting a single value representing the category confidence. Object detection models, such as Yolov3, differ from image classification models in that their final layer outputs multiple values ​​representing the confidence of the detection box and multiple values ​​representing the category confidence. Furthermore, they have multiple parallel final layers, each corresponding to the detection of objects of different sizes.

[0091] In Application Example 1, the saliency map corresponding to an object in any final layer of the Yolov3 object detection model is calculated according to the following steps:

[0092] (1) Multiply the detection box confidence and the category confidence by position to obtain multiple final confidences.

[0093] (2) Delete the outputs whose final confidence is less than the threshold, and the threshold is preferably 0.001, 0.1, or 0.5.

[0094] (3) Use non-maximum suppression technology to delete detection frames with the same category and too large overlapping areas, and record the model output corresponding to the remaining detection frames.

[0095] (4) Take the average of the recorded final confidence values ​​of the model output to obtain a single confidence value.

[0096] (5) Based on this single confidence value, the Grad-CAM technique is used to calculate the saliency map.

[0097] Since Yolov3 has multiple parallel final layers, for example, the final saliency map can be obtained by combining multiple saliency maps corresponding to objects of different sizes as follows:

[0098] (1) The saliency value in each saliency map is divided by the maximum value of the corresponding saliency map to obtain the normalized saliency map.

[0099] (2) Multiple normalized saliency maps are averaged pixel by pixel to obtain a final saliency map, which includes the saliency of objects of different sizes.

[0100] Thus, the saliency map can show the image areas based on which Yolov3 performs object detection.

[0101] Based on the final saliency map, the saliency score can be calculated, for example, by the following steps:

[0102] (1) According to the annotation of the object position or important area, the sum of the pixel values ​​of the pixels in the annotated area in the saliency map is calculated and recorded as A.

[0103] (2) The sum of the pixel values ​​of all pixels in the saliency map, denoted as B.

[0104] (3) Significance score = A / B.

[0105] (II) Application Example 2

[0106] In Application Example 2, a significant score is used to improve the effect of ISP automatic tuning. Specifically, in Application Example 2, an existing public dataset is used to reflect the improvement of the effect of ISP automatic tuning. KITTI is a commonly used dataset in the field of autonomous driving, and object recognition can be performed based on the KITTI dataset. In Application Example 2, the KITTI dataset is divided into a training set (about 80%) for training the Yolov3 object detection model; the remaining 20% ​​of the images are used to generate original images before ISP processing through ExpandNet. 256 original images are used to tune the ISP parameters, and the remaining original images are used to test the detection effect of the model on the sample images after ISP processing. In order to eliminate the randomness in the test as much as possible, 10 groups of 128 original images are randomly extracted from 20% of the data for ISP tuning, and the remaining original images are used to test the corresponding tuning results.

[0107] The ISP simulator used in Application Example 2 includes a noise reducer based on bilateral and Gaussian filtering, edge enhancement based on high-pass filtering, and a tone mapper based on the Durand tone mapping algorithm. It can simulate several important functions of the Sony Fuji series ISP. To simulate the discrete nature of parameters in hardware ISPs, the parameters used in the ISP simulator are also discrete.

[0108] In application example 2, the computer vision task is object detection, and the evaluation score is the mAP@0.5 value (hereinafter referred to as mAP). The mAP value is calculated as follows: 1. For a certain category, first set the detection confidence threshold, and the model predictions below the threshold are eliminated; 2. Calculate the intersection area and the union area of ​​the remaining model prediction detection box and the manually labeled detection box respectively. If the intersection area is greater than 0.5 times the union area, it is considered a correct detection, otherwise it is an error; 3. Based on the number of correct and incorrect results in 2, calculate the corresponding precision and recall values; 4. By adjusting the confidence threshold in 1, a curve showing the change in precision with respect to recall value can be obtained; calculate the area under the curve as the AP value of the category; average the AP values ​​of all categories to obtain the mAP value. The calculation method of mAR is similar to that of mAP, but instead of calculating the area under the curve, the average recall value is calculated.

[0109] In Application Example 2, the CMA-ES optimizer is used as the automatic optimizer. Each time, 12 sets of parameters are generated for the ISP. The optimizer's internal state is updated based on the simulated image evaluation scores based on these 12 sets of parameters. While prior art uses only mAP for optimization, this example uses mAP + significance score. mAP and significance score can be weighted differently in different situations.

[0110] Figure 4 shows the prediction results of the predetermined model for a sample image using the existing method and after ISP tuning based on the saliency score. Since saliency scores can be calculated separately for different categories, Figure 4 shows the results of tuning based on the saliency scores calculated for vehicles, pedestrians, and riders, as well as the results of tuning based on the saliency score calculated based on the average of these three categories. It can be seen that for 10 different sets of tuning and test data splits, ISP tuning based on saliency scores calculated for any category outperforms the existing technology. Tuning based on the saliency score for vehicles achieves the best results.

[0111] (III) Application Example 3

[0112] In the early stages of ISP tuning, due to random initial parameters, processed image quality is generally poor. Existing methods alone require a long time to achieve relatively good tuning results. Existing methods use mean average performance (mAP) for tuning, but improving mAP in the initial stages can be difficult, preventing the optimizer from obtaining sufficient information to determine the direction of parameter optimization. However, since the significance score changes directly based on image quality, it still provides a good early signal for tuning. Therefore, embodiments of the present invention can more quickly guide ISP tuning and improve the speed of automatic ISP tuning.

[0113] (IV) Application Example 4

[0114] When the test dataset is small, the evaluation metric changes more sparsely. Therefore, existing methods that directly optimize based on the evaluation metric take a long time to achieve relatively good results. However, because the saliency score changes directly based on image quality, it can provide image quality information to guide ISP tuning even when the evaluation metric remains unchanged. Therefore, it can achieve greater improvements than existing methods. Using the saliency score for ISP tuning can improve the utilization efficiency of the test dataset and reduce the required test dataset size.

[0115] Figure 5 shows a comparison between saliency score-based optimization and existing methods when the test dataset size varies. In Figure 5, for two pairs of mAP plots corresponding to test dataset sizes of 64 and 128, the mAP on the left in each pair corresponds to the existing method, while the mAP on the right corresponds to saliency score-based optimization. As can be seen from Figure 5, when the test dataset size decreases from 128 to 64, the improvement in mAP achieved by saliency score-based optimization compared to existing methods becomes even greater.

[0116] (V) Application Example 5

[0117] In difficult scenarios where the evaluation metric is difficult to improve, existing ISP tuning methods fail to obtain sufficient information about the metric's changes, making it difficult to achieve ideal optimization results. However, since the saliency score changes directly based on sample image quality, it can provide information about sample image quality to guide ISP tuning even when the evaluation metric remains unchanged. Therefore, it can achieve greater improvements compared to existing methods. Figure 6 shows a comparison between saliency score-based tuning and existing methods in difficult scenarios. For simplicity, the horizontal axis is abbreviated as mAP (mean average performance approximation) using saliency score tuning, and the vertical axis is mAP (mean average performance approximation) using existing methods. The dashed line in Figure 6 represents the diagonal line. As shown in Figure 6, when ISP tuning using existing methods and saliency score tuning is compared on 100 images from these difficult scenarios, the mAP of most images is improved by using the saliency score. On average, the mAP of these 100 images is 0.471 using the existing method, while the mAP of saliency score tuning is 0.523, an improvement of over 10%. This shows that using the significant score for ISP tuning can improve the tuning effect in difficult scenarios.

[0118] (Six) Application Example 6

[0119] Different scenarios require different accuracy for different categories, sizes, and objects. Since the saliency map can be calculated based on a bounding box (prediction box), the corresponding saliency score can be calculated for a specific category, size, or object, thereby increasing the flexibility of ISP tuning. In other words, using saliency scores for specific categories can improve the flexibility of ISP tuning and adapt to different scenarios.

[0120] Figures 7A to 7D show schematic diagrams of tuning based on calculating saliency scores for different objects and tuning using existing methods. In Figure 7A, when the existing method is used for tuning, in addition to detecting two vehicles, two areas covered by diagonal lines may be mistakenly detected when detecting vehicles, and therefore, there is erroneous attention on the saliency map. In Figure 7B, when the saliency scores calculated based on vehicles are tuned, two vehicles are correctly detected when detecting vehicles, and the two areas covered by diagonal lines in Figure 7A are not mistakenly detected, thereby reducing the erroneous attention on the saliency map in Figure 7A. In Figure 7C, when the existing method is used for tuning, in addition to detecting one rider, one area surrounded by dotted lines may be mistakenly detected when detecting riders, and therefore, there is erroneous attention on the saliency map. In the tuning performed based on the saliency score calculated for the rider in FIG7D , when detecting the rider, the rider is correctly detected, and the area enclosed by the dotted line in FIG7C is not erroneously detected, thereby reducing the erroneous attention on the saliency map in FIG7C .

[0121] (VII) Application Example 7

[0122] When using deep learning models, it's generally hoped that the model's decision-making process aligns with human experience and can be interpreted based on human experience. However, due to the complexity of deep learning, its judgment basis sometimes differs from that of humans. For example, in object detection, much of the training data comes from urban areas, where vehicles typically appear on roads. Therefore, when the model detects a vehicle, it may rely on the road surface, not the vehicle itself. Therefore, if the image being tuned is also from an urban area and optimization is performed solely based on the model's mAP, the road surface may become more prominent in the ISP output image. In this case, if a vehicle is located on a rural dirt road, the model's ability to detect the car may be affected. In this disclosure, the saliency score is calculated based on manually annotated object regions. For example, when the model focuses on vehicle detection, the saliency score increases when the model focuses on the vehicle region, while it decreases when the model focuses on the road surface. As a result, the ISP output image does not emphasize the road surface, but rather the vehicle itself. Therefore, by tuning the saliency score, the model's results on the ISP output image can be made more consistent with human experience and more interpretable. In other words, using the saliency score for ISP tuning can improve model interpretability.

[0123] Figure 8 is a schematic diagram showing the effect of ISP tuning based on the saliency scores calculated according to different importance masks according to an embodiment of the present disclosure. Wherein, the sample images include objects of different categories such as vehicles, pedestrians, and riders, and the task performed by the predetermined model is processed as an object detection task. The above-mentioned predetermined first thresholds for generating the importance mask are set to 0, 0.4, 0.5, and 0.6 respectively, and the pixels in the saliency map whose pixel values ​​are greater than the first predetermined threshold are set to 1, and the pixel values ​​of other pixels in the saliency map are set to 0, thereby generating masks of different importance. Figure 8 shows the mAP calculated based on the saliency scores of vehicles, pedestrians, and riders, and the ISP tuning based on the calculated saliency scores.

[0124] As shown in FIG8 , compared with the cases where the predetermined first threshold is 0.4, 0.5, and 0.6, when the predetermined first threshold is 0 (ie, equivalent to the case where the significance score is calculated using Formula 3), the mAP value is improved.

[0125] The present disclosure also provides an image processing device including the above-mentioned information processing apparatus. The image processing device can be implemented as a hardware product, such as a camera or a camcorder.

[0126] Corresponding to the above-mentioned information processing device embodiments, the present disclosure also provides embodiments of information processing methods.

[0127] FIG. 9 is a flowchart illustrating an example of a process of an information processing method S900 according to an embodiment of the present disclosure.

[0128] The information processing method S900 according to an embodiment of the present disclosure starts from S902 .

[0129] In S904 , a saliency map of the sample image is generated based on the predetermined model that performs task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs task processing.

[0130] In S906 , parameters of an image signal processor for generating the sample image are adjusted based on the saliency map and the annotated region in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

[0131] The information processing method S900 ends at S908. This method can be executed by the information processing device 100 described above, for example. For details, please refer to the description of the related processing of the information processing device 100, which will not be repeated here.

[0132] The basic principles of the present invention are described above in conjunction with specific embodiments. However, it should be pointed out that those skilled in the art will understand that all or any steps or components of the methods and devices of the present invention can be implemented in any computing device (including a processor, storage medium, etc.) or a network of computing devices in the form of hardware, firmware, software, or a combination thereof. This can be achieved by those skilled in the art using their basic circuit design knowledge or basic programming skills after reading the description of the present invention.

[0133] Furthermore, the present invention also provides a program product storing machine-readable instruction codes. When the instruction codes are read and executed by a machine, the method according to the embodiment of the present invention can be executed.

[0134] Accordingly, the storage medium for carrying the program product storing the machine-readable instruction code is also included in the disclosure of the present invention. The storage medium includes but is not limited to a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

[0135] When the present invention is implemented through software or firmware, the programs constituting the software are installed from a storage medium or a network to a computer with a dedicated hardware structure (such as the general-purpose computer 1000 shown in Figure 10). When various programs are installed on the computer, it can perform various functions, etc.

[0136] In FIG10 , a central processing unit (CPU) 1001 executes various processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 to a random access memory (RAM) 1003. In the RAM 1003, data required when the CPU 1001 executes various processes, etc., is also stored as needed. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output interface 1005 is also connected to the bus 1004.

[0137] The following components are connected to the input / output interface 1005: an input section 1006 (including a keyboard, a mouse, etc.), an output section 1007 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage section 1008 (including a hard disk, etc.), and a communication section 1009 (including a network interface card such as a LAN card, a modem, etc.). The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 may also be connected to the input / output interface 1005 as needed. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed in the drive 1010 as needed, so that a computer program read therefrom is installed in the storage section 1008 as needed.

[0138] In the case of realizing the above-described series of processing by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1011 .

[0139] It should be understood by those skilled in the art that such storage media is not limited to the removable medium 1011 shown in FIG10 , which stores the program therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1011 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memories (CD-ROMs) and digital versatile disks (DVDs)), magneto-optical disks (including minidiscs (MDs) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be the ROM 1002, a hard disk included in the storage section 1008, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0140] It should also be noted that in the apparatus, method, and system of the present invention, each component or step can be decomposed and / or recombined. Such decomposition and / or recombination should be considered equivalent solutions of the present invention. Furthermore, the steps of performing the above series of processes can naturally be performed in chronological order according to the order described, but do not necessarily need to be performed in chronological order. Certain steps can be performed in parallel or independently of each other.

[0141] Finally, it should be noted that the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Furthermore, in the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0142] Although the embodiments of the present invention have been described in detail above with reference to the accompanying drawings, it should be understood that the embodiments described above are merely illustrative of the present invention and are not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments described above without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention is limited solely by the appended claims and their equivalents.

[0143] The present technology can also be implemented as follows.

[0144] Note 1. An information processing device comprising:

[0145] The processing circuit is configured to:

[0146] generating a saliency map of the sample image based on a predetermined model that performs task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs the task processing, and

[0147] The parameters of the image signal processor generating the sample image are adjusted based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

[0148] Supplementary note 2. The information processing device according to Supplementary note 1, wherein the processing circuit is configured to calculate a saliency score corresponding to the sample image based on the saliency map and the annotated area, and perform the adjustment based on the saliency score.

[0149] Note 3. An information processing device according to Note 2, wherein the processing circuit is configured to calculate the sum of pixel values ​​of pixels located in the marked area in the saliency map as a first numerical value and calculate the sum of pixel values ​​of all pixels in the saliency map as a second numerical value, and calculate the ratio between the first numerical value and the second numerical value as the saliency score.

[0150] Note 4. The information processing device according to Note 2, wherein the processing circuit is configured to generate an importance mask reflecting the importance of pixels in the saliency map based on the saliency map, and calculate the saliency score based on the importance mask and the annotated area.

[0151] Supplement 5. The information processing device according to Supplement 4, wherein:

[0152] The processing circuit is configured to retain pixels in the saliency map whose pixel values ​​are greater than a first predetermined threshold and set the pixel values ​​of other pixels in the saliency map to 0, thereby generating the importance mask.

[0153] Supplement 6. The information processing device according to Supplement 5, wherein:

[0154] In the importance mask, the pixel values ​​of the retained pixels are the pixel values ​​of the pixels at the corresponding positions in the saliency map, or are values ​​calculated based on the pixel values ​​of the pixels at the corresponding positions in the saliency map.

[0155] Supplement 7. The information processing device according to Supplement 5 or 6, wherein:

[0156] The processing circuit is configured to calculate a first number of pixels in the importance mask located within the annotated area whose pixel values ​​are greater than a predetermined second threshold, and to calculate a second number of pixels in the importance mask whose pixel values ​​are greater than the predetermined second threshold, and to calculate a ratio between the first number and the second number as the significance score.

[0157] Supplement 8. The information processing device according to any one of Supplements 2 to 7, wherein the processing circuit is configured to perform the adjustment based on the significance score and an evaluation index of the task processing performed by the predetermined model.

[0158] Note 9. The information processing device according to Note 8, wherein the processing circuit is configured to perform the adjustment based on the sum of a first value obtained by multiplying the significance score by a first predetermined weight and a second value obtained by multiplying the evaluation index by a second predetermined weight.

[0159] Supplementary note 10. The information processing device according to Supplementary note 9, wherein the first predetermined weight is a ratio between the number of pixels in the saliency map located within the labeled area and the number of all pixels in the saliency map.

[0160] Note 11. The information processing device according to any one of Notes 2 to 10, wherein the annotation area is at least a portion of an area for annotating multiple objects in the sample image, corresponding to at least a portion of the multiple objects.

[0161] Supplement 12. The information processing device according to any one of Supplements 2 to 11, wherein the processing circuit is configured to:

[0162] Based on the output obtained after inputting the sample image into the predetermined model, using a machine learning interpretation tool to obtain at least one heat map; and

[0163] Based on the at least one heat map, the saliency map is generated.

[0164] Supplement 13. The information processing device according to Supplement 12, wherein:

[0165] The size of the saliency map is the same as the size of the sample image, and

[0166] The pixel value of a pixel in the saliency map reflects the contribution of the pixel to the task processing.

[0167] Supplement 14. The information processing device according to Supplement 12 or 13, wherein

[0168] In a case where the at least one heat map includes only one heat map, normalizing pixel values ​​of pixels in the one heat map to generate the saliency map, or

[0169] In a case where the at least one heat map includes multiple heat maps obtained for different objects included in the sample image, pixel values ​​of pixels in the multiple heat maps are normalized, and pixel values ​​of pixels at the same position in the multiple normalized heat maps are averaged to generate the saliency map.

[0170] Supplement 15. The information processing device according to any one of Supplements 12 to 14, wherein:

[0171] The machine learning interpretation tool includes at least one of Grad-CAM, Grad-CAM++, XGrad-CAM, Ablation-CAM, Score-CAM, and guided backpropagation.

[0172] Supplement 16. The information processing device according to any one of Supplements 2 to 15, wherein the processing circuit is configured to iteratively adjust parameters of the image signal processor with the goal of increasing the significance score.

[0173] Supplement 17. The information processing device according to any one of Supplements 1 to 16, wherein:

[0174] In the case where the task processing is a classification task, the marked area is an area used to determine the type of the sample image.

[0175] Supplement 18. The information processing device according to any one of Supplements 1 to 16, wherein:

[0176] In the case where the task processing is an object detection task, the marked area is a bounding box marked as an object in the sample image.

[0177] Supplement 19. The information processing device according to any one of Supplements 1 to 18, wherein:

[0178] The predetermined model is a computer vision task model.

[0179] Supplement 20. An image processing apparatus comprising the information processing device according to any one of Supplements 1 to 19.

[0180] Note 21. An information processing method comprising:

[0181] generating a saliency map of the sample image based on a predetermined model that performs task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs the task processing, and

[0182] The parameters of the image signal processor generating the sample image are adjusted based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

[0183] Note 22. A computer-readable storage medium having computer-executable instructions stored thereon, which, when executed, executes the information processing method according to Note 21.

Claims

1. An information processing device, comprising: The processing circuit is configured to: generating a saliency map of the sample image based on a predetermined model that performs task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs the task processing, and The parameters of the image signal processor generating the sample image are adjusted based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

2. The information processing device according to claim 1, wherein The processing circuit is configured to calculate a saliency score corresponding to the sample image based on the saliency map and the annotated region, and perform the adjustment based on the saliency score.

3. The information processing device according to claim 2, wherein The processing circuit is configured to calculate a sum of pixel values ​​of pixels located within the annotated area in the saliency map as a first value, calculate a sum of pixel values ​​of all pixels in the saliency map as a second value, and calculate a ratio between the first value and the second value as the saliency score.

4. The information processing device according to claim 2, wherein The processing circuit is configured to generate an importance mask reflecting the importance of pixels in the saliency map based on the saliency map, and calculate the saliency score based on the importance mask and the annotated region.

5. The information processing device according to claim 4, wherein The processing circuit is configured to retain pixels in the saliency map whose pixel values ​​are greater than a first predetermined threshold and set the pixel values ​​of other pixels in the saliency map to 0, thereby generating the importance mask. The information processing device according to claim 5 , wherein: In the importance mask, the pixel values ​​of the retained pixels are the pixel values ​​of the pixels at the corresponding positions in the saliency map, or are values ​​calculated based on the pixel values ​​of the pixels at the corresponding positions in the saliency map.

7. The information processing device according to claim 5 or 6, wherein: The processing circuit is configured to calculate a first number of pixels in the importance mask located within the annotated area whose pixel values ​​are greater than a predetermined second threshold, and to calculate a second number of pixels in the importance mask whose pixel values ​​are greater than the predetermined second threshold, and to calculate a ratio between the first number and the second number as the significance score.

8. The information processing apparatus according to any one of claims 2 to 7, wherein: The processing circuit is configured to perform the adjustment based on the significance score and an evaluation index of the task processing performed by the predetermined model.

9. The information processing device according to claim 8, wherein The processing circuit is configured to perform the adjustment based on a sum of a first value obtained by multiplying the significance score by a first predetermined weight and a second value obtained by multiplying the evaluation index by a second predetermined weight.

10. The information processing device according to claim 9, wherein The first predetermined weight is a ratio between the number of pixels in the saliency map located within the marked area and the number of all pixels in the saliency map.

11. The information processing apparatus according to any one of claims 2 to 10, wherein: The annotation region is at least a portion of a region corresponding to at least a portion of the objects among the regions for annotating the objects in the sample image.

12. The information processing apparatus according to any one of claims 2 to 11, wherein: The processing circuit is configured to: Based on the output obtained after inputting the sample image into the predetermined model, using a machine learning interpretation tool to obtain at least one heat map; and Based on the at least one heat map, the saliency map is generated.

13. The information processing device according to claim 12, wherein: The size of the saliency map is the same as the size of the sample image, and The pixel value of a pixel in the saliency map reflects the contribution of the pixel to the task processing.

14. The information processing device according to claim 12 or 13, wherein: In a case where the at least one heat map includes only one heat map, normalizing pixel values ​​of pixels in the one heat map to generate the saliency map, or In a case where the at least one heat map includes multiple heat maps obtained for different objects included in the sample image, pixel values ​​of pixels in the multiple heat maps are normalized, and pixel values ​​of pixels at the same position in the multiple normalized heat maps are averaged to generate the saliency map.

15. The information processing apparatus according to any one of claims 12 to 14, wherein: The machine learning interpretation tool includes at least one of Grad-CAM, Grad-CAM++, XGrad-CAM, Ablation-CAM, Score-CAM, and guided backpropagation.

16. The information processing apparatus according to any one of claims 2 to 15, wherein: The processing circuitry is configured to iteratively adjust parameters of the image signal processor with a goal of increasing the significance score.

17. The information processing apparatus according to any one of claims 1 to 16, wherein: In the case where the task processing is a classification task, the marked area is an area used to determine the type of the sample image.

18. The information processing apparatus according to any one of claims 1 to 16, wherein: In the case where the task processing is an object detection task, the marked area is a bounding box marked as an object in the sample image.

19. The information processing apparatus according to any one of claims 1 to 18, wherein: The predetermined model is a computer vision task model. 20 . An image processing apparatus comprising the information processing device according to claim 1 .

21. An information processing method, comprising: generating a saliency map of the sample image based on a predetermined model that performs task processing on the sample image, wherein the saliency map reflects the degree of importance attached to objects at different positions in the sample image when the predetermined model performs the task processing, and The parameters of the image signal processor generating the sample image are adjusted based on the saliency map and the annotated area in the sample image, so that the difference between the result of the task processing and the annotated value of the sample image meets a predetermined condition.

22. A computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed, the information processing method according to claim 21 is executed.

Images