Method and apparatus for predicting isp parameters

Abstract

The application provides an ISP parameter prediction method and device, the method comprising: inputting a RAW image to be processed into a feature extractor to obtain a first feature matrix; inputting the first feature matrix into a self-attention layer to obtain an attention map output by the self-attention layer, inputting the attention map into a multi-scale Swin Transformer model to obtain a second feature matrix; inputting the second feature matrix into a first branch and a second branch to obtain an ISP parameter optimal value corresponding to each pixel in the RAW image to be processed and a confidence score; and multiplying the ISP parameter optimal value corresponding to each pixel and the confidence score, and then calculating an average value to obtain an ISP parameter optimal value corresponding to the RAW image to be processed. The application can automatically predict an optimal ISP parameter for any RAW image, and the predicted parameter is suitable for a global image, thereby improving imaging quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an ISP parameter prediction method and apparatus. Background Technology

[0002] Image Signal Processors (ISPs) are widely used in smartphones, displays, surveillance equipment, and autonomous driving systems. An ISP contains algorithm modules with functions such as autofocus, automatic white balance, and automatic tone mapping. By processing a distorted RAW image, it can output a high-quality RGB image. Each algorithm module in an ISP has several configurable parameters to adjust the intensity of the algorithm's effect on the image.

[0003] Currently, the parameters of these algorithm modules rely on manual adjustment by experienced experts, which is inefficient. Furthermore, experts rely on subjective judgment to evaluate image quality during parameter tuning, making the evaluation criteria highly subjective and difficult to maintain consistency.

[0004] AI (Artificial Intelligence) algorithms possess powerful learning capabilities, and AI-based automatic parameter tuning algorithms already exist. One approach is to use deep neural networks to build AI ISP models, replacing traditional ISPs. This eliminates the need to consider ISP parameter configuration and directly learns the mapping from RAW images to high-quality RGB images. However, this method requires a large number of RAW images and high-quality RGB image pairs, resulting in high training set production costs. Furthermore, due to the complexity of distortion factors, improving the ability to handle diverse distortion types requires large-scale network models and sophisticated structural designs, leading to a large number of network parameters and making deployment on mobile devices such as smartphones difficult.

[0005] Another approach is to retain the traditional ISP and use AI algorithms to automatically solve for the configuration parameters of each algorithm module within the ISP. One representative method is to use various search algorithms to search for the image with the highest score in the parameter space. While search-based algorithms can find relatively optimal ISP parameters, they lack the ability to generalize to other images. This means that given a new image, a new search must be performed, and the search space and search time increase exponentially with the number of ISP parameters. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention provides an ISP parameter prediction method and apparatus.

[0007] This invention provides an ISP parameter prediction method, comprising:

[0008] The RAW image to be processed is input into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor.

[0009] The first feature matrix is ​​input into the self-attention layer in the ISP parameter prediction model to obtain the attention map output by the self-attention layer. The attention map is then input into the multi-scale Swin Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swin Transformer model.

[0010] The second feature matrix is ​​input into the first branch and the second branch of the ISP parameter prediction model to obtain the optimal value of the ISP parameter corresponding to each pixel in the RAW image to be processed, output by the first branch, and the confidence score corresponding to each pixel, output by the second branch.

[0011] The optimal ISP parameter value and confidence score corresponding to each pixel are multiplied to obtain the ISP parameter prediction result corresponding to each pixel. The average value of the ISP parameter prediction results corresponding to all pixels is taken as the optimal ISP parameter value corresponding to the RAW image to be processed.

[0012] According to the ISP parameter prediction method provided by the present invention, the feature extractor is a ViT model, and the step of inputting the RAW image to be processed into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor includes:

[0013] The RAW image to be processed is divided into multiple image blocks, and each image block is converted into a vector;

[0014] Input the vectors of all image patches into the ViT model to obtain the first feature matrix of each image patch;

[0015] The first feature matrix is ​​obtained by concatenating the first feature matrices of all image patches on the same channel.

[0016] According to the present invention, an ISP parameter prediction method is provided, wherein inputting the first feature matrix into the self-attention layer of the ISP parameter prediction model to obtain the attention map output by the self-attention layer includes:

[0017] The first feature matrix is ​​resized based on the self-attention layer, and the Q matrix, K matrix, and V matrix of the resized first feature matrix are calculated.

[0018] The K matrix is ​​sizing transformed to determine the product between the Q matrix and the sizing transformed K matrix;

[0019] The product is input into the softmax layer, and the output of the softmax layer is multiplied by the V matrix to obtain the attention map.

[0020] According to the ISP parameter prediction method provided by the present invention, the step of inputting the attention map into the multi-scale Swing Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swing Transformer model includes:

[0021] After the attention map is resized, it is fused with the first feature matrix through a first residual connection;

[0022] The fusion result is sequentially passed through the Swin Transformer module layer and convolutional layer in the multi-scale Swin Transformer model, and then fused with the fusion result through the second residual connection to obtain the second feature matrix.

[0023] According to an ISP parameter prediction method provided by the present invention, before the step of inputting the RAW image to be processed into the feature extractor in the ISP parameter prediction model and obtaining the first feature matrix output by the feature extractor, the method further includes:

[0024] RAW image samples are input into the initial ISP parameter prediction model to obtain the ISP parameter prediction results output by the initial ISP parameter prediction model;

[0025] Determine the first loss between the ISP parameter prediction results corresponding to the RAW image sample and the ISP parameter values ​​annotated by the expert;

[0026] The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing;

[0027] The expert-annotated ISP parameter values ​​and the RAW image samples are input into the ISP to obtain the second image after ISP processing;

[0028] Determine the second loss between the first image and the second image;

[0029] The weights of the initial ISP parameter prediction model are updated based on the first loss and the second loss to obtain the ISP parameter prediction model.

[0030] According to an ISP parameter prediction method provided by the present invention, before the step of inputting the RAW image to be processed into the feature extractor in the ISP parameter prediction model and obtaining the first feature matrix output by the feature extractor, the method further includes:

[0031] The RAW image samples are input into the initial ISP parameter prediction model to obtain the ISP parameter prediction results output by the initial ISP parameter prediction model.

[0032] The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing;

[0033] The first image is input into the target detector to obtain the predicted detection box of the first image;

[0034] Determine a third loss between the predicted detection box of the first image and the manually labeled detection box of the RAW image sample;

[0035] The weights of the initial ISP parameter prediction model are updated based on the third loss to obtain the ISP parameter prediction model.

[0036] The present invention also provides an ISP parameter prediction device, comprising:

[0037] The first extraction module is used to input the RAW image to be processed into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor.

[0038] The second extraction module is used to input the first feature matrix into the self-attention layer in the ISP parameter prediction model to obtain the attention map output by the self-attention layer, and input the attention map into the multi-scale Swin Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swin Transformer model.

[0039] The prediction module is used to input the second feature matrix into the first branch and the second branch of the ISP parameter prediction model to obtain the optimal value of the ISP parameters corresponding to each pixel in the RAW image to be processed output by the first branch, and the confidence score corresponding to each pixel output by the second branch.

[0040] The calculation module is used to multiply the optimal value of the ISP parameters corresponding to each pixel by the confidence score to obtain the ISP parameter prediction result corresponding to each pixel, and take the average value of the ISP parameter prediction results corresponding to all pixels as the optimal value of the ISP parameters corresponding to the RAW image to be processed.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the ISP parameter prediction method as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the ISP parameter prediction method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the ISP parameter prediction method as described above.

[0044] The ISP parameter prediction method and apparatus provided by this invention implicitly learns the image region of interest for ISP parameters through a self-attention layer and a multi-scale SwingTransformer model to guide the prediction of ISP parameters. It predicts a set of optimal ISP parameters and scores for each local image region through two branches, and performs weighted compromise on the prediction results of different regions to obtain a set of ISP parameters applicable to the entire image. The prediction results are more accurate, and a set of optimal ISP parameters can be automatically predicted for any RAW format image, improving imaging quality and efficiency. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the ISP parameter prediction method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the ISP parameter prediction model in the ISP parameter prediction method provided by the present invention;

[0048] Figure 3 This is a schematic diagram of the training process of the ISP parameter prediction model in the ISP parameter prediction method provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the ISP parameter prediction device provided by the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The following is combined with Figure 1 A method for predicting ISP parameters according to the present invention includes:

[0053] Step 101: Input the RAW image to be processed into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor;

[0054] The RAW image to be processed is the raw image encoded data that needs to be processed using ISP.

[0055] The Image Signal Processor (ISP) is a crucial component of camera imaging. Internally, the ISP contains a series of image processing algorithms that improve the input RAW image in terms of noise, texture, color, brightness, and contrast, thereby enhancing image quality. Each algorithm module within the ISP typically has several configurable parameters that control the intensity of the algorithm's effect on the image; these parameters are collectively referred to as ISP parameters.

[0056] This embodiment predicts ISP parameters using an ISP parameter prediction model. For example... Figure 2 As shown, the ISP parameter prediction model includes a feature extractor, a visual attention module, and a local parameter awareness module. The training process of the ISP parameter prediction model is as follows: Figure 3 As shown.

[0057] First, a feature extractor is used to extract the first feature matrix of the RAW image. This embodiment does not limit the type of feature extractor.

[0058] Step 102: Input the first feature matrix into the self-attention layer in the ISP parameter prediction model to obtain the attention map output by the self-attention layer, and input the attention map into the multi-scale Swin Transformer (SST) model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swin Transformer model.

[0059] The visual attention module includes a self-attention layer and a SST (Self-Attention Layer). The first feature matrix is ​​input into the visual attention module to obtain the second feature matrix.

[0060] Each algorithm module in the ISP processes the image from different aspects, such as noise, texture, color, brightness, and contrast. The parameters of each module have different sensitivities to different regions of the image; for example, the denoising module is more sensitive to areas with severe noise, while the edge enhancement module is more sensitive to areas with rich texture. Therefore, this embodiment implicitly learns the image regions of interest to the ISP parameters through a visual attention module to guide the prediction of the ISP parameters.

[0061] The self-attention layer in the visual attention module is used to calculate the attention map on the first feature matrix, and the SST in the visual attention module is used to extract features from different levels of the attention map to obtain the second feature matrix.

[0062] Step 103: Input the second feature matrix into the first branch and the second branch of the ISP parameter prediction model to obtain the optimal value of the ISP parameter corresponding to each pixel in the RAW image to be processed output by the first branch, and the confidence score corresponding to each pixel output by the second branch.

[0063] Considering the varying degrees and types of distortion in different regions of an image, it is difficult to directly predict a set of optimal ISP parameters based on global image information. Therefore, a local parameter awareness module predicts a set of optimal ISP parameters for each local image region and estimates a score for the prediction result for each region. A weighted average is then applied to the prediction results for different regions to obtain a set of ISP parameters applicable to the entire image.

[0064] The local parameter awareness module comprises two branches, which are used to predict the optimal ISP parameter value and confidence score for each pixel in the RAW image to be processed. Each branch can be implemented using several fully connected layers or convolutional layers to obtain the local ISP parameter optimal result image. Confidence score plot Where N is the number of ISP parameters.

[0065] Step 104: Multiply the optimal value of the ISP parameters corresponding to each pixel by the confidence score to obtain the ISP parameter prediction result corresponding to each pixel, and take the average value of the ISP parameter prediction results corresponding to all pixels as the optimal value of the ISP parameters corresponding to the RAW image to be processed.

[0066] Multiply S and Y element-wise and average them to output the prediction results of N ISP parameters. The prediction results of N ISP parameters and the RAW image to be processed can be input into the ISP to obtain a high-quality RGB image, thereby improving the quality and efficiency of imaging.

[0067] This embodiment implicitly learns the image regions of interest for ISP parameters through a self-attention layer and a multi-scale Swing Transformer model to guide the prediction of ISP parameters. It predicts a set of optimal ISP parameters and scores for each local image region through two branches, and performs weighted trade-offs on the prediction results of different regions to obtain a set of ISP parameters applicable to the entire image. The prediction results are more accurate, and a set of optimal ISP parameters can be automatically predicted for any RAW format image, improving imaging quality and efficiency.

[0068] Based on the above embodiments, the feature extractor in this embodiment is a ViT (VisionTransformer, a Transformer for visual tasks) model. The step of inputting the RAW image to be processed into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor includes:

[0069] The RAW image to be processed is divided into multiple image blocks, and each image block is converted into a vector;

[0070] Input the vectors of all image patches into the ViT model to obtain the first feature matrix of each image patch;

[0071] The first feature matrix is ​​obtained by concatenating the first feature matrices of all image patches on the same channel.

[0072] Given that CNNs (Convolutional Neural Networks) lack a global understanding of the image itself and cannot model long-distance dependencies between features, they cannot fully utilize contextual information. Furthermore, the weights of CNNs are fixed and cannot dynamically adapt to changes in the input image.

[0073] Therefore, this embodiment uses ViT as a feature extractor, which can better capture the global contextual information of the input image and extract more expressive features compared to CNN.

[0074] like Figure 2 As shown, the RAW image to be processed Where W and H represent the width and height of the RAW image to be processed, respectively.

[0075] The RAW image to be processed is divided into smaller image blocks. w and h represent the width and height of each image patch, respectively. The row data in each image patch can be concatenated to obtain a one-dimensional vector.

[0076] The one-dimensional vectors of all image patches are input into ViT, and features from several layers of the vectors are extracted. The extracted features are then concatenated along the channel dimension to obtain the basic features, i.e., the first feature matrix. C and U represent the number of channels and spatial dimension of the feature, respectively (U >> C). The spatial dimension U is the number of rows multiplied by the number of columns in the first feature matrix, and the number of channels is the number of elements in the first feature matrix.

[0077] Based on the above embodiments, the step of inputting the first feature matrix into the self-attention layer of the ISP parameter prediction model to obtain the attention map output by the self-attention layer in this embodiment includes:

[0078] The first feature matrix is ​​resized based on the self-attention layer, and the Q matrix, K matrix, and V matrix of the resized first feature matrix are calculated.

[0079] The K matrix is ​​sizing transformed to determine the product between the Q matrix and the sizing transformed K matrix;

[0080] The product is input into the softmax layer, and the output of the softmax layer is multiplied by the V matrix to obtain the attention map.

[0081] like Figure 2 As shown, when calculating the attention map for the first feature matrix, the first feature matrix is ​​first resized, such as by reshaping, to obtain a matrix of size C×U. The Q (Query) matrix, K (Key) matrix, and V (Value) matrix of this matrix are then calculated. This process can be expressed as: Q = W Q X, K = W K X, V = W V X. The dimensions of all three matrices are... Among them W (·) This can be achieved using convolutional layers or fully connected layers.

[0082] Then, the K matrix is ​​sized to obtain... The calculation of the attention map in traditional ViT is improved to the following formula, thus obtaining the attention map.

[0083]

[0084] Here, α is a learnable parameter used to adjust the magnitude of the feature. Unlike traditional ViT, Multiplying Q element-wise along the channel dimension yields a size of Compared to traditional ViT, which performs element-wise multiplication in the spatial dimension to obtain the size, This method can greatly reduce computational complexity while effectively encoding global contextual features of the image.

[0085] Based on the above embodiments, the step of inputting the attention map into the multi-scale Swing Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swing Transformer model includes:

[0086] After the attention map is resized, it is fused with the first feature matrix through a first residual connection;

[0087] The fusion result is sequentially passed through the Swing Transformer model layer and convolutional layer in the multi-scale Swing Transformer model, and then fused with the fusion result through the second residual connection to obtain the second feature matrix.

[0088] like Figure 2 As shown, the attention map A after size transformation is fused with the first feature matrix X through a residual connection. This process can be represented as:

[0089]

[0090] W can be implemented using convolutional layers or fully connected layers. Then, the features... The input is into the SST model. This model includes an STL (Swin Transformer Layer) and a convolutional layer to extract features from different levels of the image, which are then connected again via a residual connection. The process involves fusion. This process can be represented as:

[0091]

[0092] Among them, H conv and H STL These represent convolution and STL processing, respectively. β is a scaling factor used for stable training, typically ranging from 0.1 to 0.2.

[0093] Based on the above embodiments, when dealing with downstream tasks related to human visual perception, this embodiment further includes the following step before obtaining the first feature matrix output by the feature extractor in the feature extractor of the ISP parameter prediction model by inputting the RAW image to be processed into the feature extractor:

[0094] RAW image samples are input into the initial ISP parameter prediction model to obtain the ISP parameter prediction results output by the initial ISP parameter prediction model;

[0095] Determine the first loss between the ISP parameter prediction results corresponding to the RAW image sample and the ISP parameter values ​​annotated by the expert;

[0096] The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing;

[0097] The expert-annotated ISP parameter values ​​and the RAW image samples are input into the ISP to obtain the second image after ISP processing;

[0098] Determine the second loss between the first image and the second image;

[0099] The weights of the initial ISP parameter prediction model are updated based on the first loss and the second loss to obtain the ISP parameter prediction model.

[0100] In downstream tasks targeting human visual perception, given an ISP (Image Signal Processor) and N parameters requiring parameter tuning, and a training dataset D where each training sample includes a RAW image and corresponding expert-annotated ISP parameter values, the training process of the ISP parameter prediction model is as follows:

[0101] A RAW image sample is randomly selected from the training dataset D. Input the data into the initial ISP parameter prediction model and obtain the prediction results of N ISP parameters output by the initial ISP parameter prediction model. Using expert-annotated ISP parameters As labels, the following loss function is calculated for the predicted ISP parameters:

[0102] L p =||GZ||2

[0103] Where ||·||2 represents the L2 norm.

[0104] The predicted ISP parameters Z and the RAW image sample I are input into a given ISP to obtain the ISP-processed image I. z The expert-annotated ISP parameters G and the RAW image sample I are input into a given ISP to obtain the ISP-processed image I. G The following loss function is calculated for the imaging results:

[0105] L im =||I Z -I G ||2.

[0106] The overall loss function is:

[0107] L total =L p +λL im ;

[0108] Here, λ is used to balance the weights of the two loss functions. The gradient descent algorithm can be used to update the weights of the initial ISP parameter prediction model to obtain the ISP parameter prediction model.

[0109] After training, any given RAW image in RAW format is input into the trained ISP parameter prediction model. The ISP parameter prediction model will output N predicted ISP parameters. The predicted ISP parameters and the RAW image are then input into a given ISP to obtain the imaging result.

[0110] In downstream tasks oriented towards human perception, experimental results show that the ISP parameters predicted in this embodiment are very close to the expert-tuned results, with an error of only 6%. The quality of the tuned images can reach the level of expert tuning. The PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) on the test set are 42.448 and 0.985, respectively.

[0111] Based on the above embodiments, when targeting downstream tasks of machine vision analysis, taking object detection as an example, this embodiment further includes the following step before obtaining the first feature matrix output by the feature extractor in the feature extractor of the ISP parameter prediction model by inputting the RAW image to be processed into it:

[0112] The RAW image samples are input into the initial ISP parameter prediction model to obtain the ISP parameter prediction results output by the initial ISP parameter prediction model.

[0113] The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing;

[0114] The first image is input into the target detector to obtain the predicted detection box of the first image;

[0115] Determine a third loss between the predicted detection box of the first image and the manually labeled detection box of the RAW image sample;

[0116] The weights of the initial ISP parameter prediction model are updated based on the third loss to obtain the ISP parameter prediction model.

[0117] When working on downstream object detection tasks, given an ISP (Intense Predictive Array) and N parameters requiring tuning, and a training dataset D, each training sample consists of a RAW image sample and labeled object detection boxes, given a trained object detector H. The training process of the ISP parameter prediction model is as follows:

[0118] A RAW image sample is randomly selected from the training dataset D. Input the data into the initial ISP parameter prediction model and obtain the prediction results of N ISP parameters output by the initial ISP parameter prediction model.

[0119] The predicted ISP parameters Z and the RAW image sample I are input into a given ISP to obtain the ISP-processed image I. z The imaging results I z The input is fed into a given object detector H to obtain the predicted detection result [x,y,w,h,s]. Here, x, y, w, and h represent the x-coordinate and y-coordinate of the top-left vertex of the predicted detection box, as well as the width and length of the detection box, respectively, and s represents the predicted class probability. The positional loss and classification loss are calculated for both the labeled object detection boxes to obtain the object detection loss function L. obj The gradient descent algorithm can be used to update the weights of the initial ISP parameter prediction model to obtain the ISP parameter prediction model.

[0120] After training, any given RAW image in RAW format is input into the trained ISP parameter prediction model. The ISP parameter prediction model will output N predicted ISP parameters. The predicted ISP parameters and the RAW image are then input into a given ISP to obtain the imaging result.

[0121] In various machine vision analysis tasks, by cascading the corresponding machine vision algorithm model after the ISP parameter prediction model, the ISP parameter prediction model can be directly optimized for the target of the downstream task. In the downstream task of target detection, the results in Table 1 show that all three metrics in this embodiment achieve the highest scores, indicating that the imaging results obtained in this embodiment can achieve higher accuracy in target detection tasks.

[0122] Table 1 Comparison of the accuracy of imaging results of various methods in target detection tasks.

[0123] Parameter tuning method <![CDATA[mAP 0.5 ]]> <![CDATA[mAP 0.75 ]]> <![CDATA[mAP 0.5:0.95 ]]> Camera default parameters 0.34 0.22 0.21 Expert parameter adjustment 0.56 0.40 0.37 This invention 0.61 0.44 0.41

[0124] In this embodiment, during the training of the ISP parameter prediction model, the predicted ISP parameters are input into the ISP for imaging, and the imaging results are then input into the machine vision algorithm model to obtain the predicted values. The loss function is calculated with the labeled values, and the loss function is backpropagated to train the ISP parameter prediction model. This allows the ISP parameter prediction model to be directly optimized for the target of downstream tasks.

[0125] The ISP parameter prediction device provided by the present invention is described below. The ISP parameter prediction device described below and the ISP parameter prediction method described above can be referred to in correspondence.

[0126] like Figure 4 As shown, the device includes a first extraction module 401, a second extraction module 402, a prediction module 403, and a calculation module 404, wherein:

[0127] The first extraction module 401 is used to input the RAW image to be processed into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor.

[0128] The second extraction module 402 is used to input the first feature matrix into the self-attention layer in the ISP parameter prediction model to obtain the attention map output by the self-attention layer, and input the attention map into the multi-scale Swing Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swing Transformer model.

[0129] The prediction module 403 is used to input the second feature matrix into the first branch and the second branch of the ISP parameter prediction model to obtain the optimal value of the ISP parameter corresponding to each pixel in the RAW image to be processed output by the first branch, and the confidence score corresponding to each pixel output by the second branch.

[0130] The calculation module 404 is used to multiply the optimal value of the ISP parameter corresponding to each pixel and the confidence score to obtain the ISP parameter prediction result corresponding to each pixel, and take the average value of the ISP parameter prediction results corresponding to all pixels as the optimal value of the ISP parameter corresponding to the RAW image to be processed.

[0131] This embodiment implicitly learns the image regions of interest for ISP parameters through a self-attention layer and a multi-scale Swing Transformer model to guide the prediction of ISP parameters. It predicts a set of optimal ISP parameters and scores for each local image region through two branches, and performs weighted trade-offs on the prediction results of different regions to obtain a set of ISP parameters applicable to the entire image. The prediction results are more accurate, and a set of optimal ISP parameters can be automatically predicted for any RAW format image, improving imaging quality and efficiency.

[0132] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an ISP parameter prediction method. This method includes: implicitly learning the image region of interest for the ISP parameters through a self-attention layer and a multi-scale Swin Transformer model to guide the prediction of the ISP parameters; predicting a set of optimal ISP parameters and scores for each local image region through two branches; and weighting the prediction results for different regions to obtain a set of ISP parameters applicable to the entire image. This results in more accurate predictions and can automatically predict a set of optimal ISP parameters for any RAW format image, improving imaging quality and efficiency.

[0133] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the ISP parameter prediction method provided by the above methods. The method includes: implicitly learning the image region of interest for the ISP parameters through a self-attention layer and a multi-scale Swing Transformer model to guide the prediction of ISP parameters; predicting a set of optimal ISP parameters and scores for each local image region through two branches; and weighting the prediction results of different regions to obtain a set of ISP parameters applicable to the entire image. The prediction results are more accurate, and a set of optimal ISP parameters can be automatically predicted for any RAW format image, improving imaging quality and efficiency.

[0135] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the ISP parameter prediction method provided by the above methods. The method includes: implicitly learning the image region of interest for the ISP parameters through a self-attention layer and a multi-scale Swing Transformer model to guide the prediction of ISP parameters; predicting a set of optimal ISP parameters and scores for each local image region through two branches; and weighting the prediction results for different regions to obtain a set of ISP parameters applicable to the entire image. The prediction results are more accurate, and a set of optimal ISP parameters can be automatically predicted for any RAW format image, improving imaging quality and efficiency.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting ISP parameters, characterized in that, include: RAW image samples are input into the initial ISP parameter prediction model to obtain the ISP parameter prediction results output by the initial ISP parameter prediction model; Determine the first loss between the ISP parameter prediction results corresponding to the RAW image sample and the ISP parameter values ​​annotated by the expert; The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing; The expert-annotated ISP parameter values ​​and the RAW image samples are input into the ISP to obtain the second image after ISP processing; Determine the second loss between the first image and the second image; The weights of the initial ISP parameter prediction model are updated based on the first loss and the second loss to obtain the ISP parameter prediction model. The RAW image to be processed is input into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor. The first feature matrix is ​​input into the self-attention layer in the ISP parameter prediction model to obtain the attention map output by the self-attention layer. The attention map is then input into the multi-scale Swin Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swin Transformer model. The second feature matrix is ​​input into the first branch and the second branch of the ISP parameter prediction model to obtain the optimal value of the ISP parameter corresponding to each pixel in the RAW image to be processed, output by the first branch, and the confidence score corresponding to each pixel, output by the second branch. The optimal ISP parameter value and confidence score corresponding to each pixel are multiplied to obtain the ISP parameter prediction result corresponding to each pixel. The average value of the ISP parameter prediction results corresponding to all pixels is taken as the optimal ISP parameter value corresponding to the RAW image to be processed.

2. The ISP parameter prediction method according to claim 1, characterized in that, The feature extractor is a ViT model. The process of inputting the RAW image to be processed into the feature extractor of the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor includes: The RAW image to be processed is divided into multiple image blocks, and each image block is converted into a vector; Input the vectors of all image patches into the ViT model to obtain the first feature matrix of each image patch; The first feature matrix is ​​obtained by concatenating the first feature matrices of all image patches on the same channel.

3. The ISP parameter prediction method according to claim 1, characterized in that, The step of inputting the first feature matrix into the self-attention layer of the ISP parameter prediction model to obtain the attention map output by the self-attention layer includes: The first feature matrix is ​​resized based on the self-attention layer, and the Q matrix, K matrix, and V matrix of the resized first feature matrix are calculated. The K matrix is ​​sizing transformed to determine the product between the Q matrix and the sizing transformed K matrix; The product is input into the softmax layer, and the output of the softmax layer is multiplied by the V matrix to obtain the attention map.

4. The ISP parameter prediction method according to claim 1, characterized in that, The step of inputting the attention map into the multi-scale Swin Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swin Transformer model includes: After the attention map is resized, it is fused with the first feature matrix through a first residual connection; The fusion result is sequentially passed through the Swin Transformer module layer and convolutional layer in the multi-scale Swin Transformer model, and then fused with the fusion result through the second residual connection to obtain the second feature matrix.

5. The ISP parameter prediction method according to any one of claims 1-4, characterized in that, Before the step of obtaining the first feature matrix output by the feature extractor in the feature extractor of the ISP parameter prediction model, the method further includes: The RAW image samples are input into the initial ISP parameter prediction model to obtain the ISP parameter prediction results output by the initial ISP parameter prediction model. The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing; The first image is input into the target detector to obtain the predicted detection box of the first image; Determine a third loss between the predicted detection box of the first image and the manually labeled detection box of the RAW image sample; The weights of the initial ISP parameter prediction model are updated based on the third loss to obtain the ISP parameter prediction model.

6. An ISP parameter prediction device, characterized in that, include: The first extraction module is used to input RAW image samples into the initial ISP parameter prediction model to obtain the ISP parameter prediction result output by the initial ISP parameter prediction model. Determine the first loss between the ISP parameter prediction results corresponding to the RAW image sample and the ISP parameter values ​​annotated by the expert; The ISP parameter prediction results corresponding to the RAW image samples and the RAW image samples are input into the ISP to obtain the first image after ISP processing; The expert-annotated ISP parameter values ​​and the RAW image samples are input into the ISP to obtain the second image after ISP processing; a second loss between the first image and the second image is determined; the weights of the initial ISP parameter prediction model are updated according to the first loss and the second loss to obtain the ISP parameter prediction model; the RAW image to be processed is input into the feature extractor in the ISP parameter prediction model to obtain the first feature matrix output by the feature extractor. The second extraction module is used to input the first feature matrix into the self-attention layer in the ISP parameter prediction model to obtain the attention map output by the self-attention layer, and input the attention map into the multi-scale Swin Transformer model in the ISP parameter prediction model to obtain the second feature matrix output by the multi-scale Swin Transformer model. The prediction module is used to input the second feature matrix into the first branch and the second branch of the ISP parameter prediction model to obtain the optimal value of the ISP parameters corresponding to each pixel in the RAW image to be processed output by the first branch, and the confidence score corresponding to each pixel output by the second branch. The calculation module is used to multiply the optimal value of the ISP parameters corresponding to each pixel by the confidence score to obtain the ISP parameter prediction result corresponding to each pixel, and take the average value of the ISP parameter prediction results corresponding to all pixels as the optimal value of the ISP parameters corresponding to the RAW image to be processed.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the ISP parameter prediction method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the ISP parameter prediction method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the ISP parameter prediction method as described in any one of claims 1 to 5.

Images