ISP debugging method and device, image processing system, terminal and storage medium

Abstract

The application discloses an ISP debugging method, an ISP debugging device, an image processing system, a terminal and a computer readable storage medium. The ISP debugging method comprises the following steps: inputting a to-be-processed original image into a parameter learning prediction network; determining an image processing prediction parameter corresponding to an image processor according to the to-be-processed original image by using the parameter learning prediction network; and outputting the image processing prediction parameter to the image processor, so that the image processor performs image processing on the to-be-processed original image according to the image processing prediction parameter to obtain a target output image. In the application, the image processing prediction parameter corresponding to the image processor is determined according to the to-be-processed original image by using the parameter learning prediction network, the huge computing performance of the parameter learning prediction network can be fully utilized, and the optimal image processing prediction parameter of the to-be-processed original image can be found without too much human participation in application.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an ISP debugging method, an ISP debugging device, an image processing system, a terminal, and a computer-readable storage medium. Background Art

[0002] Image signal processing (ISP) performs a series of algorithmic processing on the raw image output by the image sensor to ultimately produce the target image. A complete ISP process typically includes multiple algorithm modules. By adjusting the image processing parameters within these algorithm modules, image quality (for example, enhancing image quality, brightness, and color) can be improved to meet the ISP requirements for various scenarios. However, current ISP debugging solutions primarily rely on manual tuning of the image processing parameters within the algorithm modules, which is not only inefficient but also difficult to guarantee the optimal final parameters. Summary of the Invention

[0003] Embodiments of the present application provide an ISP debugging method, an ISP debugging device, an image processing system, a terminal, and a computer-readable storage medium.

[0004] The ISP debugging method according to the embodiment of the present application includes:

[0005] Input the original image to be processed into the parameter learning prediction network;

[0006] Determining image processing prediction parameters corresponding to an image processor according to the original image to be processed using the parameter learning prediction network;

[0007] The image processing prediction parameters are output to the image processor, so that the image processor performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0008] The ISP debugging device according to the embodiment of the present application includes:

[0009] An input module, used to input the original image to be processed into the parameter learning prediction network;

[0010] a determination module, configured to determine image processing prediction parameters corresponding to an image processor according to the original image to be processed using the parameter learning prediction network;

[0011] The output module is used to output the image processing prediction parameters to the image processor, so that the image processor performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0012] The image processing system according to the embodiment of the present application includes:

[0013] An image sensor, used for collecting raw images to be processed;

[0014] ISP debugging device, used for:

[0015] Inputting the original image to be processed into a parameter learning prediction network;

[0016] Determining image processing prediction parameters corresponding to an image processor according to the original image to be processed using the parameter learning prediction network;

[0017] outputting the image processing prediction parameters to the image processor;

[0018] An image processor is used to perform image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0019] The terminal according to the embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, the ISP debugging method according to the embodiment of the present application is implemented.

[0020] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the ISP debugging method of the embodiment of the present application is implemented.

[0021] In the ISP debugging method, ISP debugging device, image processing system, terminal and computer-readable storage medium of the embodiments of the present application, a parameter learning prediction network determines the image processing prediction parameters corresponding to the image processor based on the original image to be processed, which can fully utilize the huge computing performance of the parameter learning prediction network and does not require excessive human intervention during application to find the optimal image processing prediction parameters for the original image to be processed.

[0022] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0024] Figure 1 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0025] Figure 2 Schematic diagram of a module of an ISP debugging device according to certain embodiments of the present application;

[0026] Figure 3 is a schematic structural diagram of an image processing system in certain embodiments of the present application;

[0027] Figure 4 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0028] Figure 5 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0029] Figure 6 is a schematic diagram of the structure of a parameter learning prediction network in certain embodiments of the present application;

[0030] Figure 7 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0031] Figure 8 Schematic diagram of a module of an ISP debugging device according to certain embodiments of the present application;

[0032] Figure 9 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0033] Figure 10 Schematic diagram of the training process of the parameter learning prediction network in certain embodiments of the present application;

[0034] Figure 11 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0035] Figure 12 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0036] Figure 13 is a schematic diagram of the structure of an artificial intelligence image processing network in certain embodiments of the present application;

[0037] Figure 14 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0038] Figure 15 Schematic diagram of the ISP debugging method according to certain embodiments of the present invention;

[0039] Figure 16 Schematic diagram of a module of an ISP debugging device according to certain embodiments of the present application;

[0040] Figure 17 is a schematic structural diagram of a terminal in some embodiments of the present application;

[0041] Figure 18This is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0042] The following further describes the embodiments of the present application in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions. Furthermore, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are intended only to explain the embodiments of the present application and are not to be construed as limiting the present application.

[0043] In related technologies, Image Signal Processing (ISP) performs a series of algorithmic processing on the raw image output by the image sensor to ultimately produce the target image. A complete ISP process typically includes multiple algorithm modules. By debugging the image processing parameters within these algorithm modules, image quality can be improved (e.g., enhancing image quality, brightness, and color), meeting the ISP requirements for various scenarios. However, current ISP debugging solutions primarily rely on manual debugging of the image processing parameters within the algorithm modules, which is not only inefficient but also difficult to guarantee optimal final parameter results.

[0044] See also Figure 1 and Figure 3 The present application embodiment provides an ISP debugging method. The ISP debugging method includes:

[0045] 01: Input the original image to be processed into the parameter learning prediction network 101;

[0046] 02: Using the parameter learning prediction network 101 to determine the image processing prediction parameters corresponding to the image processor 300 according to the original image to be processed;

[0047] 03: Output the image processing prediction parameters to the image processor 300, so that the image processor 300 performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0048] See also Figure 2 and Figure 3The present embodiment also provides an ISP debugging device 100. The ISP debugging device 100 includes an input module 10, a determination module 20, and an output module 30. The ISP debugging method of the present embodiment can be implemented by the ISP debugging device 100 of the present embodiment. Specifically, the input module 10 can be used to implement the method in 01, the determination module 20 can be used to implement the method in 02, and the output module 30 can be used to implement the method in 03. In other words, the input module 10 can be used to input the original image to be processed into the parameter learning and prediction network 101. The determination module 20 can be used to use the parameter learning and prediction network 101 to determine the image processing prediction parameters corresponding to the image processor 300 based on the original image to be processed. The output module 30 can be used to output the image processing prediction parameters to the image processor 300, so that the image processor 300 performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0049] In the ISP debugging method and ISP debugging device 100 of the embodiment of the present application, the parameter learning prediction network 101 determines the image processing prediction parameters corresponding to the image processor 300 based on the original image to be processed. The huge computing performance of the parameter learning prediction network 101 can be fully utilized, and the optimal image processing prediction parameters for the original image to be processed can be found without excessive human participation during application.

[0050] Specifically, see Figure 3 , the original image to be processed input to the parameter learning prediction network 101 can be the original image to be processed captured by the image sensor 200, that is, the RAW image. In addition to the parameter learning prediction network 101, the original image to be processed is also input to the image processor 300 for image processing. In the embodiment of the present application, the parameter learning prediction network (Parameters Learning Network) 101 is a convolutional neural network (CNN). The original image to be processed is input to the parameter learning prediction network 101. The parameter learning prediction network 101 can output image processing prediction parameters corresponding to the image processor 300 based on the original image to be processed, so that the image processor 300 performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0051] Understandably, when debugging an ISP, on the one hand, real-world scenarios are often complex and varied, requiring the ISP to flexibly handle any scenario. On the other hand, debuggers can adjust the algorithm parameters to achieve results that are tailored to the team's preferences, which represents a customized parameter requirement. To meet these debugging requirements, each algorithm module needs to have sufficient debugging parameters available. By setting different parameters, the algorithm's performance can be adjusted. Manually debugging the parameters in algorithm modules is not only extremely labor-intensive and inefficient, but also often requires only subjective evaluation, making it difficult to comprehensively cover all scenarios. Furthermore, subjective evaluation standards vary among different individuals, making it difficult to guarantee the optimal final debugging parameters.

[0052] The ISP debugging process of the embodiment of the present application can make full use of the huge computing performance of the parameter learning prediction network 101. On the one hand, since it does not require too much human participation during application, there is no problem of low efficiency of manual parameter debugging; on the other hand, through the parameter learning prediction network 101, the optimal image processing prediction parameters for the original image to be processed can be found among a large number of parameters, and the appropriate image processing prediction parameters can be adaptively debugged for each scene, thereby obtaining the optimal image processing effect.

[0053] See also Figure 3 and Figure 4 In some embodiments, the image processor 300 includes multiple image processing modules 310. The image processing prediction parameters include multiple image processing prediction sub-parameters. The parameter learning prediction network 101 is used to determine the image processing prediction parameters (i.e., O2) corresponding to the image processor 300 based on the original image to be processed, including:

[0054] 021: Determine, using the parameter learning prediction network 101, a plurality of image processing prediction sub-parameters corresponding to the plurality of image processing modules 310 according to the original image to be processed;

[0055] Outputting the image processing prediction parameters to the image processor 300 so that the image processor 300 performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image (i.e., 03), including:

[0056] 031: Outputting the plurality of image processing prediction sub-parameters to the image processor 300, so that the plurality of image processing modules 310 respectively perform image processing on the original image to be processed according to the corresponding plurality of image processing prediction sub-parameters to obtain a target output image.

[0057] See also Figure 2 and Figure 3In some embodiments, the image processor 300 includes multiple image processing modules 310. The image processing prediction parameters include multiple image processing prediction sub-parameters. The determination module 20 can be used to implement the method in 021, and the output module 30 can be used to implement the method in 031. That is, the determination module 20 can be used to use the parameter learning prediction network 101 to determine multiple image processing prediction sub-parameters corresponding to the multiple image processing modules 310 according to the original image to be processed. The output module 30 can be used to output the multiple image processing prediction sub-parameters to the image processor 300, so that the multiple image processing modules 310 perform image processing on the original image to be processed according to the corresponding multiple image processing prediction sub-parameters to obtain the target output image.

[0058] Specifically, see Figure 3The image processor 300 includes multiple image processing modules 310. The multiple image processing modules 310 can sequentially perform image processing on the original image to be processed. The multiple image processing modules 310 may include, for example, a Bayer processing module, a demosaicing module, an RGB processing module, a color space transformation (CST) module, a YUV processing module, etc. The image processing prediction parameters Algo parameters can be a parameter set consisting of multiple image processing prediction sub-parameters. The parameter learning prediction network 101 can determine the image processing prediction sub-parameters corresponding to each image processing module 310 based on the original image to be processed. For example, the parameter learning prediction network 101 can determine, based on the original image to be processed, the image processing prediction sub-parameter Algo parameter1 corresponding to the Bayer processing module, the image processing prediction sub-parameter Algo parameter2 corresponding to the demosaicing module, the image processing prediction sub-parameter Algo parameter3 corresponding to the RGB processing module, the image processing prediction sub-parameter Algo parameter4 corresponding to the color space conversion module, and the image processing prediction sub-parameter Algo parameter5 corresponding to the YUV processing module. Finally, the Bayer processing module performs Bayer processing, demosaicing, RGB processing, color space conversion, and YUV processing on the original image to be processed in sequence according to the image processing prediction sub-parameter Algo parameter1, the demosaicing module according to the image processing prediction sub-parameter Algo parameter2, the RGB processing module according to the image processing prediction sub-parameter Algo parameter3, the color space conversion module according to the image processing prediction sub-parameter Algo parameter4, and the YUV processing module according to the image processing prediction sub-parameter Algo parameter5, to obtain the target output image.

[0059] Of course, in other examples, the multiple image processing modules 310 may include other modules in addition to the above modules, such as a Black Level Correction (BLC) module, a Lens Shade Correction (LSC) module, a Defect Pixel Correction (DPC) module, a Wide Dynamic Range (WDR) module, a Local Tone Mapping and Global Tree Mapping (GTM and LTM) module, a Spatial Denoise module, a Temporal Denoise module, a Sharpening module, etc., which are not limited here. For each image processing module 310, corresponding image processing prediction sub-parameters can be obtained to perform corresponding image processing. In addition, the way in which the multiple image processing modules 310 perform image processing on the original image to be processed is not limited to sequential processing, but can also be parallel processing, etc.

[0060] Understandably, ISP debugging often involves strong correlations and contingencies, as the various algorithm modules are not completely independent of each other. ISP debugging requires considering the interplay between multiple algorithm modules and simultaneously debugging multiple parameters within them, which exponentially increases the number of parameter combinations that must be tried. Current manual debugging solutions rely on manually tuning the parameters within the algorithm modules, which is labor-intensive and inefficient. Furthermore, since it's impossible to exhaust every possible parameter combination, it's difficult to guarantee the optimal final tuning parameters.

[0061] The embodiment of the present application utilizes the parameter learning prediction network 101 to obtain the global optimal parameter combination of each image processing module 310 for the original image to be processed, and then outputs each image processing prediction sub-parameter to each image processing module 310 for image processing. It can take into account the mutual matching and correlation between multiple image processing prediction sub-parameters, and can find the optimal image processing prediction sub-parameter combination for the original image to be processed among a large number of parameter combinations, thereby obtaining the target output image with the best image quality.

[0062] See also Figure 5 In some embodiments, the parameter learning prediction network 101 is used to determine the image processing prediction parameters (i.e., O2) corresponding to the image processor 300 according to the original image to be processed, including:

[0063] 022: Perform convolution operation on the original image to be processed to extract low-dimensional features and high-dimensional features in the original image to be processed;

[0064] 023: Fuse low-dimensional features and high-dimensional features to obtain fused features;

[0065] 024: Output image processing prediction parameters based on fusion features.

[0066] See also Figure 2 In some embodiments, the determination module 20 may be used to implement the methods in 022, 023, and 024. That is, the determination module 20 may be used to: perform a convolution operation on the original image to be processed to extract low-dimensional features and high-dimensional features from the original image to be processed; fuse the low-dimensional features and the high-dimensional features to obtain fused features; and output image processing prediction parameters based on the fused features.

[0067] by Figure 6 For example, the process of determining the image processing prediction parameters based on the original image to be processed using the parameter learning prediction network 101 can be as follows: (1) First, a convolution operation is performed on the original image to be processed to extract low-dimensional features and high-dimensional features in the original image to be processed. For example, the convolution block (conv block) of the dual attention mechanism (ChannelAttention, SpatialAttention) can be used to perform multiple convolution operations (ATB) to respectively extract low-dimensional features and high-dimensional features from the original image to be processed. Among them, low-dimensional features are, for example, edges, colors, gradients, etc. of the original image to be processed. Low-dimensional features can be obtained by performing a convolution operation (ATB) using the convolution block of the dual attention mechanism, and high-dimensional features can be obtained by performing multiple convolution operations (ATB) using the convolution block of the dual attention mechanism. (2) Secondly, the low-dimensional features and high-dimensional features are fused to obtain fused features. For example, skip connections can be introduced in the convolution network to achieve the fusion of low-dimensional features and high-dimensional features. As Figure 6 In the example, after the first convolution operation (ATB) is performed on the original image to be processed, the result of the first convolution operation is fused with the original image to be processed without the convolution operation; then the result of the first fusion is convolution operation (ATB) performed on the result of the first fusion, and the result of the second convolution operation is fused with the result of the first convolution operation for the second time; then the result of the second fusion is convolution operation (ATB) performed on the result of the second fusion, and the result of the third convolution operation is concatenated with the original image to be processed without the convolution operation (CAT), thereby obtaining fusion features. (3) Finally, image processing prediction parameters are output based on the fusion features. For example, image processing prediction parameters (ISP parameters) can be output based on the fusion features through a normalized exponential function (Softmax).

[0068] Before performing multiple convolution operations (ATB) using the convolution block of the dual-attention mechanism, two or more front-end convolutions (conv) can be performed on the original image to be processed to facilitate feature extraction. After fusing the low-dimensional features and high-dimensional features to obtain fused features, the fused features can be subjected to two or more back-end convolutions (conv) to perform feature correction, thereby obtaining more accurate image processing prediction parameters based on the fused features. Each convolution process can use a 3*3 convolution kernel for convolution processing, which has wide applicability.

[0069] It should be pointed out that if Figure 6 The example is only one possible structure of the parameter learning prediction network 101 according to the embodiment of the present application. In other examples, the parameter learning prediction network 101 may also adopt other excellent network structures, which is not limited here.

[0070] The embodiment of the present application can learn the characteristics of the original image to be processed through the parameter learning prediction network 101, and then predict the optimal image processing prediction parameters based on the learned characteristics.

[0071] See also Figure 7 In some embodiments, the ISP debugging method further includes:

[0072] 04: Obtain a data set required for training parameter learning prediction network 101; wherein the data set includes a standard original image, standard image processing parameters corresponding to the standard original image, and a standard output image corresponding to the standard original image;

[0073] 05: Learning Prediction Networks 101 by Training Parameters on a Dataset.

[0074] See also Figure 8 In some embodiments, the ISP debugging device 100 further includes an acquisition module 40 and a training module 50. The acquisition module 40 can be used to implement the method in 04, and the training module 50 can be used to implement the method in 05. In other words, the acquisition module 40 can be used to acquire the dataset required for training the parameter learning prediction network 101; the dataset includes a standard original image, standard image processing parameters corresponding to the standard original image, and a standard output image corresponding to the standard original image. The training module 50 can be used to train the parameter learning prediction network 101 using the dataset.

[0075] Specifically, before using the parameter learning prediction network 101 to determine the image processing prediction parameters, a data set required for training the parameter learning prediction network 101 can be obtained, and the data set can be used to train the parameter learning prediction network 101. The trained parameter learning prediction network 101 can accurately output the image processing prediction parameters, thereby achieving the optimal image processing effect.

[0076] The data set required for training the parameter learning prediction network 101 includes a standard original image, image processing standard parameters corresponding to the standard original image, and a standard output image corresponding to the standard original image. Among them, the standard original image can be acquired by the image sensor 200 in various shooting scenes (for example, it can cover dark light scenes at night, bright light scenes during the day, outdoor scenes, indoor scenes, image shooting scenes, video shooting scenes, etc.). The image processing standard parameters corresponding to the standard original image can adopt the better parameters debugged by experienced engineers (although manual debugging parameters are also used here, it is only necessary to manually debug the optimal parameters in the model training link of the parameter learning prediction network 101. After the model is trained, the parameter learning prediction network 101 can be directly used to obtain the image processing prediction parameters, which saves a lot of workload). The standard output image corresponding to the standard original image can be obtained by the artificial intelligence image processing network 102 (such as Figure 10 The standard original image is processed as shown (detailed introduction will be given later).

[0077] In the embodiments of the present application, there may be multiple sets of data sets, each set comprising a standard original image, standard image processing parameters corresponding to the standard original image, and a standard output image corresponding to the standard original image. For example, the first set of data sets may comprise: standard original image 1, standard image processing parameters 1, and standard output image 1; the second set of data sets may comprise: standard original image 2, standard image processing parameters 2, and standard output image 2; the third set of data sets may comprise: standard original image 3, standard image processing parameters 3, and standard output image 3; and so on. The parameter learning and prediction network 101 may be trained sequentially using multiple sets of data sets to optimize the model of the parameter learning and prediction network 101, thereby enabling learning and prediction of the optimal parameter combination.

[0078] In addition, when the parameter learning prediction network 101 is trained through the data set, it can be set to learn only part of the image processing prediction sub-parameters in the predicted image processing prediction parameters, for example, only the image processing prediction sub-parameters corresponding to the key image processing module 310 (i.e., the image processing prediction sub-parameters corresponding to the image processing module 310 that have a significant impact on the effect of the final target output image, such as Figure 3 The image processing prediction sub-parameter Algo parameter3 corresponding to the RGB processing module, the image processing prediction sub-parameter Algo parameter5 corresponding to the YUV processing module, etc.) is used to reduce the number of training parameters, thereby reducing the training difficulty of the parameter learning prediction network 101, while obtaining a better parameter combination.

[0079] See also Figure 9 and Figure 10In some embodiments, the prediction network 101 (i.e., 05) is trained with a data set, including:

[0080] 051: Input the standard original image into the parameter learning prediction network 101;

[0081] 052: Determine actual image processing parameters corresponding to the image processor 300 according to the standard original image using the parameter learning prediction network 101;

[0082] 053: Outputting the actual image processing parameters to the image processor 300, so that the image processor 300 performs image processing on the standard original image according to the actual image processing parameters to obtain an actual output image;

[0083] 054: Determine whether the parameter learning prediction network 101 is trained based on the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image.

[0084] See also Figure 8 and Figure 10 In certain embodiments, the training module 50 can be used to implement the methods in 051, 052, 053, and 054. That is, the training module 50 can be used to: input the standard original image into the parameter learning and prediction network 101; use the parameter learning and prediction network 101 to determine actual image processing parameters corresponding to the image processor 300 based on the standard original image; output the actual image processing parameters to the image processor 300, so that the image processor 300 performs image processing on the standard original image based on the actual image processing parameters to obtain an actual output image; and determine whether the parameter learning and prediction network 101 is trained based on the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image.

[0085] Specifically, please combine Figure 10The standard original image is input into the parameter learning prediction network 101 to obtain actual image processing parameters. The image processor 300 performs image processing on the standard original image according to the actual image processing parameters to obtain an actual output image. The difference between the actual image processing parameters and the standard image processing parameters, as well as the difference between the actual output image and the standard output image, can be used to determine whether the parameter learning prediction network 101 has been trained. Generally, the smaller the difference between the actual image processing parameters and the standard image processing parameters, and the smaller the difference between the actual output image and the standard output image, the closer the actual image processing parameters determined by the parameter learning prediction network 101 are to the standard image processing parameters debugged by experienced engineers, and the closer the actual output image obtained by image processing according to the actual image processing parameters is to the standard output image obtained by image processing by the artificial intelligence image processing network 102, then the actual image processing parameters determined by the parameter learning prediction network 101 are more ideal, that is, the parameter learning prediction network 101 has been trained (or has passed the training).

[0086] It should be noted that steps 051 to 054 illustrate the process of training parameter learning prediction network 101 using only one dataset. If multiple datasets are used, the same or similar method can be used to train parameter learning prediction network 101, which will not be further described here. If parameter learning prediction network 101 is not fully trained using one dataset, the next dataset can be used to train parameter learning prediction network 101 until training is complete.

[0087] The implementation method of the present application combines the differences in the parameters themselves and the differences in the image effects to determine whether the parameter learning prediction network 101 has been trained. The judgment criteria are relatively comprehensive and reasonable. If the parameter learning prediction network 101 has not been trained, the parameter learning prediction network 101 can continue to be trained through the data set to make the model of the parameter learning prediction network 101 better.

[0088] See also Figure 11 In some embodiments, judging whether the parameter learning prediction network 101 is trained (i.e., 054) based on the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image, includes:

[0089] 0541: Calculating a first loss value using a first loss function according to a difference between actual image processing parameters and standard image processing parameters;

[0090] 0542: Calculate a second loss value using a second loss function according to the difference between the actual output image and the standard output image;

[0091] 0543: Perform weighted processing on the first loss value and the second loss value to obtain a comprehensive loss value;

[0092] 0544: When the comprehensive loss value meets the predetermined conditions, it is determined that the training of the parameter learning prediction network 101 is completed.

[0093] See also Figure 8 In some embodiments, the training module 50 can be used to implement the methods in 0541, 0542, 0543, and 0544. That is, the training module 50 can be used to: calculate a first loss value using a first loss function based on the difference between actual image processing parameters and standard image processing parameters; calculate a second loss value using a second loss function based on the difference between the actual output image and the standard output image; perform weighted processing on the first loss value and the second loss value to obtain a comprehensive loss value; and determine that the training of the parameter learning prediction network 101 is complete when the comprehensive loss value meets a predetermined condition.

[0094] Specifically, please combine Figure 10 , a first loss function can be used to calculate the difference between the actual image processing parameters and the standard image processing parameters to obtain a first loss value, and a second loss function can be used to calculate the difference between the actual output image and the standard output image to obtain a second loss value. Among them, the first loss function can use the cross entropy loss function (Cross EntropyLoss) or the normalized regression loss function (L1 loss). The second loss function can use a loss function in the field of low-level image processing, such as the mean square loss function (MSE loss), edge loss function (Edgeloss), structural similarity loss function (SSIM loss), etc.

[0095] In an example, the specific calculation formulas for the first loss value parameters loss and the second loss value image loss can be as follows:

[0096]

[0097]

[0098] Furthermore, the first loss value parameters loss and the second loss value image loss are weighted to obtain the comprehensive loss value loss. The specific calculation formula can be shown as follows:

[0099]

[0100] Among them, w p and w iare the weights of the first loss value parameters loss and the second loss value image loss in the weighted processing. n and N are the number of parameters and image pixels involved in the loss function calculation respectively. j is the standard parameter for image processing, a j is the actual parameter of image processing, y k is the image pixel value of the actual output image, Image pixel values ​​for the standard output image.

[0101] After calculating the comprehensive loss value, when the comprehensive loss value meets the predetermined conditions, the parameter learning prediction network 101 is determined to be trained. The comprehensive loss value meeting the predetermined conditions may be, for example: ① the comprehensive loss value is less than the loss value threshold, or ② the change in the comprehensive loss value is less than the change threshold; or ③ the comprehensive loss value is less than the loss value threshold and the change in the comprehensive loss value is less than the change threshold. For the first case, taking the loss value threshold as 0.1 as an example, if the comprehensive loss value obtained by training the parameter learning prediction network 101 using the data set is 0.2, the parameter learning prediction network 101 is not trained. The parameter learning prediction network 101 is trained using the data set until the comprehensive loss value obtained is less than 0.1, indicating that both the parameter difference and the image difference are small, and the parameter learning prediction network 101 is trained. For the second case, taking the change threshold of 0.02 as an example, if the change in the comprehensive loss value obtained by training the parameter learning prediction network 101 with different data sets for a predetermined number of consecutive times is 0.03, the parameter learning prediction network 101 is not trained. The parameter learning prediction network 101 is trained again using the data set until the change in the comprehensive loss value is less than 0.02, indicating that the model of the parameter learning prediction network 101 has reached the optimal level and cannot be further optimized. The training of the parameter learning prediction network 101 is completed. For the third case, the first and second cases are combined and will not be further explained here.

[0102] See also Figure 10 and Figure 12 In some embodiments, obtaining a data set (i.e., 04) required for training parameter learning prediction network 101 includes:

[0103] 041: Input the standard original image into the artificial intelligence image processing network 102;

[0104] 042: Use the artificial intelligence image processing network 102 to process the standard original image to obtain a standard output image.

[0105] See also Figure 8 and Figure 10In some embodiments, the acquisition module 40 can be used to implement the methods in 041 and 042. That is, the acquisition module 40 can be used to: input the standard original image to the artificial intelligence image processing network 102; and use the artificial intelligence image processing network 102 to perform image processing on the standard original image to obtain a standard output image.

[0106] Specifically, the embodiment of the present application introduces an artificial intelligence image processing network (AI ISPNetwork) 102 to perform image processing on the standard original image to obtain a corresponding standard output image, instead of outputting the image processing standard parameters to the image processor 300, so that the image processor 300 performs image processing on the standard original image according to the image processing standard parameters to obtain a standard output image. This can effectively utilize the big data processing function of the artificial intelligence image processing network 102 to improve the quality of the standard output image and obtain a better standard output image as a reference for comparison with the actual output image. At the same time, it also reduces the image processing error of the image processor 300 itself, and evaluates the image effect difference from another aspect (the first loss value is obtained by the difference between the actual image processing parameters and the standard image processing parameters. If the second loss value is also obtained by the difference between the actual output image obtained by the image processor 300 according to the actual image processing parameters and the standard output image obtained by the image processor 300 according to the standard image processing parameters, then the two are essentially differences caused by the same factor). This is conducive to enabling the parameter learning and prediction network 101 to learn the image processing prediction parameters with the best characteristics as much as possible, thereby ensuring that the model of the parameter learning and prediction network 101 is optimal.

[0107] It is understood that before the artificial intelligence image processing network 102 is used for image processing, the artificial intelligence image processing network 102 has already converged (ie, the model has been trained in advance) so that the standard original image can be processed to obtain a standard output image. Figure 13The process by which the artificial intelligence image processing network 102 processes a standard original image to obtain a standard output image can be divided into two parts. The first part is Bayer domain processing, which is used to process standard original images in RGGB format. The second part is RGB and YUV domain processing. The input is the RGB result output by the first part. Then, certain operations are performed on the image in the RGB and YUV domains. After the two parts are processed, a standard output image in YUV format can be obtained. In particular, multiple convolutions (conv) and optimized convolutions (RSG) are involved in both the first and second parts. The optimized convolution (RSG), like the aforementioned convolution operation (ATB), uses a convolution block (conv block) with a dual attention mechanism (Channel attention and Spatial attention). The difference is that the optimized convolution (RSG) also superimposes the input data and the output data.

[0108] It should be pointed out that if Figure 13 The example is only one possible structure of the artificial intelligence image processing network 102 of the embodiment of the present application. In other examples, the artificial intelligence image processing network 102 can also adopt other excellent network structures or software algorithm solutions, which are not limited here.

[0109] See also Figure 14 In some embodiments, the standard original image is input into the parameter learning prediction network 101 (i.e., 051), including:

[0110] 0511: Input a set of standard original images into the parameter learning prediction network 101; wherein the set of standard original images includes a plurality of different standard original images, and the image processing standard parameters corresponding to the plurality of different standard original images are the same;

[0111] The parameter learning prediction network 101 is used to determine the actual image processing parameters (i.e., 052) corresponding to the image processor 300 according to the standard original image, including:

[0112] 0521: Determine a set of actual image processing parameters corresponding to the image processor 300 according to a set of standard original images using the parameter learning prediction network 101;

[0113] Outputting the actual image processing parameters to the image processor 300 so that the image processor 300 performs image processing on the standard original image according to the actual image processing parameters to obtain an actual output image (i.e., 053), including:

[0114] 0531: Outputting a set of image processing actual parameters to the image processor 300, so that the image processor 300 performs image processing on a corresponding set of standard original images according to the set of image processing actual parameters to obtain a set of actual output images;

[0115] Judging whether the parameter learning prediction network 101 is trained (i.e., 054) based on the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image, includes:

[0116] 0545: Determine whether the parameter learning prediction network 101 is trained based on the difference between a set of actual image processing parameters and a set of standard image processing parameters, and the difference between a set of actual output images and a set of standard output images.

[0117] See also Figure 8 In certain embodiments, the training module 50 can be used to implement the methods in 0511, 0521, 0531, and 0545. That is, the training module 50 can be used to: input a set of standard original images into the parameter learning and prediction network 101; wherein the set of standard original images includes a plurality of different standard original images, and the plurality of different standard original images correspond to the same standard image processing parameters; utilize the parameter learning and prediction network 101 to determine a set of actual image processing parameters corresponding to the image processor 300 based on the set of standard original images; output the set of actual image processing parameters to the image processor 300, so that the image processor 300 performs image processing on the corresponding set of standard original images according to the set of actual image processing parameters to obtain a set of actual output images; and determine whether the parameter learning and prediction network 101 is trained based on the difference between the set of actual image processing parameters and the standard image processing parameters, and the difference between the set of actual output images and the set of standard output images.

[0118] Understandably, in video display or preview scenarios, the original image being processed changes frame by frame. Even small changes in the original image can cause significant changes in the image processing prediction parameters, and thus in the target output image. This can ultimately lead to noticeable flicker in the video. Therefore, a targeted smooth transition algorithm is needed to prevent video flicker.

[0119] The main idea of ​​the smooth transition algorithm of the embodiment of the present application is that when the parameter learning prediction network 101 is trained through a data set, a single frame of standard original image is not used to feed into the parameter learning prediction network 101 to predict the actual parameters of image processing, but a group of standard original images are input into the parameter learning prediction network 101 to obtain a corresponding set of actual parameters for image processing. Among them, a group of standard original images includes multiple standard original images with different image contents (for example, from the current standard original image, a predetermined number of standard original images are added), but the image processing standard parameters corresponding to the multiple standard original images are set to the same. When judging whether the parameter learning prediction network 101 is trained, it is based on the difference between a group of actual image processing parameters and the image processing standard parameters, and the difference between a group of actual output images and a group of standard output images. It can be understood that if the difference between a set of actual image processing parameters and the image processing standard parameters is large, the parameter learning prediction network 101 may not be able to complete convergence, and it is necessary to continue training the parameter learning prediction network 101 until a set of actual image processing parameters are close to the image processing standard parameters. In this way, the image processing prediction parameters and the target output image obtained in actual application will not change significantly, and a video without flickering problems can be obtained.

[0120] If there are significant image differences between multiple standard original images in a set of standard original images, it may not be reasonable to continue using the same standard image processing parameters. However, in this case, the set of standard output images obtained by image processing based on the set of standard original images will also be significantly different. As a result, the difference between a set of actual output images and a set of standard output images is significant. In this case, the parameter learning prediction network 101 will not be able to complete convergence, and the parameter learning prediction network 101 will continue to be trained to optimize the model of the parameter learning prediction network 101. Therefore, during the training phase of the parameter learning prediction network 101, there is no need to consider whether there are significant image differences between multiple standard original images in a set of standard original images, and the model of the parameter learning prediction network 101 can be ensured to be optimal.

[0121] See also Figure 15 In some embodiments, the ISP debugging method further includes:

[0122] 06: Calculate the image difference between the original image to be processed in the current frame and the original image to be processed in the previous frame;

[0123] 07: Set the weight ratio between the image processing prediction parameters corresponding to the original image to be processed in the previous frame and the image processing prediction parameters corresponding to the original image to be processed in the current frame according to the image difference; the weight ratio is positively correlated with the image difference;

[0124] 08: Perform weighted processing on the image processing prediction parameters corresponding to the original image to be processed in the previous frame and the image processing prediction parameters corresponding to the original image to be processed in the current frame according to the weight ratio to obtain image processing correction parameters, and update the image processing prediction parameters corresponding to the original image to be processed in the current frame as image processing correction parameters.

[0125] See also Figure 16 In some embodiments, the ISP debugging device 100 further includes a calculation module 60, a setting module 70, and a processing module 80. The calculation module 60 can be used to implement the method in 06, the setting module 70 can be used to implement the method in 07, and the processing module 80 can be used to implement the method in 08.

[0126] That is, the calculation module 60 can be used to calculate the image difference between the original image to be processed of the current frame and the original image to be processed of the previous frame. The setting module 70 can be used to set the weight ratio between the image processing prediction parameters corresponding to the original image to be processed of the previous frame and the image processing prediction parameters corresponding to the original image to be processed of the current frame according to the image difference; wherein the weight ratio is positively correlated with the image difference. The processing module 80 can be used to perform weighted processing on the image processing prediction parameters corresponding to the original image to be processed of the previous frame and the image processing prediction parameters corresponding to the original image to be processed of the current frame according to the weight ratio to obtain image processing correction parameters, and update the image processing prediction parameters corresponding to the original image to be processed of the current frame to the image processing correction parameters.

[0127] Specifically, to address the flickering problem in the aforementioned video, the smooth transition algorithm of the embodiment of the present application, when actually applied (i.e., when using the parameter learning prediction network 101 to determine the image processing prediction parameters), adopts the idea of ​​digital filtering to linearly fuse the image processing prediction parameters obtained each time with the image processing prediction parameters of the previous frame to prevent the video flickering problem caused by the discontinuity of the image processing prediction parameters. When performing linear fusion, the weight ratio is mainly based on the image difference between the previous frame and the current frame, such as the difference in image metadata. The weight ratio refers to the ratio of the weight of the image processing prediction parameters of the previous frame to the weight of the image processing prediction parameters of the current frame. The weight ratio is positively correlated with the image difference. If the image difference is large, the weight of the image processing prediction parameters of the previous frame is relatively larger (it can be larger relative to the weight of the image processing prediction parameters of the current frame, or it can be larger relative to the weight of the image processing prediction parameters of the previous frame when the image difference is small); if the image difference is small, the weight of the image processing prediction parameters of the current frame is relatively larger (it can be larger relative to the weight of the image processing prediction parameters of the previous frame, or it can be larger relative to the weight of the image processing prediction parameters of the current frame when the image difference is large). The image processing correction parameters after linear fusion of the current frame and the previous frame can be saved for use in the next frame, so that when determining the image processing correction parameters of the next frame, the image processing prediction parameters of the current frame (which have been updated to the image processing correction parameters after linear fusion) are linearly fused with the image processing prediction parameters of the next frame.

[0128] In some embodiments, the image differences include exposure parameter differences and / or color temperature parameter differences.

[0129] The exposure parameters include exposure time and / or exposure gain, and the color temperature parameters include color temperature.

[0130] Taking the exposure parameters including exposure time and exposure gain value as an example, if the exposure time and exposure gain value of the current frame are significantly different from those of the previous frame (for example, the exposure time is greater than the time threshold and the exposure gain value is greater than the gain threshold), the weight of the image processing prediction parameters of the previous frame is relatively larger; if the exposure time and exposure gain value of the current frame are slightly different from those of the previous frame (for example, the exposure time is less than or equal to the time threshold or the exposure gain value is less than or equal to the gain threshold), the weight of the image processing prediction parameters of the current frame is relatively larger.

[0131] Taking the color temperature parameter including the color temperature value as an example, if the color temperature value of the current frame is significantly different from that of the previous frame (for example, the color temperature value is greater than the color temperature threshold), the weight of the image processing prediction parameter of the previous frame is relatively larger; if the color temperature value of the current frame is slightly different from that of the previous frame (for example, the color temperature value is less than or equal to the color temperature threshold), the weight of the image processing prediction parameter of the current frame is relatively larger.

[0132] In summary, the ISP debugging method and ISP debugging device 100 according to the embodiments of the present application have at least the following effects:

[0133] First, the parameter learning prediction network 101 determines the image processing prediction parameters corresponding to the image processor 300 based on the original image to be processed, which can fully utilize the huge computing performance of the parameter learning prediction network 101 and does not require excessive human intervention during application to find the optimal image processing prediction parameters for the original image to be processed.

[0134] Second, it provides a complete neural network-based ISP debugging solution, which provides corresponding solutions from the early network design, data set generation and production, loss function design, to the smoothing strategy during final use.

[0135] Third, compared with the method of manually debugging the ISP to generate a standard output image, using the artificial intelligence image processing network 102 to generate a standard output image can further improve the quality of the standard output image, thereby enabling the parameter learning prediction network 101 to learn the image processing prediction parameters with the best characteristics as much as possible.

[0136] Fourth, a smooth transition algorithm training strategy was designed, combined with a designed digital filtering scheme, to ensure that parameter learning and prediction network 101 produces flicker-free video. Furthermore, the parameter learning and prediction network 101 and artificial intelligence image processing network 102 of the present embodiment are not limited to a specific network structure and can be adapted to any excellent network structure.

[0137] See also Figure 2 and Figure 3 The present application also provides an image processing system 400. The image processing system 400 includes an image sensor 200, an ISP debugging device 100, and an image processor 300. The image sensor 200 is used to capture a raw image to be processed. The ISP debugging device 100 is used to: input the raw image to be processed into a parameter learning and prediction network 101; use the parameter learning and prediction network 101 to determine image processing prediction parameters corresponding to the image processor 300 based on the raw image to be processed; and output the image processing prediction parameters to the image processor 300. The image processor 300 is used to perform image processing on the raw image to be processed based on the image processing prediction parameters to obtain a target output image.

[0138] In certain embodiments, the image processor 300 includes multiple image processing modules 310. Image processing prediction parameters include multiple image processing prediction sub-parameters. The ISP debugging device 100 is configured to: utilize the parameter learning prediction network 101 to determine, based on the original image to be processed, multiple image processing prediction sub-parameters corresponding to the multiple image processing modules 310; and output the multiple image processing prediction sub-parameters to the image processor 300, so that the multiple image processing modules 310 perform image processing on the original image to be processed according to the corresponding multiple image processing prediction sub-parameters to obtain a target output image.

[0139] In some embodiments, the ISP debugging device 100 is used to: perform a convolution operation on the original image to be processed to extract low-dimensional features and high-dimensional features in the original image to be processed; fuse the low-dimensional features and the high-dimensional features to obtain fused features; and output image processing prediction parameters based on the fused features.

[0140] In some embodiments, the ISP debugging device 100 is used to: obtain a data set required for training a parameter learning prediction network 101; wherein the data set includes a standard original image, standard image processing parameters corresponding to the standard original image, and a standard output image corresponding to the standard original image; and train the parameter learning prediction network 101 through the data set.

[0141] In some embodiments, the ISP debugging device 100 is used to: input a standard original image into the parameter learning prediction network 101; use the parameter learning prediction network 101 to determine actual image processing parameters corresponding to the image processor 300 based on the standard original image; output the actual image processing parameters to the image processor 300, so that the image processor 300 performs image processing on the standard original image according to the actual image processing parameters to obtain an actual output image; and determine whether the parameter learning prediction network 101 is trained based on the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image.

[0142] In some embodiments, the ISP debugging device 100 is used to: calculate a first loss value using a first loss function based on the difference between actual image processing parameters and standard image processing parameters; calculate a second loss value using a second loss function based on the difference between the actual output image and the standard output image; perform weighted processing on the first loss value and the second loss value to obtain a comprehensive loss value; when the comprehensive loss value meets a predetermined condition, it is determined that the training of the parameter learning prediction network 101 is completed.

[0143] In some embodiments, the ISP debugging device 100 is used to: input a standard original image into the artificial intelligence image processing network 102; and use the artificial intelligence image processing network 102 to perform image processing on the standard original image to obtain a standard output image.

[0144] In some embodiments, the ISP debugging device 100 is used to: input a group of standard original images into the parameter learning prediction network 101; wherein the group of standard original images includes multiple different standard original images, and the image processing standard parameters corresponding to the multiple different standard original images are the same; use the parameter learning prediction network 101 to determine a group of image processing actual parameters corresponding to the image processor 300 based on the group of standard original images; output the group of image processing actual parameters to the image processor 300, so that the image processor 300 performs image processing on the corresponding group of standard original images according to the group of image processing actual parameters to obtain a group of actual output images; and judge whether the parameter learning prediction network 101 is trained based on the difference between the group of image processing actual parameters and the image processing standard parameters, and the difference between the group of actual output images and the group of standard output images.

[0145] In some embodiments, the ISP debugging device 100 is used to: calculate the image difference between the original image to be processed of the current frame and the original image to be processed of the previous frame; set the weight ratio between the image processing prediction parameters corresponding to the original image to be processed of the previous frame and the image processing prediction parameters corresponding to the original image to be processed of the current frame according to the image difference; wherein the weight ratio is positively correlated with the image difference; perform weighted processing on the image processing prediction parameters corresponding to the original image to be processed of the previous frame and the image processing prediction parameters corresponding to the original image to be processed of the current frame according to the weight ratio to obtain image processing correction parameters, and update the image processing prediction parameters corresponding to the original image to be processed of the current frame to the image processing correction parameters.

[0146] In some embodiments, the image differences include exposure parameter differences and / or color temperature parameter differences.

[0147] It should be noted that the explanations of the ISP debugging method and the ISP debugging device 100 in the aforementioned embodiment are also applicable to the image processing system 400 in the embodiment of the present application, and will not be elaborated here.

[0148] See also Figure 17 The present application also provides a terminal 500. The terminal 500 includes one or more processors 510 and a memory 520. The memory 520 stores a computer program, and when the computer program is executed by the processor 510, the ISP debugging method of any of the above embodiments is implemented.

[0149] For example, when the computer program is executed by the processor 510, the following ISP debugging method is implemented:

[0150] 01: Input the original image to be processed into the parameter learning prediction network 101;

[0151] 02: Using the parameter learning prediction network 101 to determine the image processing prediction parameters corresponding to the image processor 300 according to the original image to be processed;

[0152] 03: Output the image processing prediction parameters to the image processor 300, so that the image processor 300 performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0153] For another example, when the computer program is executed by the processor 510, the following ISP debugging method is implemented:

[0154] 021: Determine, using the parameter learning prediction network 101, a plurality of image processing prediction sub-parameters corresponding to the plurality of image processing modules 310 according to the original image to be processed;

[0155] 031: Outputting the plurality of image processing prediction sub-parameters to the image processor 300, so that the plurality of image processing modules 310 respectively perform image processing on the original image to be processed according to the corresponding plurality of image processing prediction sub-parameters to obtain a target output image.

[0156] The terminal 500 in the embodiment of the present application includes but is not limited to a mobile phone, a tablet computer, a camera, a video camera, a personal digital assistant, a wearable device, an intelligent robot, an intelligent vehicle, etc. Among them, the wearable device includes a smart bracelet, a smart watch, smart glasses, etc.

[0157] It should be noted that the explanations of the ISP debugging method and the ISP debugging device 100 in the aforementioned embodiments are also applicable to the terminal 500 in the embodiments of the present application, and will not be further described here.

[0158] See also Figure 18 The present application also provides a computer-readable storage medium 600 on which a computer program 610 is stored. When the program 610 is executed by the processor 620, the ISP debugging method of any of the above embodiments is implemented.

[0159] For example, when the program 610 is executed by the processor 620, the following ISP debugging method is implemented:

[0160] 01: Input the original image to be processed into the parameter learning prediction network 101;

[0161] 02: Using the parameter learning prediction network 101 to determine the image processing prediction parameters corresponding to the image processor 300 according to the original image to be processed;

[0162] 03: Output the image processing prediction parameters to the image processor 300, so that the image processor 300 performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image.

[0163] For another example, when the program 610 is executed by the processor 620, the following ISP debugging method is implemented:

[0164] 021: Determine, using the parameter learning prediction network 101, a plurality of image processing prediction sub-parameters corresponding to the plurality of image processing modules 310 according to the original image to be processed;

[0165] 031: Outputting the plurality of image processing prediction sub-parameters to the image processor 300, so that the plurality of image processing modules 310 respectively perform image processing on the original image to be processed according to the corresponding plurality of image processing prediction sub-parameters to obtain a target output image.

[0166] It should be noted that the explanations of the ISP debugging method and the ISP debugging device 100 in the aforementioned embodiments are also applicable to the computer-readable storage medium 600 in the embodiments of the present application, and will not be further described here.

[0167] In summary, in the ISP debugging method, ISP debugging device 100, image processing system 400, terminal 500 and computer-readable storage medium 600 of the embodiments of the present application, the parameter learning and prediction network 101 determines the image processing prediction parameters corresponding to the image processor 300 based on the original image to be processed, which can fully utilize the huge computing performance of the parameter learning and prediction network 101. When applied, it does not require excessive human participation and can find the optimal image processing prediction parameters for the original image to be processed.

[0168] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0169] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0170] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner as necessary, and then stored in a computer memory.

[0171] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0172] Those skilled in the art will appreciate that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, the various functional units in the various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc.

[0173] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are illustrative and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. An ISP debugging method, characterized in that: include: Input the original image to be processed into the parameter learning prediction network; Determining image processing prediction parameters corresponding to an image processor according to the original image to be processed using the parameter learning prediction network; Outputting the image processing prediction parameters to the image processor, so that the image processor performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image; The ISP debugging method further includes: Calculating an image difference between the original image to be processed in the current frame and the original image to be processed in the previous frame; Setting a weight ratio between the image processing prediction parameter corresponding to the original image to be processed in the previous frame and the image processing prediction parameter corresponding to the original image to be processed in the current frame according to the image difference; wherein the weight ratio is positively correlated with the image difference; According to the weight ratio, the image processing prediction parameters corresponding to the original image to be processed in the previous frame and the image processing prediction parameters corresponding to the original image to be processed in the current frame are weighted to obtain image processing correction parameters, and the image processing prediction parameters corresponding to the original image to be processed in the current frame are updated to the image processing correction parameters.

2. The ISP debugging method according to claim 1, characterized in that: The image processor includes a plurality of image processing modules, the image processing prediction parameters include a plurality of image processing prediction sub-parameters, and the method of using the parameter learning prediction network to determine the image processing prediction parameters corresponding to the image processor according to the original image to be processed includes: Determining, by using the parameter learning prediction network, a plurality of image processing prediction sub-parameters corresponding to the plurality of image processing modules according to the original image to be processed; Outputting the image processing prediction parameters to the image processor so that the image processor performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image includes: The plurality of image processing prediction sub-parameters are output to the image processor, so that the plurality of image processing modules respectively perform image processing on the original image to be processed according to the corresponding plurality of image processing prediction sub-parameters to obtain the target output image.

3. The ISP debugging method according to claim 1, wherein: The step of using the parameter learning prediction network to determine image processing prediction parameters corresponding to the image processor according to the original image to be processed includes: Performing a convolution operation on the original image to be processed to extract low-dimensional features and high-dimensional features from the original image to be processed; Fusing the low-dimensional features and the high-dimensional features to obtain fused features; The image processing prediction parameter is output based on the fusion feature.

4. The ISP debugging method according to claim 1, wherein: The ISP debugging method further includes: Acquire a data set required for training the parameter learning prediction network; wherein the data set includes a standard original image, standard image processing parameters corresponding to the standard original image, and a standard output image corresponding to the standard original image; The parameter learning prediction network is trained using the data set.

5. The ISP debugging method according to claim 4, characterized in that: The step of training the parameter learning prediction network using the data set includes: Inputting the standard original image into the parameter learning prediction network; Determining actual image processing parameters corresponding to the image processor according to the standard original image using the parameter learning prediction network; Outputting the actual image processing parameters to the image processor, so that the image processor performs image processing on the standard original image according to the actual image processing parameters to obtain an actual output image; Whether the parameter learning prediction network is trained is determined based on the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image.

6. The ISP debugging method according to claim 5, characterized in that: The determining whether the parameter learning prediction network is trained according to the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image, includes: Calculating a first loss value using a first loss function according to a difference between the actual image processing parameters and the standard image processing parameters; Calculating a second loss value using a second loss function according to a difference between the actual output image and the standard output image; Performing weighted processing on the first loss value and the second loss value to obtain a comprehensive loss value; When the comprehensive loss value meets a predetermined condition, it is determined that the parameter learning prediction network training is completed.

7. The ISP debugging method according to claim 4, characterized in that: The step of obtaining a data set required for training the parameter learning prediction network includes: Inputting the standard original image into an artificial intelligence image processing network; The standard original image is processed using the artificial intelligence image processing network to obtain the standard output image.

8. The ISP debugging method according to claim 5, characterized in that: The step of inputting the standard original image into the parameter learning prediction network comprises: Inputting a group of the standard original images into the parameter learning prediction network; wherein the group of the standard original images includes a plurality of different standard original images, and the image processing standard parameters corresponding to the plurality of different standard original images are the same; The method of using the parameter learning prediction network to determine actual image processing parameters corresponding to the image processor according to the standard original image includes: Determining a set of image processing actual parameters corresponding to the image processor according to a set of standard original images using the parameter learning prediction network; Outputting the actual image processing parameters to the image processor so that the image processor performs image processing on the standard original image according to the actual image processing parameters to obtain an actual output image includes: Outputting a set of the image processing actual parameters to the image processor, so that the image processor performs image processing on a corresponding set of the standard original images according to the set of the image processing actual parameters to obtain a set of the actual output images; The determining whether the parameter learning prediction network is trained according to the difference between the actual image processing parameters and the standard image processing parameters, and the difference between the actual output image and the standard output image, includes: Whether the parameter learning prediction network is trained is determined based on the difference between a set of actual image processing parameters and the standard image processing parameters, and the difference between a set of actual output images and a set of standard output images.

9. The ISP debugging method according to claim 1, wherein: The image difference includes exposure parameter difference and / or color temperature parameter difference.

10. An ISP debugging device, characterized in that: include: An input module, used to input the original image to be processed into the parameter learning prediction network; a determination module, configured to determine image processing prediction parameters corresponding to an image processor according to the original image to be processed using the parameter learning prediction network; an output module, configured to output the image processing prediction parameters to the image processor, so that the image processor performs image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image; The ISP debugging device also includes: a calculation module, configured to calculate an image difference between the original image to be processed in a current frame and the original image to be processed in a previous frame; a setting module for setting a weight ratio between the image processing prediction parameter corresponding to the original image to be processed in the previous frame and the image processing prediction parameter corresponding to the original image to be processed in the current frame according to the image difference; wherein the weight ratio is positively correlated with the image difference; A processing module is used to perform weighted processing on the image processing prediction parameters corresponding to the original image to be processed in the previous frame and the image processing prediction parameters corresponding to the original image to be processed in the current frame according to the weight ratio to obtain image processing correction parameters, and update the image processing prediction parameters corresponding to the original image to be processed in the current frame to the image processing correction parameters.

11. An image processing system, characterized in that: include: An image sensor, used for collecting raw images to be processed; ISP debugging device, used for: Inputting the original image to be processed into a parameter learning prediction network; Determining image processing prediction parameters corresponding to an image processor according to the original image to be processed using the parameter learning prediction network; outputting the image processing prediction parameters to the image processor; An image processor, configured to perform image processing on the original image to be processed according to the image processing prediction parameters to obtain a target output image; The ISP debugging device is also used for: Calculating an image difference between the original image to be processed in the current frame and the original image to be processed in the previous frame; Setting a weight ratio between the image processing prediction parameter corresponding to the original image to be processed in the previous frame and the image processing prediction parameter corresponding to the original image to be processed in the current frame according to the image difference; wherein the weight ratio is positively correlated with the image difference; According to the weight ratio, the image processing prediction parameters corresponding to the original image to be processed in the previous frame and the image processing prediction parameters corresponding to the original image to be processed in the current frame are weighted to obtain image processing correction parameters, and the image processing prediction parameters corresponding to the original image to be processed in the current frame are updated to the image processing correction parameters.

12. A terminal, characterized in that: The terminal includes one or more processors and a memory, the memory stores a computer program, and when the computer program is executed by the processor, the ISP debugging method according to any one of claims 1 to 9 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the ISP debugging method according to any one of claims 1 to 9 is implemented.

Images