Image processing model training method, device, electronic device and storage medium

Abstract

The present invention discloses a training method, apparatus, electronic device, and storage medium for an image processing model. The method comprises: obtaining a plurality of sample image data; the sample image data being RAW image data; using the obtained plurality of sample image data as training data, and performing quantitative perception model training on the plurality of sample image data to obtain a quantitative perception model; wherein, when performing the quantitative perception model training on the plurality of sample image data, weight parameters of the quantitative perception model are updated based on at least one influencing factor related to an image signal processing (ISP) unit.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a training method, device, electronic device and storage medium for an image processing model. Background Art

[0002] With the rapid development of image processing technology, in order to improve the efficiency of image processing, the trained image processing model can be used to map the input image data to obtain floating-point image data, and then the fixed-point model can be used to convert the floating-point image data into fixed-point image data. Finally, image signal processing (ISP) is performed on the fixed-point image data output by the fixed-point model to obtain processed image data. However, in actual applications, the quality of images output by the image processing model and the fixed-point model is better, but the quality of the image obtained after ISP processing is worse.

[0003] Therefore, it is urgent to find a technical solution that can improve image quality. Summary of the Invention

[0004] In view of this, embodiments of the present invention hope to provide a training method, device, electronic device and storage medium for an image processing model.

[0005] The technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a method for training an image processing model, the method comprising:

[0007] Acquire a plurality of sample image data; the sample image data is RAW image data;

[0008] Using the obtained multiple sample image data as training data, the multiple sample image data are trained with a quantization perception model to obtain a quantization perception model; the quantization perception model is used to quantize the target image data to obtain floating-point target image data; and the floating-point target image data is converted into fixed-point target image data, so that the fixed-point target image data meets the processing requirements of the ISP unit;

[0009] Wherein, when the quantitative perception model training is performed on the multiple sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit.

[0010] In the above solution, updating the weight parameters of the quantitative perception model based on at least one influencing factor related to the ISP unit includes:

[0011] Determining at least one image processing function of the ISP unit for image processing;

[0012] determining at least one mapping function corresponding to the at least one image processing function;

[0013] Using the at least one mapping function as the at least one influencing factor;

[0014] constructing an error function based on the at least one influencing factor;

[0015] Based on the error function, weight parameters of the quantized perception model are updated.

[0016] In the above solution, constructing an error function based on the at least one influencing factor includes:

[0017] determining, based on an execution order of at least one mapping function corresponding to the at least one influencing factor during image processing, an execution order of the at least one influencing factor when constructing the error function;

[0018] Based on the determined execution order, the error function is constructed.

[0019] In the above solution, determining at least one image processing function of the ISP unit for image processing includes:

[0020] determining a first scenario for acquiring a plurality of sample image data;

[0021] Determining a first scene type corresponding to the first scene from a correspondence between scenes and scene types;

[0022] determining, from a correspondence between scene types and image processing functions, at least one image processing function corresponding to the first scene type;

[0023] The at least one image processing function is used as the at least one image processing function that the ISP unit has when performing image processing.

[0024] In the above solution, updating the weight parameters of the quantized perception model based on the error function includes:

[0025] determining a gradient of the error function;

[0026] Based on the determined gradient and learning rate, the weight parameters of the quantized perception model are updated.

[0027] In the above solution, the method further includes:

[0028] Obtain target image data;

[0029] The target image data is input into the quantized perceptual model to obtain floating-point target image data output by the quantized perceptual model.

[0030] In the above solution, the method further includes:

[0031] Inputting the floating-point target image data into a fixed-point model to obtain fixed-point target image data output by the fixed-point model;

[0032] The ISP unit is used to perform image processing on the fixed-point target image data to obtain the target image data after image processing.

[0033] An embodiment of the present invention provides a training device for an image processing model, comprising:

[0034] An acquisition unit, configured to acquire a plurality of sample image data; the sample image data being RAW image data;

[0035] a processing unit configured to use the acquired plurality of sample image data as training data, perform quantization perception model training on the plurality of sample image data, and obtain a quantization perception model; the quantization perception model is configured to quantize target image data to obtain floating-point target image data; and convert the floating-point target image data into fixed-point target image data, so that the fixed-point target image data meets the processing requirements of the ISP unit;

[0036] Wherein, when the quantitative perception model training is performed on the multiple sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit.

[0037] In the above solution, the processing unit is specifically used to:

[0038] Determining at least one image processing function of the ISP unit for image processing;

[0039] determining at least one mapping function corresponding to the at least one image processing function;

[0040] Using the at least one mapping function as the at least one influencing factor;

[0041] constructing an error function based on the at least one influencing factor;

[0042] Based on the error function, weight parameters of the quantized perception model are updated.

[0043] In the above solution, the processing unit is specifically used to:

[0044] determining, based on an execution order of at least one mapping function corresponding to the at least one influencing factor during image processing, an execution order of the at least one influencing factor when constructing the error function;

[0045] Based on the determined execution order, the error function is constructed.

[0046] In the above solution, the processing unit is specifically used to:

[0047] determining a first scenario for acquiring a plurality of sample image data;

[0048] Determining a first scene type corresponding to the first scene from a correspondence between scenes and scene types;

[0049] determining, from a correspondence between scene types and image processing functions, at least one image processing function corresponding to the first scene type;

[0050] The at least one image processing function is used as the at least one image processing function that the ISP unit has when performing image processing.

[0051] In the above solution, the processing unit is specifically used to:

[0052] determining a gradient of the error function;

[0053] Based on the determined gradient and learning rate, the weight parameters of the quantized perception model are updated.

[0054] In the above solution, the processing unit is further used to:

[0055] Obtain target image data;

[0056] The target image data is input into the quantized perceptual model to obtain floating-point target image data output by the quantized perceptual model.

[0057] In the above solution, the processing unit is further used to:

[0058] Inputting the floating-point target image data into a fixed-point model to obtain fixed-point target image data output by the fixed-point model;

[0059] The ISP unit is used to perform image processing on the fixed-point target image data to obtain the target image data after image processing.

[0060] An embodiment of the present invention provides an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor.

[0061] Wherein, when the processor is used to run the computer program, the steps of any of the above methods are implemented when executing the program.

[0062] An embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0063] The training method, device, electronic device, and storage medium of an image processing model provided by the embodiments of the present invention obtain multiple sample image data; the sample image data is RAW image data; the obtained multiple sample image data are used as training data, and a quantization perception model is trained on the multiple sample image data to obtain a quantization perception model; the quantization perception model is used to quantize target image data to obtain floating-point target image data; and the floating-point target image data is converted into fixed-point target image data so that the fixed-point target image data meets the processing requirements of the ISP unit; wherein, when the quantization perception model is trained on the multiple sample image data, the weight parameters of the quantization perception model are updated based on at least one influencing factor related to the ISP unit. According to the technical solution of the embodiments of the present invention, since the influencing factor for updating the weight parameters of the quantization perception model is related to the ISP unit, that is, when training the quantization perception model, the influence of the relevant factors in the ISP unit on the image quality is already considered, thereby improving the image processing performance of the quantization perception model trained based on the multiple sample image data. When the ISP unit is subsequently used to process the image data output by the quantization perception model, the quality of the image data processed by the ISP can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of the image processing process of the related technology;

[0065] Figure 2 It is a schematic diagram of the Quantization Aware Training (QAT) model in the related art;

[0066] Figure 3 This is a schematic diagram of the ISP processing flow in related technologies;

[0067] Figure 4 Schematic diagram of the implementation flow of the training method of the image processing model according to an embodiment of the present invention;

[0068] Figure 5 Schematic diagram of gamma mapping curves for different gamma values ​​according to an embodiment of the present invention;

[0069] Figure 6Schematic diagram of the slope of a gamma mapping curve with an 8-bit bit width and a gamma value equal to 1 / 2 according to an embodiment of the present invention;

[0070] Figure 7 This is a schematic diagram of the implementation process of using RAW image data to train a quantitative perception model according to an embodiment of the present invention;

[0071] Figure 8 A schematic diagram of an implementation flow of constructing an error function based on at least one influencing factor related to an ISP unit according to an embodiment of the present invention;

[0072] Figure 9 Schematic diagram of the structure of a training device for an image processing model according to an embodiment of the present invention;

[0073] Figure 10 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] Before describing the technical solutions of the embodiments of the present invention in detail, the relevant technologies are first introduced.

[0075] Figure 1 It is a schematic diagram of the image processing process in related technologies, such as Figure 1 As shown, the image processing model trained using QAT can be used to map the input raw image data (represented by RAW images) to obtain floating-point image data. Then, when deployed on the hardware chip, in order to improve computing efficiency, the floating-point QAT model can be converted into a fixed-point model based on deep learning (DL) through tools. The fixed-point model is used to convert the floating-point image data into fixed-point image data. Finally, the fixed-point image data output by the fixed-point model is processed by ISP to generate image data that can actually be displayed, such as RGB images. Figure 2 is a schematic diagram of the QAT model in related technology, such as Figure 2 As shown, the QAT model can be obtained by using quantization-aware training. The quantization-aware training can refer to introducing fixed-point forward reasoning during training, while the reverse operation still uses floating-point operations. Figure 3 This is a schematic diagram of the ISP processing flow in related technologies, such as Figure 3 As shown in FIG, the ISP processing flow may include: lens shading correction, bad pixel correction, noise reduction, automatic white balance correction, color correction, gamma correction, etc.

[0076] In summary, since the quantization perception model trained based on RAW image data and the corresponding RAW image processed based on ISP are independent steps, and the mapping functions used in the ISP processing flow, such as the gain function and the mapping function, have an amplifying effect on the quantization effect of the final image, if the impact of the subsequent ISP on the image quality is not considered in the process of training the image processing model based on the RAW image, the quality of the RGB image obtained after the RAW image output by the image processing model is processed using the ISP processing flow will be poor, that is, the accuracy of the image data output by the image processing model and the fixed-point model is good, but the accuracy of the image data obtained after ISP processing is poor.

[0077] Based on this, in various embodiments of the present invention, multiple sample image data are acquired; the sample image data are RAW image data; the acquired multiple sample image data are used as training data, and a quantitative perception model is trained on the multiple sample image data to obtain a quantitative perception model; the quantitative perception model is used to quantize the target image data to obtain floating-point target image data; and the floating-point target image data is converted into fixed-point target image data so that the fixed-point target image data meets the processing requirements of the ISP unit; wherein, when the quantitative perception model is trained on the multiple sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit.

[0078] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] The embodiment of the present invention provides a method for training an image processing model. Figure 4 Schematic diagram of the implementation flow of the training method of the image processing model according to the embodiment of the present invention; Figure 4 As shown, the method includes:

[0080] Step 401: Acquire multiple sample image data;

[0081] Step 402: Using the obtained plurality of sample image data as training data, a quantization perception model is trained on the plurality of sample image data to obtain a quantization perception model; the quantization perception model is used to quantize target image data to obtain floating-point target image data; and the floating-point target image data is converted into fixed-point target image data, so that the fixed-point target image data meets the processing requirements of the ISP unit;

[0082] Step 403: When performing quantitative perception model training on the multiple sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit.

[0083] Here, in step 401, the sample image data is RAW image data; the RAW image data is the original data obtained after the image sensor converts the captured light source signal into a digital signal; the multiple sample image data can be acquired through an image acquisition unit; the image acquisition unit can refer to a sensor unit in an electronic device that can capture images, such as a charge coupled device (CCD); the image can be a two-dimensional image, a three-dimensional image, etc.

[0084] Here, in step 402, in actual application, the format of the initial image data may be floating-point type. In order to increase the speed of subsequent image processing on the hardware chip, the floating-point image data may be converted into fixed-point image data. Therefore, a floating-point quantization perception model may be trained based on a plurality of sample image data, and the collected image data may be converted into floating-point image data using the quantization perception model. Subsequently, the floating-point image data may be converted into fixed-point image data using the fixed-point model.

[0085] Here, in step 403, in order to improve the accuracy of image data processing by the ISP unit, factors related to improving the accuracy of image data processing in the ISP unit can be used as influencing factors to reconstruct the error function of the quantization perception model, and when training the quantization perception model, the weight parameters of each layer of the quantization perception model are updated using the influencing factors related to the ISP unit. The ISP unit is used to convert the RAW image into a three-primary color RGB image.

[0086] In actual applications, the image processing functions possessed by the ISP units of different electronic devices may be different. For example, the image processing functions possessed by the ISP unit of electronic device 1 may include: lens shading correction, bad pixel correction, and noise reduction; the image processing functions possessed by the ISP unit of electronic device 2 may include: automatic white balance correction, color correction, and gamma correction; wherein, each image processing function can be implemented by a corresponding mapping function. Therefore, the influencing factors of constructing the error function can be determined in combination with the mapping functions corresponding to the image processing functions possessed by the ISP units.

[0087] Based on this, in one embodiment, updating the weight parameters of the quantitative perception model based on at least one influencing factor related to the ISP unit includes:

[0088] Determining at least one image processing function of the ISP unit for image processing;

[0089] determining at least one mapping function corresponding to the at least one image processing function;

[0090] Using the at least one mapping function as the at least one influencing factor;

[0091] constructing an error function based on the at least one influencing factor;

[0092] Based on the error function, weight parameters of the quantized perception model are updated.

[0093] Here, the mapping function can be a gamma mapping function, a tone mapping function, an S-type mapping function, etc.; the mapping function can also include a gain function; wherein the gain function can be other gain functions such as white balance gain, analog gain, digital gain, including but not limited to linear gain, and can also be nonlinear gain.

[0094] For example, the at least one image processing function of the ISP unit for image processing may include: lens shading correction, bad pixel correction, noise reduction, automatic white balance correction, color correction, gamma correction, etc., and may also include demosaicing.

[0095] Lens shading correction is used to eliminate the effects of vignetting in RAW image data. Vignetting is a phenomenon in which the center of an image is brighter, the edges darker, and the brightness decreases with distance from the image center. A defective pixel is a pixel with erroneous information in the image due to manufacturing defects in the pixel array on the image sensor or errors in the conversion of light signals into electrical signals. White balance describes the accuracy of the white color generated by mixing the three primary colors of red (R), green (G), and blue (B) in a display. White balance correction is used to restore the normal color of an image. The basic principle is to restore white objects to white in any environment. White balance correction can be achieved through white balance gain. De-mosaicing is used to restore the true color of RAW image data to match the display device's display. Color correction is used to correct the color of the image and can be achieved through a hue function. Gamma correction adjusts the grayscale of the image to make the processed image more pleasing to the human eye. Gamma correction is achieved through a gamma mapping function.

[0096] In actual application, the ISP unit can determine the execution order of at least one influencing factor when constructing the error function based on the degree of influence of at least one influencing factor on the image quality, wherein the greater the influence of the influencing factor on the image quality, the higher the execution order of the influencing factor when constructing the error function.

[0097] Based on this, in one embodiment, constructing an error function based on the at least one influencing factor includes:

[0098] determining, based on an execution order of at least one mapping function corresponding to the at least one influencing factor during image processing, an execution order of the at least one influencing factor when constructing the error function;

[0099] Based on the determined execution order, the error function is constructed.

[0100] For example, assume that the image processing functions of an electronic device's ISP include automatic white balance processing and gamma mapping functions; the mapping function corresponding to the automatic white balance processing is a white balance gain function, and the mapping function corresponding to the gamma mapping function is a gamma mapping function. If the ISP unit performs white balance gain processing before gamma mapping during image processing, the error function can be constructed using the white balance gain function first and then the gamma mapping function.

[0101] Taking the error function as the variance function as an example, formula (1) represents the initial error function and the updated model is:

[0102] Loss = (yf(x)) 2 (1)

[0103] Among them, Loss represents the initial error function, f(x) represents the image data output by the quantized perception model, and y is the real image data input by the quantized perception model.

[0104] It should be noted that, in practical applications, the error function includes but is not limited to a loss function in the form of variance.

[0105] Here, the error function constructed by first using the white balance gain function and then using the gamma mapping function is expressed by formula (2).

[0106] Loss'={(gamma i (gain i ×y)-gamma i (gain i ×f(x)))} 2 (2)

[0107] Among them, Loss'' represents the error function after construction, gain i ×f(x) represents the white balance processing of the image data output by the quantized perception model, gamma i (gain i ×f(x)) represents gamma mapping of the image data after white balance processing, gain i ×y represents the white balance processing of the real image data input by the quantized perception model, gamma i (gain i×y) represents the gamma mapping process performed on the real image data after white balance processing.

[0108] Here, the mapping function corresponding to the white balance processing is a white balance gain function, as shown in formula (3).

[0109] f(x)=x×gain x (3)

[0110] Wherein, x represents the color value of the x channel corresponding to a pixel in the fixed-point image data output by the fixed-point model, and x is the R channel, G channel or B channel. x Represents the gain value corresponding to the x channel in the fixed-point image data output by the fixed-point model.

[0111] It should be noted that in the white balance gain function, the gain value of the G channel is G =1, the gain value of R channel is gain R It can be the ratio between the color mean of the G channel and the color mean of the R channel, and the gain value of the B channel. B It can be the ratio between the color mean of the G channel and the color mean of the B channel.

[0112] Here, the mapping function corresponding to the gamma mapping function is a gamma mapping function, as shown in formula (4).

[0113]

[0114] Where i is a positive integer representing the input grayscale value of the i-th pixel in the RAW image; gamma(i) represents the output grayscale value of the i-th pixel in the RAW image; and γ represents the set gamma value. γ can be 1 / 10, 1 / 5, 1 / 2, 1, 2, 5, or 10.

[0115] It should be noted that the data bit width in formula (4) is 8 bits, and the data range is [0, 255]. In other embodiments, the data bit width in the gamma mapping function may also be other values, such as 16 bits.

[0116] Figure 5 Gamma mapping curves for different gamma values, such as Figure 5 As shown, the numerical slopes at different positions are different, which is manifested as different degrees of numerical amplification and reduction. Figure 6 The slope of the gamma mapping curve with an 8-bit width and a gamma value of 1 / 2 is as follows: Figure 6As shown in the figure, when the pixel value is low, the slope of the gamma mapping curve is large. Assuming the slope is greater than 2, it means that when the input pixel difference is 1, the pixel difference after gamma mapping is greater than 2. This manifests in the image as a larger difference between adjacent values, grayscale phenomenon, and excessive image unevenness. When the pixel value is large, the slope of the gamma mapping curve is small. Assuming the slope is less than 1 / 2, it means that when the pixel difference is greater than 2, the pixel difference after gamma mapping is 1, and they can share a common grayscale, resulting in a more uniform image.

[0117] In actual applications, the image processing functions performed by the ISP unit may differ for image data collected in different scenarios. For example, for image data collected in indoor scenarios, the image processing functions provided by the ISP unit may include automatic white balance and demosaicing; for image data collected in outdoor scenarios, the image processing functions provided by the ISP unit may include lens shading correction, bad pixel correction, and noise reduction. Therefore, based on the scenario in which the image acquisition unit collects multiple sample image data, at least one image processing function performed by the ISP unit can be determined.

[0118] Based on this, in one embodiment, determining at least one image processing function of the ISP unit for image processing includes:

[0119] determining a first scenario for acquiring a plurality of sample image data;

[0120] Determining a first scene type corresponding to the first scene from a correspondence between scenes and scene types;

[0121] determining, from a correspondence between scene types and image processing functions, at least one image processing function corresponding to the first scene type;

[0122] The at least one image processing function is used as the at least one image processing function that the ISP unit has when performing image processing.

[0123] In practical applications, the quantized perceptual model can be a feedback neural network model. That is, during backpropagation, the gradient of the error function is used to update the weight parameters of each layer of the quantized perceptual model. For example, assuming that the quantized perceptual model includes an input layer, a first intermediate layer, a second intermediate layer, a third intermediate layer, and an output layer, backpropagation means updating the weight parameters of each layer in the order of output layer, third intermediate layer, second intermediate layer, and first intermediate layer.

[0124] Based on this, in one embodiment, updating the weight parameters of the quantized perception model based on the error function includes:

[0125] determining a gradient of the error function;

[0126] Based on the determined gradient and learning rate, the weight parameters of the quantized perception model are updated.

[0127] It should be noted that in actual application, in addition to the error function and learning rate constructed based on at least one influencing factor related to ISP, the weight parameters of the quantitative perception model can also be updated based on the image data output by the quantitative perception model; wherein, the image data output by the quantitative perception model can be obtained based on the input sample image data.

[0128] In actual application, after the quantized perception model is obtained through training based on the plurality of sample image data, the input target image data can be converted into floating-point image data using the quantized perception model.

[0129] Based on this, in one embodiment, the method further includes:

[0130] Obtain target image data;

[0131] The target image data is input into the quantized perceptual model to obtain floating-point target image data output by the quantized perceptual model.

[0132] It should be noted that, in actual application, in order to improve the training accuracy of the quantitative perception model, the obtained target image data can also be used as sample image data to re-train the quantitative perception model.

[0133] In actual application, after using the trained quantization perception model to quantize the input target image data to obtain floating-point image data, the floating-point image data can be converted into fixed-point image data, and the ISP unit can be used to perform image processing on the fixed-point image data to obtain image data that can be displayed in the display unit.

[0134] Based on this, in one embodiment, the method further includes:

[0135] Inputting the floating-point target image data into a fixed-point model to obtain fixed-point target image data output by the fixed-point model;

[0136] The ISP unit is used to perform image processing on the fixed-point target image data to obtain the target image data after image processing.

[0137] In one example, if Figure 7 As shown in Figure 2, the process of training a quantitative perceptual model using RAW image data is described, including:

[0138] Step 701: Acquire multiple RAW image data;

[0139] Step 702: using the acquired multiple RAW image data as training data, and performing quantitative perception model training on the at least one RAW image data;

[0140] Step 703: When training the quantized perception model, updating weight parameters of the quantized perception model based on an error function constructed using at least one influencing factor related to the ISP unit;

[0141] Step 704: Determine whether the quantized perceptual model has converged. If it is determined that the quantized perceptual model has converged, execute step 705; otherwise, execute step 702.

[0142] Step 705: End the training of the quantization-aware model.

[0143] Here, as Figure 8 As shown, a process of constructing an error function based on at least one influencing factor associated with an ISP unit is described, including:

[0144] Step 801: Determine at least one image processing function that the ISP unit has for image processing, and determine at least one mapping function corresponding to the at least one image processing function;

[0145] Step 802: determining an execution order of the at least one influencing factor when constructing an error function based on an execution order of the at least one mapping function during image processing;

[0146] Step 803: Construct the error function based on the determined execution order.

[0147] Here, when using RAW image data to train the quantized perception model, an error function is constructed based on at least one influencing factor related to the ISP unit to update the weight parameters of the quantized perception model, which has the following advantages:

[0148] (1) When training the quantitative perception model, the influence of relevant factors in the ISP unit on the image quality has been taken into account. Therefore, the image processing performance of the quantitative perception model trained based on the loss function can be improved. When the image data output by the quantitative perception model is subsequently processed by the ISP unit, the accuracy of the image data processed by the ISP can be improved.

[0149] (2) The original RAW domain quantization perception model training is combined with the ISP processing flow. When training the quantization perception model, the image processing modules in the ISP unit that have a great influence on the processing effect of the final image, such as the white balance gain processing module and the gamma mapping module, are taken into consideration, thereby improving the actual effect of RAW data and image data processing by the ISP unit.

[0150] (3) In addition to the gain function and the mapping function, in actual applications, other functions that have a greater impact on the image processing effect can be added to the loss function in combination with the actual ISP processing flow to improve the image quality processed by IPS.

[0151] By adopting the technical solution of the embodiment of the present invention, since the influencing factors for updating the weight parameters of the quantitative perception model are related to the ISP unit, that is, when training the quantitative perception model, the influence of relevant factors in the ISP unit on the image quality has been considered. Therefore, the image processing performance of the quantitative perception model trained based on multiple sample image data can be improved. When the ISP unit is used to process the image data output by the quantitative perception model subsequently, the quality of the image data after ISP processing can be improved.

[0152] In order to implement the training method of the image processing model in the embodiment of the present invention, the embodiment of the present invention also provides a training device for the image processing model, which is arranged on a terminal. Figure 9 FIG. 1 is a schematic diagram of the structure of a training device for an image processing model according to an embodiment of the present invention; FIG. Figure 9 As shown, the device includes:

[0153] An acquisition unit 91 is configured to acquire a plurality of sample image data; the sample image data is RAW image data;

[0154] The processing unit 92 is configured to use the obtained plurality of sample image data as training data, perform quantization perception model training on the plurality of sample image data, and obtain a quantization perception model; the quantization perception model is used to quantize the target image data to obtain floating-point target image data; and convert the floating-point target image data into fixed-point target image data, so that the fixed-point target image data meets the processing requirements of the ISP unit;

[0155] Wherein, when the quantitative perception model training is performed on the multiple sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit.

[0156] In one embodiment, the processing unit 92 is specifically configured to:

[0157] Determining at least one image processing function of the ISP unit for image processing;

[0158] determining at least one mapping function corresponding to the at least one image processing function;

[0159] Using the at least one mapping function as the at least one influencing factor;

[0160] constructing an error function based on the at least one influencing factor;

[0161] Based on the error function, weight parameters of the quantized perception model are updated.

[0162] In one embodiment, the processing unit 92 is specifically configured to:

[0163] determining, based on an execution order of at least one mapping function corresponding to the at least one influencing factor during image processing, an execution order of the at least one influencing factor when constructing the error function;

[0164] Based on the determined execution order, the error function is constructed.

[0165] In one embodiment, the processing unit 92 is specifically configured to:

[0166] determining a first scenario for acquiring a plurality of sample image data;

[0167] Determining a first scene type corresponding to the first scene from a correspondence between scenes and scene types;

[0168] determining, from a correspondence between scene types and image processing functions, at least one image processing function corresponding to the first scene type;

[0169] The at least one image processing function is used as the at least one image processing function that the ISP unit has when performing image processing.

[0170] In one embodiment, the processing unit 92 is specifically configured to:

[0171] determining a gradient of the error function;

[0172] Based on the determined gradient and learning rate, the weight parameters of the quantized perception model are updated.

[0173] In one embodiment, the processing unit 92 is further configured to:

[0174] Obtain target image data;

[0175] The target image data is input into the quantized perceptual model to obtain floating-point target image data output by the quantized perceptual model.

[0176] In one embodiment, the processing unit 92 is further configured to:

[0177] Inputting the floating-point target image data into a fixed-point model to obtain fixed-point target image data output by the fixed-point model;

[0178] The ISP unit is used to perform image processing on the fixed-point target image data to obtain the target image data after image processing.

[0179] In actual application, the acquisition unit 91 can be implemented by a communication interface in the device; the processing unit 92 can be implemented by a processor in the device; the processor can be a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a field-programmable gate array (FPGA).

[0180] It should be noted that the apparatus provided in the above embodiment, when training an image processing model, only uses the division of the above-mentioned program modules as an example. In actual application, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the terminal can be divided into different program modules to complete all or part of the above-described processing. In addition, the apparatus provided in the above embodiment and the training method embodiment of the image processing model are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0181] Based on the hardware implementation of the above device, an embodiment of the present invention further provides an electronic device, Figure 10 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Figure 10 As shown, the electronic device 100 includes a memory 103, a processor 102, and a computer program stored in the memory 103 and executable on the processor 102; when the processor 102 executes the program, the method provided by one or more of the above technical solutions is implemented.

[0182] Specifically, the processor executes the following computer program: acquiring a plurality of sample image data; the sample image data being RAW image data; using the acquired plurality of sample image data as training data, performing quantization perception model training on the plurality of sample image data to obtain a quantization perception model; using the quantization perception model to quantize target image data to obtain floating-point target image data; and converting the floating-point target image data into fixed-point target image data, so that the fixed-point target image data meets the processing requirements of the ISP unit;

[0183] Wherein, when the quantitative perception model training is performed on the multiple sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit.

[0184] In one embodiment, the processor 102 specifically executes the following computer program:

[0185] Determining at least one image processing function of the ISP unit for image processing;

[0186] determining at least one mapping function corresponding to the at least one image processing function;

[0187] Using the at least one mapping function as the at least one influencing factor;

[0188] constructing an error function based on the at least one influencing factor;

[0189] Based on the error function, weight parameters of the quantized perception model are updated.

[0190] In one embodiment, the processor 102 specifically executes the following computer program:

[0191] determining, based on an execution order of at least one mapping function corresponding to the at least one influencing factor during image processing, an execution order of the at least one influencing factor when constructing the error function;

[0192] Based on the determined execution order, the error function is constructed.

[0193] In one embodiment, the processor 102 specifically executes the following computer program:

[0194] determining a first scenario for acquiring a plurality of sample image data;

[0195] Determining a first scene type corresponding to the first scene from a correspondence between scenes and scene types;

[0196] determining, from a correspondence between scene types and image processing functions, at least one image processing function corresponding to the first scene type;

[0197] The at least one image processing function is used as the at least one image processing function that the ISP unit has when performing image processing.

[0198] In one embodiment, the processor 102 specifically executes the following computer program:

[0199] determining a gradient of the error function;

[0200] Based on the determined gradient and learning rate, the weight parameters of the quantized perception model are updated.

[0201] In one embodiment, the processor 102 is further configured to execute the following computer program:

[0202] Obtain target image data;

[0203] The target image data is input into the quantized perceptual model to obtain floating-point target image data output by the quantized perceptual model.

[0204] In one embodiment, the processor 102 is further configured to execute the following computer program:

[0205] Inputting the floating-point target image data into a fixed-point model to obtain fixed-point target image data output by the fixed-point model;

[0206] The ISP unit is used to perform image processing on the fixed-point target image data to obtain the target image data after image processing.

[0207] It should be noted that the specific steps implemented when the processor 102 executes the program have been described in detail above and will not be repeated here.

[0208] As will be appreciated, electronic device 100 also includes a communication interface 101, which is used to exchange information with other devices. Furthermore, the various components within electronic device 100 are coupled together via a bus system 104. As will be appreciated, bus system 104 is configured to enable connectivity and communication between these components. In addition to a data bus, bus system 104 also includes a power bus, a control bus, and a status signal bus.

[0209] It is understood that the memory 103 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0210] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 102. Processor 102 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 102 or by software instructions. The above processor 102 may be a general-purpose processor, a DSP, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. Processor 102 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium located in a memory. Processor 102 reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0211] The embodiments of the present invention further provide a storage medium, specifically a computer storage medium, more specifically a computer-readable storage medium, on which computer instructions, i.e., a computer program, are stored. When the computer instructions are executed by a processor, the method provided by one or more of the above technical solutions is implemented.

[0212] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and intelligent devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0213] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0215] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, ROM, RAM, disks or optical disks, etc. Various media that can store program codes.

[0216] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0217] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0218] In addition, the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0219] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A training method for an image processing model, characterized in that: The method comprises: Acquire a plurality of sample image data; the sample image data is RAW image data; Using the obtained plurality of sample image data as training data, performing quantization perception model training on the plurality of sample image data to obtain a quantization perception model; the quantization perception model is used to quantize the target image data to obtain floating-point target image data; and Convert the floating-point target image data into fixed-point target image data, so that the fixed-point target image data meets the processing requirements of the image signal processing ISP unit, When the quantitative perception model training is performed on the plurality of sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit; The updating of the weight parameters of the quantitative perception model based on at least one influencing factor related to the ISP unit includes: Determining at least one image processing function of the ISP unit for image processing; determining at least one mapping function corresponding to the at least one image processing function; Using the at least one mapping function as the at least one influencing factor; constructing an error function based on the at least one influencing factor; Based on the error function, weight parameters of the quantized perception model are updated.

2. The method according to claim 1, characterized in that The constructing an error function based on the at least one influencing factor includes: determining, based on an execution order of at least one mapping function corresponding to the at least one influencing factor during image processing, an execution order of the at least one influencing factor when constructing the error function; Based on the determined execution order, the error function is constructed.

3. The method according to claim 1, characterized in that The determining of at least one image processing function of the ISP unit for image processing includes: determining a first scenario for acquiring a plurality of sample image data; Determining a first scene type corresponding to the first scene from a correspondence between scenes and scene types; determining, from a correspondence between scene types and image processing functions, at least one image processing function corresponding to the first scene type; The at least one image processing function is used as the at least one image processing function that the ISP unit has when performing image processing.

4. The method according to any one of claims 1 to 3, characterized in that The updating of the weight parameters of the quantized perception model based on the error function includes: determining a gradient of the error function; Based on the determined gradient and learning rate, the weight parameters of the quantized perception model are updated.

5. The method according to claim 1, wherein The method further comprises: Obtain target image data; The target image data is input into the quantized perceptual model to obtain floating-point target image data output by the quantized perceptual model.

6. The method according to claim 5, characterized in that The method further comprises: Inputting the floating-point target image data into a fixed-point model to obtain fixed-point target image data output by the fixed-point model; The ISP unit is used to perform image processing on the fixed-point target image data to obtain the target image data after image processing.

7. A training device for an image processing model, characterized in that: include: an acquisition unit, configured to acquire a plurality of sample image data; The sample image data is RAW image data; a processing unit, configured to use the obtained plurality of sample image data as training data, perform quantitative perception model training on the plurality of sample image data, and obtain a quantitative perception model; The quantization perception model is used to quantize the target image data to obtain floating-point target image data; and convert the floating-point target image data into fixed-point target image data so that the fixed-point target image data meets the processing requirements of the ISP unit; When the quantitative perception model training is performed on the plurality of sample image data, the weight parameters of the quantitative perception model are updated based on at least one influencing factor related to the ISP unit; The updating of the weight parameters of the quantitative perception model based on at least one influencing factor related to the ISP unit includes: Determining at least one image processing function of the ISP unit for image processing; determining at least one mapping function corresponding to the at least one image processing function; Using the at least one mapping function as the at least one influencing factor; constructing an error function based on the at least one influencing factor; Based on the error function, weight parameters of the quantized perception model are updated.

8. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images