Image processing model training method, image processing method and electronic device

Abstract

The present application discloses an image processing model training method, an image processing method and an electronic device. The image processing model training method comprises: inputting at least one first image into the image processing model, obtaining a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is used to perform set image processing on the RAW image; the first image and the second image are both RAW images; using a loss function configured with at least one set influence factor, calculating the loss value corresponding to each first image in the at least one first image; the set influence factor represents the influence of the image signal processing ISP on the quality of the image; the loss value is calculated based on the corresponding second image and the corresponding third image; according to the loss value corresponding to each first image in the at least one first image, updating the weight parameter of the image processing model.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing model training method, an image processing method and an electronic device. Background Art

[0002] The related art provides an image processing method: using an image processing model to process a RAW image, and performing image signal processing (ISP) on the processed RAW image to obtain a three-primary color RGB image. In the above image processing process, there may be a problem of image quality degradation of the RGB image. Among them, the RAW image data is the original data after the image sensor converts the captured light source signal into a digital signal. Summary of the invention

[0003] In view of this, the embodiments of the present application hope to provide an image processing model training method, an image processing method, an apparatus and an electronic device to solve the technical problem that the image quality of RGB images in the related art may deteriorate.

[0004] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0005] The present application provides an image processing model training method, including:

[0006] Inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is used to perform set image processing on a RAW image; the first image and the second image are both RAW images;

[0007] A loss function configured with at least one set influencing factor is used to calculate a loss value corresponding to each first image in the at least one first image; wherein the set influencing factor represents the influence of an image signal processing ISP on the quality of the image; the ISP is used to convert a RAW image into a three-primary color RGB image; the loss value is calculated based on a corresponding second image and a corresponding third image; the third image is a calibration image of the corresponding first image;

[0008] Update the weight parameters of the image processing model according to the loss value corresponding to each first image in the at least one first image.

[0009] In the above solution, the set influencing factor includes at least one of the following:

[0010] At least one set gain function; each gain function in the at least one set gain function corresponds to a type of image gain behavior in the ISP;

[0011] At least one set mapping function; each mapping function in the at least one set mapping function corresponds to a type of image mapping behavior in the ISP.

[0012] In the above solution, the loss function is configured with at least two set influencing factors; the method further includes:

[0013] Determining an execution order of each of the at least two set influencing factors when calculating the loss value;

[0014] Based on the determined execution order, a loss function configured with the at least two set influencing factors is generated.

[0015] In the above solution, the step of determining the execution order of each of the at least two set influencing factors when calculating the loss value includes:

[0016] Based on the execution order of each of the at least two set influencing factors in the ISP, the execution order of each of the at least two set influencing factors when calculating the loss value is determined.

[0017] In the above solution, when calculating the loss value corresponding to each first image in the at least one first image, the method includes:

[0018] Processing the corresponding second image using the loss function to obtain a fourth image;

[0019] Processing the corresponding third image using the loss function to obtain a fifth image;

[0020] Based on the fourth image and the fifth image, a loss value corresponding to the first image is calculated.

[0021] In the above solution, the calculating the loss value corresponding to the first image based on the fourth image and the fifth image includes:

[0022] Calculating a grayscale difference value corresponding to each pixel in the first image based on a first grayscale value of each pixel in the fourth image and a second grayscale value of each pixel in the fifth image;

[0023] Based on the grayscale difference corresponding to each pixel in the first image, a loss value of the grayscale value corresponding to the first image is calculated.

[0024] The present application also provides an image processing method, including:

[0025] The first RAW image is processed by using the first model to obtain a second RAW image; wherein,

[0026] The first model is an image processing model trained using any of the above-mentioned image processing model training methods.

[0027] The present application also provides an image processing method, including:

[0028] Processing the first RAW image using the first model to obtain a second RAW image;

[0029] Performing image signal processing on the second RAW image to obtain a three-primary color RGB image corresponding to the second RAW image; wherein:

[0030] The first model is an image processing model trained using any of the above-mentioned image processing model training methods.

[0031] The present application also provides an electronic device, including:

[0032] An image processing unit, used for inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is used for performing image processing on a RAW image; the first image and the second image are both RAW images;

[0033] A calculation unit, configured to calculate a loss value corresponding to each first image in the at least one first image by using a loss function configured with at least one set influencing factor; wherein the set influencing factor represents the influence of an image signal processing ISP on the quality of the image; the ISP is used to convert a RAW image into a three-primary color RGB image; the loss value is calculated based on a corresponding second image and a corresponding third image; the third image is a calibration image of the corresponding first image;

[0034] An updating unit is used to update the weight parameters of the image processing model according to the loss value.

[0035] The present application also provides an electronic device, including:

[0036] A processing unit is used to process the first RAW image using a first model to obtain a second RAW image; wherein the first model is an image processing model trained using the image processing model training method in any of the above embodiments.

[0037] The present application also provides an electronic device, including:

[0038] A first image processing unit, configured to process the first RAW image using a first model to obtain a second RAW image;

[0039] The second image processing unit is used to perform image signal processing on the second RAW image to obtain a three-primary color RGB image corresponding to the second RAW image; wherein the first model is an image processing model trained by the image processing model training method in any of the above embodiments.

[0040] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0041] Wherein, when the processor is used to run the computer program, any of the following:

[0042] Execute the steps of any of the above-mentioned image processing model training methods;

[0043] Execute the steps of any of the above image processing methods.

[0044] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, any one of the following is achieved:

[0045] The steps of any of the above image processing model training methods;

[0046] The steps of any of the above image processing methods.

[0047] In the solution provided by the embodiment of the present application, at least one first image is input into the image processing model to obtain a second image corresponding to each of the at least one first image output by the image processing model, a loss function configured with at least one set influencing factor is used to calculate the loss value corresponding to each of the at least one first image, and the weight parameters of the image processing model are updated according to the calculated loss value. Since at least one set influencing factor is configured in the loss function, and the set influencing factor represents the effect of the ISP on the quality of the image, the effect of the ISP on the quality of the image has been considered when training the image processing model. Therefore, the image processing performance of the image processing model trained based on the loss function can be improved, and the RAW image output by the image processing model is processed based on the ISP processing flow, and the quality of the obtained RGB image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of an image processing flow provided for related technologies;

[0049] Figure 2 A schematic diagram of an ISP processing flow provided for related technologies;

[0050] Figure 3 A schematic diagram of the implementation flow of the image processing model training method provided in the embodiment of the present application;

[0051] Figure 4 A schematic diagram of the implementation process of calculating the loss value in the image processing model training method provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a gamma curve provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of an implementation flow of calculating a loss value in an image processing model training method provided in another embodiment of the present application;

[0054] Figure 7 A schematic diagram of the implementation flow of the image processing method provided in the embodiment of the present application;

[0055] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application;

[0057] Fig.10 Schematic diagram of the hardware structure of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0058] Figure 1 FIG. 1 is a schematic diagram of an image processing flow provided by the related art. Figure 1 As shown, in the related art, a trained image processing model is usually used to process the RAW image collected by the shooting device to reduce the image noise in the RAW image, improve the resolution of the RAW image, etc., to obtain a processed RAW image, and the processed RAW image is processed according to the ISP processing flow to convert the RAW image into an RGB image that is easy to transmit, store and view.

[0059] Since training the image processing model based on RAW images and processing the corresponding RAW images based on the ISP processing flow are independent steps, the impact of the subsequent ISP on the image quality is not considered during the process of training the image processing model based on RAW images, resulting in poor quality of the RGB image obtained after processing the RAW image output by the image processing model using the ISP processing flow.

[0060] Please refer to Figure 2 , Figure 2 FIG. 1 is a schematic diagram of an ISP processing flow provided by the related art. Figure 2As shown, the ISP processing flow may include: lens shading correction, bad pixel correction, denoising, automatic white balance correction, color correction, gamma correction, etc. In actual applications, the ISP processing flow may be more complicated (i.e., it includes more functional items), and the execution order of each functional item in the ISP processing flow may also be different. Here, the functional items include the above-mentioned lens shading correction, bad pixel correction, denoising, automatic white balance correction, color correction, gamma correction, etc., and may also include demosaicing. Among them,

[0061] Lens shading correction is used to eliminate the effects of vignetting in RAW image data. Vignetting refers to the phenomenon that the center of an image is brighter, the edges are darker, and the brightness decreases as you move away from the center of the image.

[0062] A bad pixel refers to a pixel with erroneous information on the image due to a process defect in the corresponding array of the pixel on the image sensor, or due to an error in the process of converting light signals into electrical signals.

[0063] White balance is an indicator that describes the accuracy of white after the three primary colors of red (R), green (G), and blue (B) are mixed in the display. White balance correction is used to restore the normal color of the image. The basic principle is to restore white objects to white objects in any environment. White balance correction can be achieved through white balance gain.

[0064] De-mosaicing is used to restore the true colors of the display device from the RAW image data.

[0065] Color correction is used to correct the color of an image and can be achieved through a hue function.

[0066] Gamma correction is used to adjust the grayscale of an image so that the processed image looks more comfortable to the human eye. Gamma correction is achieved through the gamma mapping function.

[0067] In order to solve the above problems, an embodiment of the present application provides an image processing model training method, by configuring at least one set influence factor in the loss function, the set influence factor characterizes the influence of ISP on the quality of the image. In the process of training the image processing model, the corresponding loss value is calculated using at least one set influence factor, so that the weight parameters of the image processing model are updated based on the calculated loss value. Since the influence of ISP on the quality of the image has been considered when training the image processing model, the image processing performance of the image processing model trained based on the loss function can be improved. Compared with the RAW image output by the image processing model, the quality of the RGB image obtained by processing the RAW image output by the image processing model based on the ISP processing flow is improved.

[0068] The technical solution of the present application is further elaborated in detail below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0069] Figure 3 The following is a schematic diagram of the implementation process of the image processing model training method provided in the embodiment of the present application. In the embodiment of the present application, the execution subject of the image processing model training method can be an electronic device such as a terminal or a server.

[0070] Reference Figure 3 , the image processing model training method provided in the embodiment of the present application includes:

[0071] S101: Input at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is used to perform set image processing on a RAW image; the first image and the second image are both RAW images.

[0072] Here, the image processing model includes but is not limited to a model built based on a U-Net network model or a convolutional neural network (CNN) model.

[0073] Each first image in at least one first image corresponds to a corresponding third image. The third image is a calibration image of the corresponding first image. The image quality of the first image is worse than the image quality of the corresponding third image, and the first image and the corresponding third image are both obtained based on the same shooting scene. In other words, the difference between the first image and the corresponding third image lies in the image quality. Here, the evaluation index of image quality includes but is not limited to: resolution, image noise, color difference, etc.

[0074] In one embodiment, the electronic device can acquire at least one first image, input the acquired at least one first image into an image processing model, perform image processing on each of the at least one first image using the image processing model, and acquire a second image corresponding to each of the at least one first image output by the image processing model.

[0075] In one embodiment, the electronic device may acquire at least one batch of image samples, wherein each batch in the at least one batch includes at least one image sample pair, and each image sample pair includes a first image and a corresponding third image.

[0076] During each training, the electronic device inputs the first images of all image sample pairs in a batch into the image processing model in batches, performs image processing on the input first images using the image processing model, and obtains the second image corresponding to each first image in the batch output by the image processing model.

[0077] In this embodiment, an image processing model may be used to perform denoising, super-resolution processing, etc. on the input first image. Super-resolution processing refers to the process of restoring an image of a second resolution based on an image or image sequence of a first resolution, where the first resolution is smaller than the second resolution.

[0078] S102: Using a loss function configured with at least one set influencing factor, calculate the loss value corresponding to each first image in the at least one first image; wherein the set influencing factor represents the influence of the ISP on the quality of the image; the ISP is used to convert the RAW image into a three-primary color RGB image; the loss value is calculated based on the corresponding second image and the corresponding third image; the third image is a calibration image of the corresponding first image.

[0079] In a case where an image processing model outputs a second image corresponding to each first image in at least one first image, the electronic device calculates a loss value corresponding to each first image in at least one first image based on each second image output by the image processing model and on a third image corresponding to each corresponding first image, using a loss function configured with at least one set influencing factor.

[0080] It should be noted that the loss function is used to measure the inconsistency between the image output by the model and the corresponding calibration image. In practical applications, model training is achieved by minimizing the loss function.

[0081] In practical applications, factors that have a greater impact on image quality in the ISP processing flow can be preferentially selected and configured in the loss function.

[0082] S103: Update the weight parameters of the image processing model according to the loss value corresponding to each first image in the at least one first image.

[0083] The electronic device updates the weight parameters of the image processing model according to the loss value corresponding to each first image in at least one first image, so as to improve the quality of the image output by the image processing model. The electronic device back-propagates the loss value corresponding to each first image in the at least one first image in the image processing model, and in the process of back-propagating the loss value corresponding to each first image in the at least one first image to each layer of the image processing model, calculates the gradient of the loss function according to the loss value corresponding to each first image in the at least one first image, and updates the weight parameters back-propagated to the current layer along the descending direction of the gradient.

[0084] The electronic device uses the updated weight parameters as weight parameters used in the image processing model when training the image processing model next time.

[0085] Here, an update stop condition can be set. When the update stop condition is met, the weight parameter obtained by the last update is determined as the weight parameter used by the trained image processing model. The update stop condition is such as a set training epoch. A training epoch is a process of training the image processing model once based on at least one first image and a corresponding third image. Of course, the update stop condition is not limited to this. For example, the loss value corresponding to each first image in at least one first image can be less than or equal to a set loss threshold.

[0086] Back propagation is relative to forward propagation. Forward propagation refers to the feedforward processing of the model, while the direction of back propagation is opposite to that of forward propagation. Back propagation refers to updating the weight parameters of each layer of the model according to the results of the model output. For example, if the model includes the first convolution layer, the second convolution layer, and the third convolution layer, then forward propagation refers to processing in the order of the first convolution layer-the second convolution layer-the third convolution layer, and back propagation refers to updating the weight parameters of each layer in the order of the third convolution layer-the second convolution layer-the first convolution layer.

[0087] In the solution provided by this embodiment, at least one first image is input into the image processing model to obtain a second image corresponding to each of the at least one first image output by the image processing model, a loss function configured with at least one set influencing factor is used to calculate the loss value corresponding to each of the at least one first image, and the weight parameters of the image processing model are updated according to the calculated loss value. Since at least one set influencing factor is configured in the loss function, and the set influencing factor represents the influence of the ISP on the quality of the image, the influence of the ISP on the quality of the image has been considered when training the image processing model. Therefore, the image processing performance of the image processing model trained based on the loss function can be improved, and the RAW image output by the image processing model is processed based on the ISP processing flow, and the quality of the obtained RGB image is improved.

[0088] As another embodiment of the present application, the set influencing factor includes at least one of the following:

[0089] At least one set gain function; each gain function in the at least one set gain function corresponds to a type of image gain behavior in the ISP;

[0090] At least one set mapping function; each mapping function in the at least one set mapping function corresponds to a type of image mapping behavior in the ISP.

[0091] Here, at least one set gain function includes but is not limited to a white balance gain function and an automatic gain function. The white balance gain function corresponds to the automatic white balance correction in the ISP, and the automatic gain function corresponds to the automatic gain control in the ISP. At least one set mapping function includes but is not limited to a gamma mapping function and a tone mapping function. The gamma mapping function corresponds to the gamma correction in the ISP, and the tone mapping function corresponds to the color correction in the ISP.

[0092] The gain function is used to perform gain processing on the color value corresponding to each pixel in the RAW image. Each pixel in the RAW image corresponds to a color. The data in the RAW image is arranged in the format of RGRG / BGBG. The expression of the gain function can be: f(x) = x × gain x , where x represents the color value of the x channel corresponding to a pixel in the RAW image, and x is the R channel, G channel, or B channel. gain x Indicates the gain value corresponding to the x channel in the RAW image.

[0093] It should be noted that in some gain functions, the gain values ​​corresponding to R, G, and B are all set gain values. G is 1, the gain value of the R channel is gainR It can be the ratio of the color mean of the G channel to the color mean of the R channel, and the gain value of the B channel is gain. B It can be the ratio between the color mean of the G channel and the color mean of the B channel.

[0094] The gamma mapping function is used to perform nonlinear operations on the grayscale values ​​of the input image, so that the grayscale values ​​of the output image are exponentially related to the grayscale values ​​of the input image. In practical applications, when the data bit width corresponding to each pixel in the RAW image is 8 bits (bit) and the data range corresponding to each pixel is [0, 255], the expression of the gamma mapping function can be: Where i represents the input grayscale value of the i-th pixel in the RAW image, and i is a positive integer; gamma(i) represents the output grayscale value of the i-th pixel in the RAW image, and γ represents the set gamma value. γ can be 1 / 10, 1 / 5, 1 / 2, 1, 2, 5, or 10.

[0095] It should be noted that, in other embodiments, the data bit width may also be other values, such as 16 bits.

[0096] As another embodiment of the present application, the loss function is configured with at least two set influencing factors; the image processing model training method further includes:

[0097] Determining an execution order of each of the at least two set influencing factors when calculating the loss value;

[0098] Based on the determined execution order, a loss function configured with the at least two set influencing factors is generated.

[0099] Here, the at least two set impact factors configured in the loss function may be determined according to the degree of impact of each set impact factor on the quality of the RGB image in the ISP processing flow. For example, based on the degree of impact of all set impact factors on the quality of the RGB image, at least two set impact factors are determined from the multiple set impact factors in descending order.

[0100] In practical applications, the execution order of each of the at least two set influencing factors when calculating the loss value can be random. The electronic device can also determine the execution order of each of the at least two set influencing factors when calculating the loss value based on set conditions.

[0101] Here, the set condition may be the degree of influence of each of the at least two set influence factors on the quality of the RGB image. For example, based on the degree of influence of each of the at least two set influence factors on the quality of the RGB image, the execution order of each of the at least two set influence factors when calculating the loss value is determined in descending order. The higher the degree of influence of the set influence factor, the higher the execution order of the set influence factor when calculating the loss value.

[0102] The set condition may also be the execution order of the function items corresponding to each of the at least two set influencing factors when calculating the loss value. The function item corresponding to the white balance gain function is automatic white balance correction. The function item corresponding to the gamma mapping function is gamma correction. The function item corresponding to the hue function is color correction.

[0103] In practical applications, when at least two set influencing factors include a white balance gain function and a gamma mapping function, the expression of the generated loss function is as follows:

[0104] Loss = gamma(y)-gamma(f(x))

[0105] Wherein, y represents the gain image obtained after the third image corresponding to the first image is gain-processed by using the gain function. gamma(y) represents the mapping image obtained after the gain image corresponding to the third image is gamma-mapped by using the gamma mapping function. f(x) represents the gain image obtained after the second image corresponding to the first image is gain-processed by using the gain function. gamma(f(x)) represents the mapping image obtained after the gain image corresponding to the second image is gamma-mapped by using the gamma mapping function.

[0106] Alternatively, the resulting loss function can be expressed as:

[0107] Loss = gain(y i )-gain(gamma(i))

[0108] Among them, y i represents a mapped image obtained by gamma mapping the third image corresponding to the first image using a gamma mapping function. gain(y i ) represents the gain image obtained by performing gain processing on the mapping image corresponding to the third image using the gain function. gamma(i) represents the mapping image obtained by performing gamma mapping on the second image corresponding to the first image using the gamma mapping function. gain(gamma(i)) represents the gain image obtained by performing gain processing on the mapping image corresponding to the second image using the gain function.

[0109] It should be noted that, in some embodiments, when the at least two set influencing factors include a white balance gain function and a gamma mapping function, the expression of the generated loss function may also be:

[0110] Loss = {gamma(y)-gamma(f(x))} 2 , or Loss = {gain(y i )-gain(gamma(i))} 2

[0111] Here, the difference between the expression of the loss function in this embodiment and the expression of the loss function in the above embodiment is that the loss function in this embodiment is used to characterize the variance value calculated based on the third image and the first image as the loss value, and the loss function in the above embodiment is used to characterize the difference value calculated based on the third image and the first image as the loss value. The meanings of the parameters in the expression of the loss function in this embodiment are the same as those of the parameters in the expression of the loss function in the above embodiment.

[0112] As another embodiment of the present application, determining the execution order of each of the at least two set influencing factors when calculating the loss value includes:

[0113] Based on the execution order of each of the at least two set influencing factors in the ISP, the execution order of each of the at least two set influencing factors when calculating the loss value is determined.

[0114] Here, it is assumed that the ISP processing flow is: lens shading correction, bad pixel correction, automatic white balance correction, denoising, demosaicing, and gamma correction. When both automatic white balance correction and gamma correction affect the quality of the RGB image, the at least two set influencing factors determined include: white balance gain function and gamma mapping function. At this time, based on the ISP processing flow, the execution order of each of the at least two set influencing factors when calculating the loss value is to first execute the influencing factor corresponding to the automatic white balance correction, and then execute the influencing factor corresponding to the gamma correction. At this time, the expression of the generated loss function can be:

[0115] Loss=gamma(y)-gamma(f(x)), or Loss={gamma(y)-gamma(f(x))} 2 .

[0116] In some embodiments, when automatic white balance correction, denoising and gamma correction all affect the quality of RGB images, the at least two set influencing factors determined include: white balance gain function, denoising function, gamma mapping function; at this time, based on the ISP processing flow, the execution order of each of the at least two set influencing factors when calculating the loss value is to first execute the influencing factor corresponding to the automatic white balance correction, then the influencing factor corresponding to the denoising, and finally the influencing factor corresponding to the gamma correction. The expression of the generated loss function can be:

[0117] Loss=gamma(Deniose(y))-gamma(Deniose(f(x))).

[0118] Wherein, Deniose(y) represents the denoised image obtained by denoising the gain image corresponding to the third image corresponding to the first image using a denoising function. gamma(Deniose(y)) represents the mapped image obtained by gamma mapping the denoised image corresponding to the third image corresponding to the first image using a gamma mapping function. Deniose(f(x)) represents the denoised image obtained by denoising the gain image corresponding to the second image corresponding to the first image using a denoising function. gamma(Deniose(f(x))) represents the mapped image obtained by gamma mapping the denoised image corresponding to the second image corresponding to the first image using a gamma mapping function.

[0119] Alternatively, in the execution order of each of the at least two set influencing factors when calculating the loss value, the influencing factor corresponding to the automatic white balance correction is executed first, then the influencing factor corresponding to the denoising point is executed, and finally the influencing factor corresponding to the gamma correction is executed. The expression of the generated loss function can also be:

[0120] Loss={gamma(Deniose(y))-gamma(Deniose(f(x)))} 2 .

[0121] As another embodiment of the present application, Figure 4 The following is a schematic diagram showing the implementation process of calculating the loss value in the image processing model training method provided in the embodiment of the present application. Figure 4 As shown, when calculating the loss value corresponding to each first image in the at least one first image, the method includes:

[0122] S201: Processing the corresponding second image using the loss function to obtain a fourth image.

[0123] S202: Processing the corresponding third image using the loss function to obtain a fifth image.

[0124] S203: Calculate a loss value corresponding to the first image based on the fourth image and the fifth image.

[0125] In actual applications, when at least one set influencing factor configured in the loss function includes a white balance mapping function and a gamma mapping function, and the loss function is Loss = gamma(y)-gamma(f(x)), the electronic device uses the gain function in the loss function to perform gain processing on the second image corresponding to each first image in at least one first image output by the image processing model to obtain a gain image corresponding to the second image corresponding to each first image in at least one first image, and uses the gamma mapping function in the loss function to perform gamma mapping on the gain image corresponding to the corresponding second image to obtain a fourth image corresponding to the corresponding second image.

[0126] The electronic device uses the gain function in the loss function to perform gain processing on the third image corresponding to each first image in at least one first image to obtain a gain image corresponding to the third image corresponding to each first image in at least one first image, and uses the gamma mapping function in the loss function to perform gamma mapping on the gain image corresponding to the corresponding third image to obtain a fifth image corresponding to the corresponding third image.

[0127] The electronic device calculates a loss value corresponding to each first image in at least one first image based on a fourth image corresponding to the second image corresponding to each first image in at least one first image and a fifth image corresponding to the third image corresponding to each first image in at least one first image, and calculates an average loss value corresponding to at least one first image based on the loss value corresponding to each first image in at least one first image.

[0128] In practical applications, the loss function is Loss = {gamma (y) - gamma (f (x))} 2 In this case, after calculating the loss value corresponding to each first image in at least one first image, the electronic device performs a variance operation based on the loss value corresponding to each first image in at least one first image to calculate the variance value of the loss value corresponding to at least one first image.

[0129] It should be noted that, in the process of performing gain processing on the third image corresponding to each first image in at least one first image using the gain function in the loss function, the gain value of the G channel in the third image is G 1, the gain value of the R channel in the third image is gain RIt can be the ratio of the color mean of all G channels in the third image to the color mean of all R channels in the third image, and the gain value of the B channel in the third image is gain. B It may be a ratio between a color mean of all G channels in the third image and a color mean of all B channels in the third image.

[0130] In the process of performing gain processing on the second image corresponding to each first image in at least one first image by using the gain function in the loss function, the gain of the G channel in the corresponding second image is gain' G =1. The corresponding gain of the R channel in the second image is gain' R Can be compared with the gain value of the R channel in the third image R The same can also be the ratio of the color mean of all G channels in the corresponding second image to the color mean of all R channels in the second image. The gain of the B channel in the corresponding second image is gain' B Can be compared with the gain value of the B channel in the third image B Similarly, it may also be the ratio between the color mean of all G channels in the corresponding second image and the color mean of all B channels in the second image.

[0131] It should be noted that, for the convenience of calculation, when the gamma mapping function in the loss function is used to map the corresponding image, the grayscale value of each pixel in the corresponding image can be normalized, so as to convert the grayscale value of each pixel in the corresponding image into a value between 0 and 1. Please refer to Figure 5 , Figure 5 A schematic diagram of a gamma curve provided in an embodiment of the present application is shown. Figure 5 In , X represents the input grayscale value of each pixel in the corresponding image, and y represents the output grayscale value of each pixel in the corresponding image. The input grayscale value refers to the grayscale value before the image is mapped using the gamma mapping curve, and the output grayscale value refers to the grayscale value obtained after the image is mapped using the gamma mapping curve.

[0132] As another embodiment of the present application, Figure 6 FIG. 2 is a schematic diagram showing a process flow of calculating a loss value in an image processing model training method provided in another embodiment of the present application. Figure 6 As shown, the calculating the loss value corresponding to the first image based on the fourth image and the fifth image includes:

[0133] S301: Calculate a grayscale difference value corresponding to each pixel in the first image based on a first grayscale value of each pixel in the fourth image and a second grayscale value of each pixel in the fifth image.

[0134] S302: Calculate a loss value of the grayscale value corresponding to the first image based on the grayscale difference corresponding to each pixel in the first image.

[0135] Here, when the loss function is Loss = gamma(y)-gamma(f(x)), the electronic device calculates the grayscale difference corresponding to each pixel in the first image based on the first grayscale value of each pixel in the fourth image corresponding to the corresponding first image and the second grayscale value of each pixel in the fifth image corresponding to the first image; based on the grayscale difference corresponding to each pixel in each of the first images in at least one first image, calculates the mean of the grayscale difference values ​​to obtain the loss value of the grayscale value corresponding to the first image.

[0136] The loss function is Loss = {gamma (y) - gamma (f (x))} 2 In this case, after calculating the grayscale difference value corresponding to each first image in at least one first image, the variance value of the grayscale difference value is calculated based on the grayscale difference value corresponding to each pixel in each first image in at least one first image to obtain the loss value of the grayscale value corresponding to the first image.

[0137] As another embodiment of the present application, after the image processing model is trained, the image processing model can be put into use. It should be noted that the electronic device in the embodiment corresponding to the training image processing model may be different from the electronic device in this embodiment that uses the trained image processing model to process the RAW image. For example, a first electronic device trains an image processing model, and a second electronic device transplants the trained image processing model into a second electronic device, and uses the trained image processing model to process the first RAW image to obtain a second RAW image. Here, the processing capability of the first electronic device is more powerful than that of the second electronic device. In actual applications, the first electronic device may be a server, and the second electronic device may be a terminal device such as a mobile phone, a tablet computer, or a wearable device.

[0138] The implementation process of the electronic device using the trained image processing model to process the RAW image is as follows:

[0139] The electronic device processes the first RAW image using the trained image processing model to obtain a second RAW image, wherein the resolution of the second RAW image is greater than the resolution of the first RAW image.

[0140] Here, the first RAW image may be captured by the electronic device, or may be captured by other devices and sent to the electronic device.

[0141] It should be noted that in the scenario of super-resolution reconstruction of the RAW image, the electronic device can use the image processing model trained by the above embodiment to perform super-resolution processing on the RAW image to obtain a second RAW image. At this time, the resolution of the second RAW image is greater than the resolution of the first RAW image.

[0142] In the scenario of denoising a RAW image, the electronic device can use the trained image processing model to perform denoising on the first RAW image to obtain a second RAW image. At this time, the noise signal in the second RAW image is smaller than the noise signal in the first RAW image.

[0143] In practical applications, when the electronic device cannot execute the ISP processing flow or the ISP processing performance is poor, when the electronic device inputs the collected first RAW image into the trained image processing model to obtain the second RAW image output by the trained image processing model, the second RAW image can be sent to the device for executing the ISP processing flow, so that the device for executing the ISP processing flow processes the second RAW image to obtain an RGB image corresponding to the second RAW image.

[0144] In the solution provided in this embodiment, the RAW image is processed by using an image processing model trained with a loss function configured with at least one set influencing factor. Since the image processing model has taken into account the influence of ISP on the image quality during the training process, the RAW image processed by the trained image processing model has a better image effect.

[0145] As another embodiment of the present application, Figure 7 A schematic diagram of the implementation flow of the image processing method provided by an embodiment of the present application is shown. It should be noted that the electronic device in the embodiment corresponding to the training image processing model may be different from the electronic device in this embodiment that uses the trained image processing model to process the RAW image. For example, the first electronic device trains the image processing model, the third electronic device transplants the trained image processing model into the third electronic device, and uses the trained image processing model to process the RAW image to obtain an RGB image. Here, the processing capability of the first electronic device is more powerful than that of the third electronic device. In practical applications, the first electronic device may be a server, and the third electronic device may be a terminal device such as a mobile phone, a tablet computer, or a wearable device.

[0146] Reference Figure 7 , the image processing method provided in the embodiment of the present application includes:

[0147] S401: Processing a first RAW image using a first model to obtain a second RAW image.

[0148] S402: Perform image signal processing on the second RAW image to obtain a three-primary color RGB image corresponding to the second RAW image; wherein the first model is an image processing model trained using the image processing model training method described in any of the above embodiments.

[0149] In the solution provided in this embodiment, the RAW image is processed by using an image processing model trained with a loss function configured with at least one set influencing factor. Since the image processing model has taken into account the influence of ISP on the image quality during the training process, the RAW image output by the trained image processing model is processed based on the ISP processing flow, and the quality of the obtained RGB image is better, which can improve the quality of the RGB image.

[0150] To implement the method of the embodiment of the present application, the embodiment of the present application further provides an electronic device, the electronic device includes a terminal, a server, etc. Figure 8 As shown, the electronic device includes:

[0151] An image processing unit 81 is used to input at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is used to perform set image processing on a RAW image; the first image and the second image are both RAW images;

[0152] A calculation unit 82 is used to calculate a loss value corresponding to each first image in the at least one first image by using a loss function configured with at least one set influencing factor; wherein the set influencing factor represents the influence of the image signal processing ISP on the quality of the image; the ISP is used to convert the RAW image into a three-primary color RGB image; the loss value is calculated based on the corresponding second image and the corresponding third image; the third image is a calibration image of the corresponding first image;

[0153] The updating unit 83 is used to update the weight parameters of the image processing model according to the loss value corresponding to each first image in the at least one first image.

[0154] In one embodiment, the set influencing factor includes at least one of the following:

[0155] At least one set gain function; each gain function in the at least one set gain function corresponds to a type of image gain behavior in the ISP;

[0156] At least one set mapping function; each mapping function in the at least one set mapping function corresponds to a type of image mapping behavior in the ISP.

[0157] In one embodiment, the loss function is configured with at least two set influencing factors; the image processing model training device also includes:

[0158] A determination unit, used to determine the execution order of each of the at least two set influencing factors when calculating the loss value;

[0159] A generating unit is used to generate a loss function configured with the at least two set influencing factors based on the determined execution order.

[0160] In one embodiment, the determination unit is specifically used to: determine the execution order of each of the at least two set influencing factors when calculating the loss value based on the execution order of each of the at least two set influencing factors in the ISP.

[0161] In one embodiment, when calculating the loss value of each first image in the at least one first image, the calculation unit 82 is specifically configured to:

[0162] Processing the corresponding second image using the loss function to obtain a fourth image;

[0163] Processing the corresponding third image using the loss function to obtain a fifth image;

[0164] Based on the fourth image and the fifth image, a loss value corresponding to the first image is calculated.

[0165] In one embodiment, the calculation unit 82 is specifically configured to:

[0166] Calculating a grayscale difference value corresponding to each pixel in the first image based on a first grayscale value of each pixel in the fourth image and a second grayscale value of each pixel in the fifth image;

[0167] Based on the grayscale difference corresponding to each pixel in the first image, a loss value of the grayscale value corresponding to the first image is calculated.

[0168] In actual application, each unit included in the electronic device can be implemented by a processor in the electronic device. Of course, the processor needs to run the program stored in the memory to implement the functions of the above program modules.

[0169] It should be noted that: when the electronic device provided in the above embodiment trains the image processing model, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the electronic device is divided into different program modules to complete all or part of the processing described above. In addition, the electronic device provided in the above embodiment and the image processing model training method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0170] In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides another electronic device, the electronic device includes a terminal, a server, etc., and the electronic device includes:

[0171] A processing unit is used to process the first RAW image using a first model to obtain a second RAW image; wherein the first model is an image processing model trained using the image processing model training method in any of the above embodiments.

[0172] In actual application, the processing unit included in the electronic device can be implemented by a processor in the electronic device. Of course, the processor needs to run the program stored in the memory to implement the functions of the above program modules.

[0173] To implement the method of the embodiment of the present application, another embodiment of the present application further provides an electronic device, the electronic device includes a terminal, a server, etc. Fig. 9 As shown, the electronic device includes:

[0174] A first image processing unit 91 is used to process the first RAW image using the first model to obtain a second RAW image;

[0175] The second image processing unit 92 is used to perform image signal processing on the second RAW image to obtain a three-primary color RGB image corresponding to the second RAW image; wherein the first model is an image processing model trained by the image processing model training method in any of the above embodiments.

[0176] In actual application, each unit included in the electronic device can be implemented by a processor in the electronic device. Of course, the processor needs to run the program stored in the memory to implement the functions of the above program modules.

[0177] It should be noted that: when the electronic device provided in the above embodiment processes RAW images, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the electronic device is divided into different program modules to complete all or part of the above-described processing. In addition, the electronic device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0178] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Fig.10 Schematic diagram of the hardware structure of the electronic device of the present application embodiment. Fig.10 As shown, the electronic equipment includes:

[0179] Communication interface 1, capable of exchanging information with other devices such as electronic devices;

[0180] Processor 2 is connected to communication interface 1 to implement information exchange with other devices and is used to execute any of the following when running a computer program:

[0181] The image processing model training method provided by one or more of the above technical solutions;

[0182] One or more of the above technical solutions provide an image processing method.

[0183] The computer program is stored in the memory 3 .

[0184] Of course, in actual application, the various components in the electronic device are coupled together through the bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 4.

[0185] The memory 3 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0186] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 3 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0187] The method disclosed in the above embodiment of the present application can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 2 or the instruction in the form of software. The above processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 3, and the processor 2 reads the program in the memory 3 and completes the steps of the above method in combination with its hardware.

[0188] When the processor 2 executes the program, the corresponding processes in the various methods of the embodiments of the present application are implemented, which will not be repeated here for the sake of brevity.

[0189] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 3 storing a computer program, and the computer program can be executed by a processor 2 to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0192] In addition, all functional units in the embodiments of the present application may be integrated into one processing module, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0193] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0194] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0195] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training an image processing model, characterized in that: include: Inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; The image processing model is used to perform a set image processing on the RAW image, wherein the set image processing at least includes denoising processing and / or super-resolution processing; the first image and the second image are both RAW images; A loss function configured with at least one set influencing factor is used to calculate a loss value corresponding to each first image in the at least one first image; wherein the set influencing factor represents the influence of an image signal processing ISP on the quality of the image; the ISP is used to convert a RAW image into a three-primary color RGB image; the loss value is calculated based on a corresponding second image and a corresponding third image; the third image is a calibration image of the corresponding first image; According to the loss value corresponding to each first image in the at least one first image, the weight parameter of the image processing model is updated; wherein, The set influencing factors include at least one of the following: At least one set gain function; each gain function in the at least one set gain function corresponds to a type of image gain behavior in the ISP; At least one set mapping function; each mapping function in the at least one set mapping function corresponds to a type of image mapping behavior in the ISP.

2. The image processing model training method according to claim 1, characterized in that: The loss function is configured with at least two set influencing factors; the method further comprises: Determining the execution order of each of the at least two set influencing factors when calculating the loss value; Based on the determined execution order, a loss function configured with the at least two set influencing factors is generated.

3. The image processing model training method according to claim 2, characterized in that: The determining the execution order of each of the at least two set influencing factors when calculating the loss value includes: Based on the execution order of each of the at least two set influencing factors in the ISP, the execution order of each of the at least two set influencing factors when calculating the loss value is determined.

4. The image processing model training method according to claim 1, characterized in that: When calculating the loss value corresponding to each first image in the at least one first image, the method includes: Processing the corresponding second image using the loss function to obtain a fourth image; Processing the corresponding third image using the loss function to obtain a fifth image; Based on the fourth image and the fifth image, a loss value corresponding to the first image is calculated.

5. The image processing model training method according to claim 4, characterized in that: The calculating the loss value corresponding to the first image based on the fourth image and the fifth image includes: Calculating a grayscale difference value corresponding to each pixel in the first image based on a first grayscale value of each pixel in the fourth image and a second grayscale value of each pixel in the fifth image; Based on the grayscale difference corresponding to each pixel in the first image, a loss value of the grayscale value corresponding to the first image is calculated.

6. An image processing method, characterized in that: include: The first RAW image is processed by using the first model to obtain a second RAW image; wherein, The first model is an image processing model trained using the image processing model training method as described in any one of claims 1 to 5.

7. An image processing method, characterized in that: include: Processing the first RAW image using the first model to obtain a second RAW image; Performing image signal processing on the second RAW image to obtain a three-primary color RGB image corresponding to the second RAW image; wherein: The first model is an image processing model trained using the image processing model training method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: An image processing unit, configured to input at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is configured to perform a set image processing on the RAW image, the set image processing at least comprising a denoising process and / or a super-resolution process; the first image and the second image are both RAW images; A calculation unit, configured to calculate a loss value corresponding to each first image in the at least one first image by using a loss function configured with at least one set influencing factor; wherein the set influencing factor represents the influence of an image signal processing ISP on the quality of an image; the ISP is used to convert a RAW image into a three-primary color RGB image; the loss value is calculated based on a corresponding second image and a corresponding third image; the third image is a calibration image of the corresponding first image; the set influencing factor comprises at least one of the following: at least one set gain function, each gain function in the at least one set gain function corresponds to a type of image gain behavior in the ISP; at least one set mapping function, each mapping function in the at least one set mapping function corresponds to a type of image mapping behavior in the ISP; An updating unit is used to update the weight parameters of the image processing model according to the loss value corresponding to each first image in the at least one first image.

9. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it performs any one of the following: The steps of the image processing model training method according to any one of claims 1 to 5; The steps of the image processing method according to claim 6; The steps of the image processing method according to claim 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, any of the following is achieved: The steps of the image processing model training method according to any one of claims 1 to 5; The steps of the image processing method according to claim 6; The steps of the image processing method according to claim 7.

Images