Image reconstruction model training method, image reconstruction method and device

Abstract

The embodiments of the present application provide a training method, an image reconstruction method, and an apparatus for an image reconstruction model, wherein the training method specifically includes: obtaining training data; inputting a first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and a second RAW image; updating parameters of the image reconstruction model based on the loss information; wherein weight information is determined based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; determining first loss information based on the weight information; utilizing convolution processing to convert the reconstructed RAW image into a first grayscale image, and the second RAW image into a second grayscale image; and determining second loss information based on the first grayscale image and the second grayscale image. The embodiments of the present application can improve the effect of image reconstruction.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a training method for an image reconstruction model, an image reconstruction method, and an image reconstruction device. Background Art

[0002] As camera resolution, frame rate, and sampling depth continue to increase, the bandwidth required to transmit the images or videos they output is increasing. To alleviate data transmission pressure, the images or videos output by the camera are typically compressed before being transmitted to the computing end. Because this compression process inevitably leads to a decrease in image or video quality, the computing end often needs to reconstruct the received compressed images or videos to obtain a reconstructed image. The quality of the reconstructed image is crucial for subsequent image processing tasks (such as object detection and semantic segmentation).

[0003] A RAW image can be the original data saved after the image sensor converts the captured light source signal into a digital signal, which mainly includes the physical information of the scene's light intensity and the color of the object. Compared with the RGB (Red, Green, Blue) image obtained by ISP (Image Signal Processing), the RAW image has more complete information and relatively simple noise. Therefore, using RAW images for image reconstruction can reduce the image processing process, thereby reducing the power consumption of the device corresponding to the image reconstruction. Therefore, in a related technology, a compressed image in RAW format (hereinafter referred to as RAW compressed image) is input into an image reconstruction model to obtain a reconstructed image output by the image reconstruction model.

[0004] In practical applications, RAW images have characteristics such as being too dark and unevenly distributed, as well as spatial discontinuity. These characteristics easily affect the accuracy of loss information. When the accuracy of loss information is affected, it will be detrimental to the training of the image reconstruction model, thereby affecting the effect of image reconstruction. Summary of the Invention

[0005] The embodiment of the present application provides a training method for an image reconstruction model, which can improve the effect of image reconstruction.

[0006] Accordingly, an embodiment of the present application also provides a training device for an image reconstruction model to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present invention discloses a method for training an image reconstruction model, which includes:

[0008] Acquire training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image;

[0009] Inputting the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0010] Determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information;

[0011] updating the parameters of the image reconstruction model according to the loss information;

[0012] The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0013] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0014] In order to solve the above problems, an embodiment of the present application discloses an image reconstruction method, which includes:

[0015] Receive the RAW image to be processed;

[0016] Inputting the to-be-processed RAW image into the trained image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0017] Outputting the reconstructed RAW image;

[0018] The training process of the image reconstruction model includes: acquiring training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information;

[0019] The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0020] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0021] In order to solve the above problems, the present application discloses a training device for an image reconstruction model, the device comprising:

[0022] An acquisition module is configured to acquire training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image;

[0023] a model processing module, configured to input the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0024] A loss determination module is configured to determine loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information;

[0025] a parameter updating module, configured to update the parameters of the image reconstruction model according to the loss information;

[0026] The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0027] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0028] Optionally, the use of convolution processing to convert the reconstructed RAW image into a first grayscale image, and convert the second RAW image into a second grayscale image, includes: extracting RAW sub-images corresponding to four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image.

[0029] Optionally, the convolution kernel corresponding to the RAW sub-image is a 3×3 matrix, and element values ​​of the matrix are determined according to conversion coefficients corresponding to positions and colors of the elements.

[0030] Optionally, the difference information includes: an absolute value of a difference between corresponding pixel values ​​of the reconstructed RAW image and the second RAW image;

[0031] Determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image, includes determining the weight information based on a ratio of the difference information to the pixel values ​​corresponding to the second RAW image.

[0032] Optionally, determining the weight information based on the ratio of the difference information to the pixel value corresponding to the second RAW image includes: adjusting the ratio according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

[0033] Optionally, the acquisition module includes:

[0034] A second RAW image acquisition module, configured to acquire a second RAW image from the data set;

[0035] A conversion module, configured to convert the second RAW image into channel images corresponding to four color channels respectively;

[0036] A reduction processing module is used to perform reduction processing on the channel images corresponding to the four color channels to obtain reduced channel images corresponding to the four color channels;

[0037] The fusion module is used to fuse the reduced channel images corresponding to the four color channels to obtain a first RAW image.

[0038] In order to solve the above problems, an embodiment of the present application discloses an image reconstruction device, which includes:

[0039] A receiving module, configured to receive a RAW image to be processed;

[0040] a model processing module, configured to input the RAW image to be processed into a trained image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0041] An output module, configured to output the reconstructed RAW image;

[0042] The training process of the image reconstruction model includes: acquiring training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information;

[0043] The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0044] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0045] The embodiments of the present application include the following advantages:

[0046] In the technical solution of the embodiment of the present application, training data is obtained; the training data may specifically include: a first RAW image and a second RAW image; the first RAW image is input into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; loss information is determined based on the reconstructed RAW image and the second RAW image; the loss information may specifically include: first loss information and / or second loss information; and the parameters of the image reconstruction model are updated based on the loss information.

[0047] Among them, the process of determining the first loss information can specifically include: determining weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information. Since the weight information of the embodiment of the present application takes into account the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image; in this way, when the difference corresponding to the difference information is small and the pixel values ​​corresponding to the second RAW image are also small, the weight information can increase the proportion of the loss information corresponding to a single pixel point in the loss information of the entire image, as well as the contribution of the loss information of a single pixel point to the loss information of the entire image. On this basis, the embodiment of the present application can improve the accuracy of the first loss information and can improve the reconstruction effect of the dark area in the RAW image.

[0048] The process of determining the second loss information may specifically include: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; and determining the second loss information based on the first gradient information and the second gradient information. For an image, the grayscale values ​​on both sides of its edge portion differ greatly, so the gradient value is large, so calculating the gradient of an image can highlight the edge information of the image. Therefore, the second loss information of the embodiment of the present application can characterize the edge information of the image and can be used to improve the contrast of the edge of the image. In this way, the embodiment of the present application updates the parameters of the image reconstruction model based on the second loss information, which can improve the reconstruction effect of the edge area in the RAW image.

[0049] In addition, the embodiment of the present application adopts convolution processing in the process of determining the second loss information. The above convolution processing takes into account both the human eye perception effect of the RAW image and the spatial continuity of the RAW image; at the same time, by determining the second loss information through convolution processing, the second loss information can also be made differentiable, which is conducive to applying the second loss information to the training process of the image reconstruction model.

[0050] In summary, the first loss information of the embodiment of the present application can be used to improve the reconstruction effect of the dark area in the RAW image, and the second loss information can be used to improve the contrast of the image edge and the reconstruction effect of the edge area in the RAW image. Any one or combination of the first loss information and the second loss information can improve the image reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of an application scenario of an image reconstruction model according to an embodiment of the present application;

[0052] Figure 2 This is a flowchart of the steps of a method for training an image reconstruction model according to one embodiment of the present application;

[0053] Figure 3 This is a flowchart of a process for acquiring training data according to an embodiment of the present application;

[0054] Figure 4 is a schematic diagram of a processing flow of an image reconstruction model according to an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a grid effect according to an embodiment of the present application;

[0056] Figure 6(a) is a schematic diagram of the convolution kernel corresponding to the red channel;

[0057] Figure 6(b) is a schematic diagram of the convolution kernel corresponding to the Gr and Gb channels;

[0058] Figure 6(c) is a schematic diagram of the convolution kernel corresponding to the blue channel;

[0059] Figure 7 is a schematic diagram of an image in RGGB format according to an embodiment of the present application;

[0060] Figure 8 This is a flowchart of the steps of an image reconstruction method according to an embodiment of the present application;

[0061] Figure 9 1 is a schematic structural diagram of a training device for an image reconstruction model according to an embodiment of the present application;

[0062] Figure 10 is a structural diagram of an image reconstruction device according to an embodiment of the present application;

[0063] Figure 11 It is a structural diagram of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0065] In the embodiments of the present application, image reconstruction technology can use low-quality and / or low-resolution images to generate high-quality and / or high-resolution images. Super-resolution reconstruction technology is a type of image reconstruction technology that can convert low-resolution images into high-resolution images with better visual effects through processing. Image reconstruction technology can be applied to application scenarios such as autonomous driving and medical imaging. Taking the autonomous driving scenario as an example, the reconstructed image obtained based on image reconstruction can be applied to various image processing tasks such as target detection, semantic segmentation, and detection of traffic lights and lane lines.

[0066] The image reconstruction model can be a machine learning model or a deep learning model with image reconstruction capabilities. The image reconstruction model can receive a raw image to be processed as input and output a reconstructed raw image corresponding to the raw image to be processed. The reconstructed raw image can be a high-quality and / or high-resolution image corresponding to the raw image to be processed. In other words, the quality of the reconstructed raw image can be higher than that of the raw image to be processed, and the resolution of the reconstructed raw image can be greater than or equal to the resolution of the raw image to be processed.

[0067] In the embodiments of the present application, a mathematical model can be trained to obtain an image reconstruction model. The mathematical model can be trained using machine learning, deep learning methods, and the like. Machine learning methods may include linear regression, decision trees, and random forests, while deep learning methods may include CNN (Convolutional Neural Networks), LSTM (Long Short-Term Memory), and GRU (Gated Recurrent Unit).

[0068] Reference Figure 1 , which shows a schematic diagram of an application scenario of an image reconstruction model according to an embodiment of the present application. Taking an autonomous driving scenario as an example, an image sensor 101 mounted on a vehicle captures original RAW images of the vehicle's surroundings. These original RAW images are then compressed by an encoder 102 to produce compressed RAW images. Because the precision of the compressed RAW images is lower than that of the original RAW images, the bandwidth requirements for network transmission can be reduced.

[0069] The compressed RAW image can be transmitted over a network to a computing end, which can be a server or a terminal. The decoder 103 at the computing end can use an image reconstruction model to reconstruct the compressed RAW image to obtain a reconstructed image.

[0070] The compressed RAW image is input into an image reconstruction model to obtain a reconstructed image output by the image reconstruction model. The image reconstruction model is trained using training data. The training data typically includes: a first RAW image and a second RAW image, wherein a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image. During the training process of the image reconstruction model, the first RAW image can be input into the image reconstruction model, and the image reconstruction model outputs a reconstructed RAW image. Furthermore, loss information can be determined based on the reconstructed RAW image and the second RAW image, and the parameters of the image reconstruction model can be updated based on the loss information.

[0071] In practical applications, the reconstructed RAW image and the second RAW image have characteristics such as being too dark and unevenly distributed, as well as spatial continuity. These characteristics easily affect the accuracy of the loss information. When the accuracy of the loss information is affected, it will be detrimental to the training of the image reconstruction model, thereby affecting the effect of image reconstruction.

[0072] The quality of reconstructed images is crucial for processing tasks such as object detection, semantic segmentation, and traffic light and lane line detection. For example, poor reconstructed image quality can make it difficult to accurately identify objects in object detection tasks, potentially posing a significant risk to autonomous driving.

[0073] Therefore, an embodiment of the present application provides a method for training an image reconstruction model, the method specifically comprising: obtaining training data; the training data specifically comprising: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image being less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information specifically comprising: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information;

[0074] The process of determining the first loss information may specifically include: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0075] The process of determining the second loss information may specifically include: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0076] Image reconstruction models usually use L1 loss (L1 loss function). The formula of L1 loss is as follows:

[0077]

[0078] in, represents the pixel value of the reconstructed RAW image, and Y represents the pixel value of the second RAW image. In practical applications, the absolute value of the difference between all pixels of the reconstructed RAW image and the second RAW image is usually averaged to obtain the loss value L1.

[0079] It can be seen from formula (1) that for a pixel point in the reconstructed RAW image and the second RAW image, the larger the absolute value of the difference between the corresponding pixel values ​​of the reconstructed RAW image and the second RAW image, the larger the loss value corresponding to the pixel point, and the greater the contribution of the loss value to the optimization of the image reconstruction model.

[0080] However, RAW images are often too dark and unevenly distributed. In particular, when using 8 bits (binary bits) to represent RAW images, the pixel values ​​of dark pixels in the RAW image are compressed, reducing the absolute value of the difference and, in turn, the proportion of the loss value corresponding to the dark pixels in the loss value L1. In this case, applying the loss value L1 to the training of the image reconstruction model will cause blurring of the dark areas in the RAW image, thus affecting the image reconstruction effect.

[0081] In response to the technical problem in the related art that the loss value L1 affects the reconstruction effect of the dark area in the RAW image, the embodiment of the present application proposes the probability of the first loss information, which can be used to improve the reconstruction effect of the dark area in the RAW image.

[0082] In an embodiment of the present application, the process of determining the first loss information may specifically include: determining weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information.

[0083] Because the weight information in the embodiment of the present application takes into account the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, as well as the pixel values ​​corresponding to the second RAW image, when the difference information corresponding to the difference information is small and the pixel values ​​corresponding to the second RAW image are also small, the weight information can increase the proportion of the loss information corresponding to a single pixel point in the loss information of the entire image, as well as the contribution of the loss information of a single pixel point to the loss information of the entire image. On this basis, the embodiment of the present application can improve the accuracy of the first loss information and improve the reconstruction effect of dark areas in the RAW image.

[0084] The present application also proposes the concept of second loss information. The process of determining this second loss information may specifically include: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining first gradient information corresponding to the first grayscale image, and determining second gradient information corresponding to the second grayscale image; and determining second loss information based on the first gradient information and the second gradient information.

[0085] The gradient of an image represents the rate of change in the image's grayscale values. For an image, the grayscale values ​​on both sides of its edge differ significantly, resulting in a large gradient value. Therefore, calculating the gradient of an image can highlight the image's edge information. Therefore, the second loss information in the embodiments of the present application can characterize the image's edge information and can be used to increase the contrast of the image's edges. Thus, the embodiments of the present application update the parameters of the image reconstruction model based on the second loss information, thereby improving the reconstruction effect of edge areas in RAW images.

[0086] The embodiment of the present application adopts convolution processing in the process of determining the second loss information. The above convolution processing takes into account both the human eye perception effect of the RAW image and the spatial continuity of the RAW image; at the same time, by determining the second loss information through convolution processing, the second loss information can also be made differentiable, which is conducive to applying the second loss information to the training process of the image reconstruction model.

[0087] In summary, the first loss information of the embodiment of the present application can be used to improve the reconstruction effect of the dark area in the RAW image, and the second loss information can be used to improve the contrast of the image edge and the reconstruction effect of the edge area in the RAW image. Any one or combination of the first loss information and the second loss information can improve the image reconstruction effect.

[0088] Method Example 1

[0089] Reference Figure 2 , shows a schematic flow chart of the steps of a method for training an image reconstruction model according to an embodiment of the present application. The method may specifically include the following steps:

[0090] Step 201: Acquire training data; the training data may specifically include: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image;

[0091] Step 202: input the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0092] Step 203: determining loss information based on the reconstructed RAW image and the second RAW image; the loss information may specifically include: first loss information and / or second loss information;

[0093] Step 204: Update the parameters of the image reconstruction model according to the loss information;

[0094] The process of determining the first loss information may specifically include: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0095] The process of determining the second loss information may specifically include: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0096] Figure 2 The method shown can be used to update the parameters of the image reconstruction model during the training process of the image reconstruction model.

[0097] The training process of the image reconstruction model can include: forward propagation and back propagation.

[0098] Among them, forward propagation can calculate the output information (i.e., reconstruct the RAW image) in sequence from the input layer to the output layer according to the parameters of the image reconstruction model. The reconstructed RAW image can be used to determine the loss information.

[0099] Backward propagation can calculate and update the parameters of the image reconstruction model in sequence from the output layer to the input layer based on the loss information. The image reconstruction model usually adopts the structure of a neural network, and the parameters of the image reconstruction model may include parameters such as the weights of the neural network. In the back propagation process, the gradient information of the parameters of the image reconstruction model can be determined, and the parameters of the image reconstruction model can be updated using the gradient information. For example, back propagation can calculate and store the gradient information of the parameters of the processing layers (including the input layer, intermediate layer, and output layer) of the image reconstruction model in sequence from the output layer to the input layer based on the chain rule in calculus.

[0100] In step 201 , the training data may specifically include: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image.

[0101] In practical applications, the above-mentioned process of obtaining training data may specifically include: obtaining a second RAW image from a data set; converting the second RAW image into channel images corresponding to four color channels respectively; reducing the channel images corresponding to the four color channels respectively to obtain reduced channel images corresponding to the four color channels respectively; and fusing the reduced channel images corresponding to the four color channels respectively to obtain a first RAW image.

[0102] The embodiments of the present application are not limited to a specific data set. In one example, the data set may include: M RAW images, and the M RAW images may be randomly divided into two parts, one as a training data set and the other as a test data set.

[0103] Training an image reconstruction model requires paired images: a low-resolution image and its corresponding high-resolution image, and / or a low-quality image and a high-quality image of the same resolution. Specifically, in this embodiment, the paired images include a first RAW image and a second RAW image; the first RAW image has a first resolution less than or equal to the second RAW image's second resolution, and the quality of the first RAW image is lower than the quality of the second RAW image.

[0104] In the embodiment of the present application, a second RAW image may be first acquired from a data set, and then a low-resolution first RAW image may be produced according to the second RAW image.

[0105] The process of producing a low-resolution first RAW image based on a second RAW image specifically includes: converting the second RAW image into channel images corresponding to four color channels; reducing the channel images corresponding to the four color channels to obtain reduced channel images corresponding to the four color channels; and fusing the reduced channel images corresponding to the four color channels to obtain the first RAW image.

[0106] Assuming the height and width of the second RAW image are 2H and 2W, respectively, the 2H×2W second RAW image can be converted into a channel image of size H×W×4. The arrangement of the channel image is related to the Bayer format of the second RAW image. If the Bayer format of the second RAW image is RGGB, the channel image format is RGGB, that is, the first channel is R, the second channel is Gr (G closer to R), the third channel is Gb (G closer to B), and the fourth channel is B.

[0107] The Bayer format of the second RAW image can also be other Bayer formats such as BGGR, RGBG, etc. For the second RAW images in other Bayer formats, since their corresponding channel images are similar to the channel images corresponding to RGGB, they are not described here in detail and can be referenced to each other.

[0108] In the embodiment of the present application, an interpolation method may be used to reduce the channel images corresponding to the four color channels. For a channel image of size H×W×4, the reduced channel image may be sized to (H / 2)×(W / 2)×4.

[0109] Furthermore, the embodiment of the present application can perform a first fusion on the reduced channel images corresponding to the four color channels. The corresponding first fusion process can be a process of filling the values ​​of the reduced channel image into the first RAW image according to the Bayer format of the second RAW image. When the size of the reduced channel image is (H / 2)×(W / 2)×4, the size of the first RAW image can be H×W. The above-mentioned first fusion process can be equivalent to the inverse process of converting the second RAW image into the channel images corresponding to the four color channels.

[0110] Reference Figure 3 , shows a flowchart of the training data acquisition process of an embodiment of the present application, wherein the second RAW image in RGGB format can be converted into channel images corresponding to the R, Gr, Gb and B channels respectively, and the channel images corresponding to the R, Gr, Gb and B channels respectively are reduced to obtain reduced channel images; then, the reduced channel images corresponding to the R, Gr, Gb and B channels respectively are fused to obtain the first channel image.

[0111] To increase the diversity of training data, embodiments of the present application may further perform data augmentation on a set of training data (including a second RAW image and a first RAW image) to obtain multiple sets of training data. Since multiple sets of training data can be generated from a single set of training data, the diversity of the training data can be increased. Data augmentation may include random cropping, random flipping, and the like.

[0112] Random cropping can randomly obtain a partial image from the complete image to enhance the robustness of the image reconstruction model. Random flipping can include horizontal flipping and vertical flipping.

[0113] In step 202, the image reconstruction model may include a feature extraction unit and an image reconstruction unit. The feature extraction unit extracts features from an input image, such as a first RAW image, to obtain image features. The image reconstruction unit may be configured to reconstruct the image based on the image features to obtain a reconstructed RAW image.

[0114] In practical applications, the feature extraction unit can construct a convolutional neural network based on a series of convolutional layers and activation functions, and use the convolutional neural network to extract image features (such as an image feature map) from the input image. The image reconstruction unit can use operations such as deconvolution and upsampling to upsample the image feature map to obtain a reconstructed RAW image.

[0115] Reference Figure 4 , shows a schematic diagram of the processing flow of the image reconstruction model of an embodiment of the present application, wherein, before the first RAW image is input into the image reconstruction model, the first RAW image with a size of H×W can be converted into a channel image of (H / 2)×(W / 2)*4, and the channel image of (H / 2)×(W / 2)*4 is input into the image reconstruction model, and the image reconstruction model outputs the reconstructed RAW image, so that the computational complexity of the image reconstruction model can be reduced.

[0116] Figure 4 The reconstructed RAW image output by the image reconstruction model can be in the form of channel images corresponding to the four color channels. In this embodiment of the present application, a second fusion process can be performed on the channel images corresponding to the four color channels to obtain a reconstructed RAW image. The corresponding second fusion process can be similar to the first fusion process and can be referenced to each other.

[0117] In step 203, the first loss information can be used to improve the reconstruction effect of the dark area in the RAW image, and the second loss information can be used to improve the contrast of the image edge and the reconstruction effect of the edge area in the RAW image. Any one or combination of the first loss information and the second loss information can improve the image reconstruction effect.

[0118] The process of determining the first loss information may specifically include: determining weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information.

[0119] Because the weight information in the embodiment of the present application takes into account the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, as well as the pixel values ​​corresponding to the second RAW image, when the difference information corresponding to the difference information is small and the pixel values ​​corresponding to the second RAW image are also small, the weight information can increase the proportion of the loss information corresponding to a single pixel point in the loss information of the entire image, as well as the contribution of the loss information of a single pixel point to the loss information of the entire image. On this basis, the embodiment of the present application can improve the accuracy of the first loss information and improve the reconstruction effect of dark areas in the RAW image.

[0120] In a specific implementation, the difference information may include: an absolute value of a difference between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image.

[0121] The process of determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel value corresponding to the second RAW image, may specifically include: determining the weight information based on a ratio of the difference information to the pixel value corresponding to the second RAW image. The ratio w of the difference information to the pixel value corresponding to the second RAW image is as shown in formula (2), where the denominator of formula (2) represents the pixel value corresponding to the second RAW image, and the numerator represents the absolute value of the difference between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image.

[0122]

[0123] In practical applications, the ratio w can be directly used as weight information. Alternatively, the ratio w can be further processed to obtain weight information.

[0124] In the case where the contrast value w is further processed, the above process of determining the weight information based on the ratio of the difference information to the pixel value corresponding to the second RAW image may specifically include: adjusting the ratio according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

[0125] Adjusting the ratio according to the preset range can reduce the impact of the imbalance of the ratio w distribution on the image reconstruction effect to a certain extent. The preset range can be [1, m], where m can be a positive integer. Those skilled in the art can determine the value of m based on actual application requirements. When m is 5, the process of adjusting the ratio according to the preset range is shown in formula (3), where w' represents the adjusted ratio:

[0126] w′=min(w,5) (3)

[0127] Using the exponential function a w’ By processing the adjustment ratio, the proportion of the first loss information corresponding to the dark area in the complete image can be further increased while improving the image reconstruction effect of the brightness area in the RAW image.

[0128] In the embodiment of the present application, a RAW image may include: a bright area, a dark area, and a mid-tone area. The bright area may refer to the brightest portion near the highlight; the dark area may refer to the darker portion of the boundary between light and dark, and its corresponding pixel value may be around 0. The mid-tone area may refer to the gray portion near the boundary between light and dark.

[0129] When the base of the exponential function is 2, the value range of the weight information w" is 1-32 (when m is 5), and the weight information w" fully considers the difference information between the reconstructed image and the second RAW image and the size of the pixel value of the second RAW image itself. It can be seen from formula (4) that when the absolute value of the difference corresponding to the difference information is small, but the pixel value of the second RAW image itself is also small, the weight information can increase the proportion of the loss information corresponding to a single pixel point in the loss information of the entire image, as well as the contribution of the loss information of a single pixel point to the loss information of the entire image. On this basis, the embodiment of the present application can improve the accuracy of the first loss information and improve the reconstruction effect of the dark area in the RAW image.

[0130] w″=2 w′ (4)

[0131] The process of determining the first loss information L′1 according to the weight information in the embodiment of the present application is shown in formula (5):

[0132]

[0133] Using image edge information for auxiliary training is an effective method to improve image reconstruction. However, RAW images are acquired through a Bayer filter and have certain spatial discontinuities. Therefore, directly extracting edge information does not produce true image edge information. A common method is to convert the RAW image to RGB and perform gradient calculations on the RGB image. However, the RAW to RGB conversion process is relatively complex and not suitable for training.

[0134] An embodiment of the present application provides a method for determining the second loss information, which can overcome the spatial discontinuity of the RAW image and is simple to derive, and can be directly used as the loss function of the image reconstruction model to achieve excellent visualization effects.

[0135] The process of determining the second loss information may specifically include: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0136] The embodiment of the present application first adopts a differentiable convolution processing method to convert the RAW image into a grayscale image, and then uses a gradient operator to determine the gradient information corresponding to the grayscale image. The gradient information may include: first gradient information and second gradient information; then, based on the first gradient information and the second gradient information, the second loss information is determined.

[0137] The above-mentioned process of converting the reconstructed RAW image into the first grayscale image and converting the second RAW image into the second grayscale image using convolution processing can specifically include: extracting RAW sub-images corresponding to the four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image.

[0138] The conversion method from RGB image to grayscale image is shown in formula (6). This method can reflect the sensitivity of human eyes to color and is a commonly used grayscale conversion method:

[0139] Gray=0.3*R + 0.59*G + 0.11*B (6)

[0140] Common grayscale conversion methods usually directly use formula (6) to expand the RAW image into a grayscale image. This expansion method does not take into account the perception effect of the human eye and will have an obvious grid effect.

[0141] Reference Figure 5 , which shows a schematic diagram of a grid effect according to an embodiment of the present application, Figure 5 The image can be used to reconstruct a part of the RAW image. Figure 5 The networks in the image are caused by the spatial discontinuities in the RAW image.

[0142] In response to the grid effect problem existing in the commonly used grayscale conversion method, the embodiment of the present application uses convolution processing to convert the RAW image into a grayscale image, which can overcome the grid effect problem caused by the spatial discontinuity of the RAW image, thereby improving the human eye perception of the image.

[0143] Taking an RGGB RAW image (which may include the aforementioned reconstructed RAW image or the second RAW image) as an example, assuming that the height of the RAW image is H, the width is W, and the starting pixel arrangement is 0, the process of converting the RAW image to a grayscale image is as follows:

[0144] Step 1: Extract RAW sub-images corresponding to the four color channels from the RGGB RAW image, so that the starting points of the RAW sub-images are R, Gr, Gb, and B respectively;

[0145] Rraw = Raw[0:W-1,0:H-1], RAW sub-image starting at R pixel, with a resolution of (H-1, W-1)

[0146] Grraw = Raw[1:W,0:H-1], a RAW sub-image starting at the Gr pixel with a resolution of (H-1, W-1)

[0147] Gbraw=Raw[0:W-1,1:H], RAW sub-image starting with Gb pixels, resolution (H-1, W-1)

[0148] Braw = Raw[1:W,1:H], a RAW sub-image starting with pixel B, with a resolution of (H-1, W-1)

[0149] Step 2: Convolution processing is performed on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels;

[0150] The convolution processing corresponding to the red channel is shown in formula (7), where k represents the convolution kernel, stride represents the step size parameter of the convolution, and padding represents the padding parameter of the convolution.

[0151] R raw = onv(Rraw,k= m, stride=2, padding=1) (7)

[0152] The size of the RAW sub-image after convolution processing is 1 / 4 of the RAW image, which is obtained by performing convolution operation on the surrounding 3*3 pixels.

[0153] The convolution kernel corresponding to the RAW sub-image may be a 3×3 matrix, and the element values ​​of the matrix may be determined according to the position corresponding to the element and the conversion coefficient of the color corresponding to the element. The color conversion coefficient may be the conversion coefficient corresponding to R, G, and B in formula (6), specifically, the conversion coefficient corresponding to R is 0.3, the conversion coefficient corresponding to G is 0.59, and the conversion coefficient corresponding to B is 0.11.

[0154] The embodiment of the present application can determine the element value of the matrix corresponding to the convolution kernel according to the position corresponding to the element and the conversion coefficient of the color corresponding to the element.

[0155] According to the characteristics of convolution processing, the 3×3 pixel values ​​in the RAW sub-image are multiplied by the 3×3 convolution kernel, and then added to obtain the output result.

[0156] For the RAW sub-image corresponding to the red channel, the pixel values ​​of the 3×3 matrix in the RAW sub-image are shown in the image on the left of Figure 6(a), and the convolution kernel Rm corresponding to the red channel is shown in the matrix on the right of Figure 6(a). According to formula (6), the element value corresponding to the R position in the convolution kernel Rm is 0.3, the element value corresponding to the B position is 0.11 / 4, and the element value corresponding to the G position is 0.59 / 4.

[0157] Therefore, in the convolution kernel corresponding to the red channel, the element value corresponding to the red position is the red conversion coefficient, the element value corresponding to the blue position is the ratio of the blue conversion coefficient to 4, and the element value corresponding to the green position is the ratio of the green conversion coefficient to 4;

[0158] During the convolution process for the red channel, the combination and spatial distribution of color information are taken into account, with pixels closer to the center being given greater weight. A step size of 2 allows the convolution process to be centered around the red R pixel in the RAW sub-image.

[0159] The convolution processing corresponding to the Gr channel is shown in formula (8):

[0160] Gr raw = onv(Gr raw ,k= m, stride=2, padding=1) (8)

[0161] The implementation process of the convolution processing corresponding to the Gr channel is similar to that corresponding to the red channel, except that the convolution kernel used is Gm, and the convolution processing is performed on the Gr color pixels in the RAW sub-image. The 3×3 pixel values ​​in the Gr channel RAW sub-image are shown in the image on the left of Figure 6(b), and the convolution kernel Gm corresponding to the Gr channel is shown in the matrix on the right of Figure 6(b). Corresponding to formula (6), the element value corresponding to the R position in the convolution kernel Gm is 0.3 / 2, and the element value corresponding to the B position is 0.11 / 2. The conversion coefficient 0.59 corresponding to the green G is divided into four 0.1 and one 0.19. Among them, the element value corresponding to the G position at the center is 0.19, and the element value corresponding to the G position at the periphery is 0.1. In this way, the pixel value at the convolution center can be given a greater weight, which can maintain the image itself and prevent excessive smoothing.

[0162] Therefore, in the convolution kernel corresponding to the green channel, the element value corresponding to the red position is the ratio of the red conversion coefficient to 2, the element value corresponding to the blue position is the ratio of the blue conversion coefficient to 2, and the element value corresponding to the green position is the allocated value corresponding to the green conversion coefficient. The allocated values ​​specifically include the four 0.1s and the one 0.19 mentioned above.

[0163] The convolution processing corresponding to the Gb channel is shown in formula (9):

[0164] Gb raw = onv(Gb raw ,k= m, stride=2, padding=1) (9)

[0165] The convolution process for the Gb channel is similar to that for the red channel, except that the convolution kernel used is Gm, which is used to convolve the Gb pixels in the RAW sub-image. The convolution kernel Gm for the Gb channel is shown in the matrix on the right side of Figure 6(b).

[0166] The convolution processing corresponding to the B channel is shown in formula (10):

[0167] B raw = onv(B raw , =B m ,stride=2,padding=1) (10)

[0168] The implementation process of Braw is the same as Rraw, except that it uses the convolution kernel Bm, which processes all blue pixels in Raw.

[0169] The convolution process for the B channel is similar to that for the red channel, except that the convolution kernel used is Bm, which performs convolution on the B-color pixels in the RAW sub-image. The 3×3 pixel values ​​in the B channel RAW sub-image are shown in the image on the left in Figure 6(c), and the convolution kernel Bm corresponding to the B channel is shown in the matrix on the right in Figure 6(c). According to formula (6), the element value corresponding to the R position in the convolution kernel Bm is 0.3 / 4, the element value corresponding to the B position is 0.11, and the element value corresponding to the G position is 0.59 / 4.

[0170] Therefore, in the convolution kernel corresponding to the blue channel, the element value corresponding to the red position is the ratio of the red conversion coefficient to 4, the element value corresponding to the blue position is the blue conversion coefficient, and the element value corresponding to the green position is the ratio of the green conversion coefficient to 4.

[0171] Step 3: Fuse the RAW convolution result images corresponding to the four color channels to obtain a grayscale image.

[0172] like Figure 7 As shown, assuming that the grayscale image is arranged in RGGB format, the pixel values ​​corresponding to the RAW convolution result images corresponding to the four color channels can be filled into the positions of the corresponding colors in the grayscale image. In other words, the RAW convolution result images corresponding to the four color channels are components of the pixel values ​​of the grayscale image.

[0173] The sources of the grayscale images in the embodiments of this application are as follows:

[0174] Gray[0:2:W,0:2:H]=Rraw. This expression means that the value of the R pixel in the grayscale image comes from the RAW convolution result image Rraw corresponding to the red channel.

[0175] Assuming the size of the grayscale image is W×H, the coordinate of the red pixel in the width direction can be 2i, and the coordinate of the red pixel in the height direction can be 2j, where i and j are zero or positive integers. The element values ​​of the RAW convolution result image Rraw corresponding to the red channel can be filled to the positions corresponding to the red pixels in order from top to bottom and from left to right.

[0176] Gray[1:2:W,0:2:H]=Grraw.

[0177] Assuming the size of the grayscale image is W×H, the coordinates of the Gr color pixel in the width direction can be 2i+1, and the coordinates of the Gr color pixel in the height direction can be 2j. The element values ​​of the RAW convolution result image Grraw corresponding to the Gr color channel can be filled to the positions corresponding to the Gr color pixels in order from top to bottom and from left to right.

[0178] Gray[0:2:W,1:2:H]=Gbraw

[0179] Assuming the size of the grayscale image is W×H, the coordinate of the Gb color pixel in the width direction can be 2i, and the coordinate of the Gr color pixel in the height direction can be 2j+1. The element values ​​of the RAW convolution result image Gbraw corresponding to the Gb color channel can be filled to the positions corresponding to the Gb color pixels in order from top to bottom and from left to right.

[0180] Gray[1:2:W,1:2:H]=Braw

[0181] Assuming the size of the grayscale image is W×H, the coordinate of the blue pixel in the width direction can be 2i+1, and the coordinate of the blue pixel in the height direction can be 2j+1. The element values ​​of the RAW convolution result image Braw corresponding to the blue channel can be filled to the position corresponding to the blue pixel in order from top to bottom and from left to right.

[0182] Steps 1 through 3 illustrate the process of converting a RAW image to a grayscale image, primarily using the RGGB format as an example. As you can understand, for non-RGGB RAW images, the conversion process is similar to the process for converting RAW images to grayscale, so they can be cross-referenced and omitted for clarity. In the process of converting non-RGGB RAW images to grayscale, the Rm convolution kernel is used for R pixels, the Bm convolution kernel is used for B pixels, and the Gm convolution kernel is used for G pixels. The Rm, Bm, and Gm convolution kernels take into account the human eye's sensory perception while giving greater weight to the center position.

[0183] The reconstructed RAW image is converted into a first grayscale image, and the second RAW image is converted into a second grayscale image; first gradient information corresponding to the first grayscale image is determined, and second gradient information corresponding to the second grayscale image is determined; and second loss information is determined based on the first gradient information and the second gradient information.

[0184] The embodiment of the present application uses a gradient operator to determine the gradient information corresponding to the grayscale image. The gradient operator may include: a horizontal gradient operator and a vertical gradient operator. Examples of gradient operators may include: a Sobel operator, etc.

[0185] An example of the horizontal gradient operator is shown in formula (11):

[0186]

[0187] An example of the vertical gradient operator is shown in formula (12):

[0188]

[0189] In combination with formula (6) to formula (12), the process of determining the second loss information may include: using the matrix corresponding to the convolution kernel to convert the reconstructed RAW image and the second RAW image into corresponding grayscale images respectively; determining the corresponding horizontal gradient information and vertical gradient information based on the grayscale images corresponding to the reconstructed RAW image and the second RAW image respectively; determining the second loss information based on the horizontal gradient information and vertical gradient information corresponding to the reconstructed RAW image and the second RAW image respectively.

[0190] The second loss information may include horizontal dimension loss information Lx and vertical dimension loss information Ly. Formula (13) shows the process of determining Lx, formula (14) shows the process of determining Ly, and formula (15) shows the process of adding Lx and Ly to obtain the second loss information LG.

[0191]

[0192]

[0193] L G =Lx+Ly (15)

[0194] In formula (13) and formula (14), the first term represents the horizontal gradient information and vertical gradient information corresponding to the reconstructed RAW image, respectively, and the second term represents the horizontal gradient information and vertical gradient information corresponding to the second RAW image, respectively. A horizontal gradient operator can be used as a convolution kernel k to perform a convolution operation to obtain horizontal gradient information; a vertical gradient operator can be used as a convolution kernel k to perform a convolution operation to obtain vertical gradient information.

[0195] In summary, the loss information of the embodiment of the present application may include: any one or a combination of the first loss information and the second loss information. Among them, when the first loss information is used, the function shown in formula (5) can be used as the loss function of the image reconstruction model. When the second loss information is used, the function shown in formula (15) can be used as the loss function of the image reconstruction model. When the combination of the first loss information and the second loss information is used, the function shown in formula (16) can be used as the loss function of the image reconstruction model.

[0196] loss = 2 × L G +1×L′1 (16)

[0197] In formula (16), w1 and w2 represent the weights of the first loss information and the second loss information, respectively. The loss function shown in formula (16) can not only optimize the reconstruction effect of the dark area of ​​the RAW image, but also highlight the high-frequency information of the image by increasing the weight of the edge information, thereby achieving better reconstruction effect.

[0198] In step 204, regardless of whether the loss function shown in formula (5) or formula (15) is used, or the loss function shown in formula (16) is used, since the loss function in formula (5), formula (15) and formula (16) is It can include a series of processing processes based on the parameters of the network reconstruction model, so in formula (5), formula (15) and formula (16) It may include parameters of the network reconstruction model. In this way, the loss function of the embodiment of the present application can characterize the mapping relationship between the loss information and the parameters of the network reconstruction model. In practical applications, the partial derivatives of the parameters of the loss function (including the parameters of the network reconstruction model) can be calculated, and the partial derivatives of the obtained parameters can be written in the form of a vector. The vector corresponding to the partial derivative can be called the gradient information corresponding to the parameter. The update amount corresponding to the parameter can be obtained based on the gradient information and the step size information.

[0199] When using a gradient descent method, batch gradient descent, stochastic gradient descent, or mini-batch gradient descent can be used. In a specific implementation, iterations can be performed based on a single set of training data, or multiple sets of training data. The convergence condition for the iterations can be that the loss information meets a preset condition. The preset condition can be that the absolute value of the difference between the loss information and a preset value is less than a difference threshold, or that the number of iterations exceeds a threshold. In other words, if the loss information meets the preset condition, the iterations can be terminated; in this case, the target parameter values ​​for the network reconstruction model can be obtained.

[0200] In summary, the training method of the image reconstruction model of the embodiment of the present application obtains training data; the training data may specifically include: a first RAW image and a second RAW image; the first RAW image is input into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; loss information is determined based on the reconstructed RAW image and the second RAW image; the loss information may specifically include: first loss information and / or second loss information; and the parameters of the image reconstruction model are updated based on the loss information.

[0201] Among them, the process of determining the first loss information can specifically include: determining weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information. Since the weight information of the embodiment of the present application takes into account the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image; in this way, when the difference corresponding to the difference information is small and the pixel values ​​corresponding to the second RAW image are also small, the weight information can increase the proportion of the loss information corresponding to a single pixel point in the loss information of the entire image, as well as the contribution of the loss information of a single pixel point to the loss information of the entire image. On this basis, the embodiment of the present application can improve the accuracy of the first loss information and can improve the reconstruction effect of the dark area in the RAW image.

[0202] The process of determining the second loss information may specifically include: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; and determining the second loss information based on the first gradient information and the second gradient information. For an image, the grayscale values ​​on both sides of its edge portion differ greatly, so the gradient value is large, so calculating the gradient of an image can highlight the edge information of the image. Therefore, the second loss information of the embodiment of the present application can characterize the edge information of the image and can be used to improve the contrast of the edge of the image. In this way, the embodiment of the present application updates the parameters of the image reconstruction model based on the second loss information, which can improve the reconstruction effect of the edge area in the RAW image.

[0203] In addition, the embodiment of the present application adopts convolution processing in the process of determining the second loss information. The above convolution processing takes into account both the human eye perception effect of the RAW image and the spatial continuity of the RAW image; at the same time, by determining the second loss information through convolution processing, the second loss information can also be made differentiable, which is conducive to applying the second loss information to the training process of the image reconstruction model.

[0204] In summary, the first loss information of the embodiment of the present application can be used to improve the reconstruction effect of the dark area in the RAW image, and the second loss information can be used to improve the contrast of the image edge and the reconstruction effect of the edge area in the RAW image. Any one or combination of the first loss information and the second loss information can improve the image reconstruction effect.

[0205] Method Example 2

[0206] Reference Figure 8 , shows a schematic flow chart of the steps of an image reconstruction method according to an embodiment of the present application, which may specifically include the following steps:

[0207] Step 801: receiving a RAW image to be processed;

[0208] Step 802: input the RAW image to be processed into the trained image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0209] Step 803: output the reconstructed RAW image;

[0210] The training process of the image reconstruction model specifically includes: obtaining training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information;

[0211] The process of determining the first loss information specifically includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0212] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0213] Figure 8 The illustrated method embodiment can utilize the trained image reconstruction model to process the RAW image to be processed to obtain a reconstructed RAW image.

[0214] Since the first loss information and / or the second loss information are used in the training process of the image reconstruction model, the first loss information of the embodiment of the present application can be used to improve the reconstruction effect of the dark area in the RAW image, and the second loss information can be used to improve the contrast of the image edge and the reconstruction effect of the edge area in the RAW image. Any one or combination of the first loss information and the second loss information can improve the image reconstruction effect; therefore, the embodiment of the present application can improve the image reconstruction effect of the RAW image to be processed.

[0215] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0216] Based on the above embodiment, this embodiment also provides a training device for an image reconstruction model. Figure 9The device may specifically include: an acquisition module 901, a model processing module 902, a loss determination module 903 and a parameter updating module 904.

[0217] The acquisition module 901 is configured to acquire training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image;

[0218] A model processing module 902 is configured to input the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0219] A loss determination module 903 is configured to determine loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information;

[0220] a parameter updating module 904, configured to update the parameters of the image reconstruction model according to the loss information;

[0221] The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0222] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0223] Optionally, the use of convolution processing to convert the reconstructed RAW image into a first grayscale image, and convert the second RAW image into a second grayscale image, includes: extracting RAW sub-images corresponding to four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image.

[0224] Optionally, the convolution kernel corresponding to the RAW sub-image is a 3×3 matrix, and element values ​​of the matrix are determined according to conversion coefficients corresponding to positions and colors of the elements.

[0225] Optionally, the difference information includes: an absolute value of a difference between corresponding pixel values ​​of the reconstructed RAW image and the second RAW image;

[0226] Determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and the pixel values ​​corresponding to the second RAW image, includes determining the weight information based on a ratio of the difference information to the pixel values ​​corresponding to the second RAW image.

[0227] Optionally, determining the weight information based on the ratio of the difference information to the pixel value corresponding to the second RAW image includes: adjusting the ratio according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

[0228] Optionally, the acquisition module includes:

[0229] A second RAW image acquisition module, configured to acquire a second RAW image from the data set;

[0230] A conversion module, configured to convert the second RAW image into channel images corresponding to four color channels respectively;

[0231] A reduction processing module is used to perform reduction processing on the channel images corresponding to the four color channels to obtain reduced channel images corresponding to the four color channels;

[0232] The fusion module is used to fuse the reduced channel images corresponding to the four color channels to obtain a first RAW image.

[0233] Based on the above embodiment, this embodiment also provides a training device for an image reconstruction model. Figure 10 The device may specifically include: a receiving module 1001, a model processing module 1002 and an output module 1003.

[0234] The receiving module 1001 is configured to receive a RAW image to be processed;

[0235] The model processing module 1002 is configured to input the RAW image to be processed into a trained image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model;

[0236] An output module 1003 is used to output the reconstructed RAW image;

[0237] The training process of the image reconstruction model includes: acquiring training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information;

[0238] The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information;

[0239] The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining the first gradient information corresponding to the first grayscale image, and determining the second gradient information corresponding to the second grayscale image; determining the second loss information based on the first gradient information and the second gradient information.

[0240] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0241] The present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In the present application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0242] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include: a terminal device, a server (cluster), and other electronic devices. Figure 11 An exemplary apparatus 1100 that can be used to implement various embodiments described in this application is schematically illustrated.

[0243] For one embodiment, Figure 11An exemplary apparatus 1100 is shown having one or more processors 1102, a control module (chip set) 1104 coupled to at least one of the processor(s) 1102, a memory 1106 coupled to the control module 1104, a non-volatile memory (NVM) / storage device 1108 coupled to the control module 1104, one or more input / output devices 1110 coupled to the control module 1104, and a network interface 1112 coupled to the control module 1104.

[0244] The processor 1102 may include one or more single-core or multi-core processors, and the processor 1102 may include any combination of general-purpose processors or dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 1100 can serve as a terminal device, server (cluster), or other device described in the embodiments of the present application.

[0245] In some embodiments, the apparatus 1100 may include one or more computer-readable media (e.g., memory 1106 or NVM / storage 1108) having instructions 1114 and one or more processors 1102 configured in conjunction with the one or more computer-readable media to execute the instructions 1114 to implement a module to perform the actions described in the present disclosure.

[0246] For one embodiment, the control module 1104 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1102 and / or any suitable device or component in communication with the control module 1104 .

[0247] The control module 1104 may include a memory controller module to provide an interface to the memory 1106. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0248] The memory 1106 may be used, for example, to load and store data and / or instructions 1114 for the device 1100. For one embodiment, the memory 1106 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1106 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0249] For one embodiment, the control module 1104 may include one or more input / output controllers to provide an interface to the NVM / storage device 1108 and the input / output device(s) 1110 .

[0250] For example, NVM / storage 1108 may be used to store data and / or instructions 1114. NVM / storage 1108 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0251] The NVM / storage device 1108 may include storage resources that are physically part of the device on which the apparatus 1100 is installed, or it may be accessible to the device without being part of the device. For example, the NVM / storage device 1108 may be accessible over a network via the input / output device(s) 1110.

[0252] (One or more) input / output devices 1110 may provide an interface for apparatus 1100 to communicate with any other appropriate device. Input / output devices 1110 may include communication components, audio components, sensor components, etc. Network interface 1112 may provide an interface for apparatus 1100 to communicate via one or more networks. Apparatus 1100 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0253] For one embodiment, at least one of the processor(s) 1102 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be packaged together with the logic of one or more controllers of the control module 1104 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with the logic of one or more controllers of the control module 1104. For one embodiment, at least one of the processor(s) 1102 may be integrated on the same die with the logic of one or more controllers of the control module 1104 to form a system-on-chip (SoC).

[0254] In various embodiments, the apparatus 1100 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 1100 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 1100 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0255] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information corresponding to the camera equipment, etc. are stored in the memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0256] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0257] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0258] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, terminal device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flow chart and / or block diagram, and the combination of the process and / or box in the flow chart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flow chart and / or one box or multiple boxes in the block diagram.

[0259] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0260] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0261] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0262] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0263] The above is a detailed introduction to a training method and device for an image reconstruction model, an image reconstruction method and device, an electronic device and a machine-readable medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A training method for an image reconstruction model, characterized in that: The method comprises: Acquire training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; Inputting the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; Determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; updating the parameters of the image reconstruction model according to the loss information; The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information; The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining first gradient information corresponding to the first grayscale image, and determining second gradient information corresponding to the second grayscale image; and determining second loss information based on the first gradient information and the second gradient information. The method of converting the reconstructed RAW image into the first grayscale image and converting the second RAW image into the second grayscale image by using convolution processing includes: extracting RAW sub-images corresponding to four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image. The difference information includes: the absolute value of the difference between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image; determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image and the pixel value corresponding to the second RAW image includes: adjusting the ratio of the difference information to the pixel value corresponding to the second RAW image according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

2. The method according to claim 1, characterized in that The convolution kernel corresponding to the RAW sub-image is a 3×3 matrix, and the element values ​​of the matrix are determined according to the conversion coefficients corresponding to the positions and colors of the elements; the conversion coefficients are determined according to the conversion method from RGB image to grayscale image.

3. The method according to claim 2, characterized in that In the convolution kernel corresponding to the red channel, the element value corresponding to the red position is the red conversion coefficient, the element value corresponding to the blue position is the ratio of the blue conversion coefficient to 4, and the element value corresponding to the green position is the ratio of the green conversion coefficient to 4; In the convolution kernel corresponding to the green channel, the element value corresponding to the red position is the ratio of the red conversion coefficient to 2, the element value corresponding to the blue position is the ratio of the blue conversion coefficient to 2, and the element value corresponding to the green position is the distribution value corresponding to the green conversion coefficient; In the convolution kernel corresponding to the blue channel, the element value corresponding to the red position is the ratio of the red conversion coefficient to 4, the element value corresponding to the blue position is the blue conversion coefficient, and the element value corresponding to the green position is the ratio of the green conversion coefficient to 4.

4. The method according to any one of claims 1 to 3, characterized in that The obtaining of training data includes: Get a second RAW image from the dataset; Converting the second RAW image into channel images corresponding to the four color channels; Performing reduction processing on the channel images corresponding to the four color channels to obtain reduced channel images corresponding to the four color channels; The reduced channel images corresponding to the four color channels are fused to obtain a first RAW image.

5. An image reconstruction method, characterized in that: The method comprises: Receive the RAW image to be processed; Inputting the to-be-processed RAW image into the trained image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; Outputting the reconstructed RAW image; The training process of the image reconstruction model includes: acquiring training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information; The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information; The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining first gradient information corresponding to the first grayscale image, and determining second gradient information corresponding to the second grayscale image; and determining second loss information based on the first gradient information and the second gradient information. The method of converting the reconstructed RAW image into the first grayscale image and converting the second RAW image into the second grayscale image by using convolution processing includes: extracting RAW sub-images corresponding to four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image. The difference information includes: the absolute value of the difference between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image; determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image and the pixel value corresponding to the second RAW image includes: adjusting the ratio of the difference information to the pixel value corresponding to the second RAW image according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

6. A training device for an image reconstruction model, characterized in that: The device comprises: An acquisition module is configured to acquire training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; a model processing module, configured to input the first RAW image into an image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; A loss determination module, configured to determine loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; A parameter updating module, configured to update the parameters of the image reconstruction model according to the loss information; The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information; The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining first gradient information corresponding to the first grayscale image, and determining second gradient information corresponding to the second grayscale image; and determining second loss information based on the first gradient information and the second gradient information. The method of converting the reconstructed RAW image into the first grayscale image and converting the second RAW image into the second grayscale image by using convolution processing includes: extracting RAW sub-images corresponding to four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image. The difference information includes: the absolute value of the difference between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image; determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image and the pixel value corresponding to the second RAW image includes: adjusting the ratio of the difference information to the pixel value corresponding to the second RAW image according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

7. An image reconstruction device, characterized in that: The device comprises: A receiving module, configured to receive a RAW image to be processed; a model processing module, configured to input the RAW image to be processed into a trained image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; An output module, configured to output the reconstructed RAW image; The training process of the image reconstruction model includes: acquiring training data; the training data includes: a first RAW image and a second RAW image; a first resolution corresponding to the first RAW image is less than or equal to a second resolution corresponding to the second RAW image; inputting the first RAW image into the image reconstruction model to obtain a reconstructed RAW image output by the image reconstruction model; determining loss information based on the reconstructed RAW image and the second RAW image; the loss information includes: first loss information and / or second loss information; and updating parameters of the image reconstruction model based on the loss information; The process of determining the first loss information includes: determining weight information based on difference information between pixel values ​​corresponding to the reconstructed RAW image and the second RAW image, and pixel values ​​corresponding to the second RAW image; and determining the first loss information based on the weight information; The process of determining the second loss information includes: using convolution processing to convert the reconstructed RAW image into a first grayscale image, and converting the second RAW image into a second grayscale image; determining first gradient information corresponding to the first grayscale image, and determining second gradient information corresponding to the second grayscale image; and determining second loss information based on the first gradient information and the second gradient information. The method of converting the reconstructed RAW image into the first grayscale image and converting the second RAW image into the second grayscale image by using convolution processing includes: extracting RAW sub-images corresponding to four color channels from the reconstructed RAW image and the second RAW image; performing convolution processing on the RAW sub-images corresponding to the four color channels to obtain RAW convolution result images corresponding to the four color channels; and fusing the RAW convolution result images corresponding to the four color channels to obtain the first grayscale image or the second grayscale image. The difference information includes: the absolute value of the difference between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image; determining the weight information based on the difference information between the pixel values ​​corresponding to the reconstructed RAW image and the second RAW image and the pixel value corresponding to the second RAW image includes: adjusting the ratio of the difference information to the pixel value corresponding to the second RAW image according to a preset range to obtain an adjusted ratio; and processing the adjusted ratio using an exponential function to obtain the weight information.

Images