Image reconstruction method, device, equipment and storage medium

Abstract

The present application belongs to the field of digital image processing, and specifically relates to an image reconstruction method, apparatus, device and storage medium, which comprises: obtaining a pre-trained image reconstruction model, wherein a first network branch in the image reconstruction model is used to generate an initial reconstructed image based on first image data; a second network branch is used to extract a color correction coefficient from second image data; an output layer is used to adjust the initial reconstructed image using the color correction coefficient; the first image data and the second image data are input into the image reconstruction model to obtain a reconstructed image; the problem that a traditional image reconstruction model cannot restore image data collected in dark or bright areas in a wide dynamic scene, resulting in poor image reconstruction effect, can be solved; using an image sensor to collect the first image data to reconstruct an image can ensure that the reconstructed image does not lose details of the light and dark parts, and at the same time, using the second image data to extract the color correction coefficient for color adjustment can improve the sensory effect of the reconstructed image.

Description

[Technical field]

[0001] The present invention belongs to the field of digital image processing, and in particular relates to an image reconstruction method, device, equipment and storage medium. [Background technology]

[0002] Image reconstruction refers to the process of reconstructing the original image data collected by the image acquisition device into an image with better light and dark details.

[0003] The traditional image reconstruction method includes: obtaining the original image data collected by the image sensor, performing image signal processing on the original image data to obtain processed image data; inputting the processed image data into a pre-trained image reconstruction model to obtain a reconstructed image. The image reconstruction model is trained using sample images that have undergone image signal processing.

[0004] However, sample images that have undergone image signal processing usually lose information about the dark or bright areas of the scene. Therefore, the trained image reconstruction model cannot restore image data collected in the dark or bright areas of the wide dynamic scene, which leads to poor image reconstruction effects. [Summary of the invention]

[0005] The present application provides an image reconstruction method, apparatus, device and storage medium, which can solve the problem that the trained image reconstruction model cannot restore the image data collected in the dark or bright areas in the wide dynamic scene, resulting in poor image reconstruction effect. The present application provides the following technical solutions:

[0006] In a first aspect, an image reconstruction method is provided, the method comprising:

[0007] Acquire first image data and second image data, wherein the first image data refers to image data output by an image sensor, and the second image data refers to image data obtained after image signal processing is performed on the first image data;

[0008] Obtain a pre-trained image reconstruction model, the image reconstruction model comprising a first network branch, a second network branch, and an output layer connected to the first network branch and the second network branch respectively; the first network branch is used to generate an initial reconstructed image based on the first image data; the second network branch is used to extract a color correction coefficient from the second image data; the output layer is used to adjust the initial reconstructed image using the color correction coefficient;

[0009] The first image data and the second image data are input into the image reconstruction model to obtain a reconstructed image.

[0010] Optionally, the second network branch includes a first network sub-branch and a second network sub-branch respectively connected to the output layer;

[0011] The first network sub-branch is used to extract a global correction coefficient in the color correction coefficient from the second image data, and the global correction coefficient is used to perform global white balance correction on the initial reconstructed image;

[0012] The second network sub-branch is used to extract local correction coefficients in the color correction coefficients from the second image data, and the local correction coefficients are used to perform local color correction on the first initial reconstructed image.

[0013] Optionally, the first network sub-branch includes a convolutional layer, a rectified linear unit ReLU function, a global average pooling layer and a nonlinear activation function in sequence;

[0014] The second network sub-branch includes a convolutional layer, a ReLU function and a nonlinear activation function in sequence.

[0015] Optionally, obtaining a pre-trained image reconstruction model includes:

[0016] Acquire training data, the training data comprising: sample sensor data, processed data corresponding to the sample sensor data, and a standard image corresponding to the sample sensor data;

[0017] Inputting the sample sensing data and the processed data into a pre-created initial network model to obtain a first training result; the model structure of the initial network model is the same as the model structure of the image reconstruction model;

[0018] Inputting the first training result and the standard image into a preset loss function to obtain a first loss function value;

[0019] Iteratively train the initial network model using the first loss function value to obtain an intermediate network model;

[0020] determining a mask weight based on the processed data;

[0021] Inputting the sample sensing data and the processed data into the intermediate network model to obtain a second training result;

[0022] Inputting the second training result and the standard image into a preset first loss function to obtain a second loss function value;

[0023] Determine the product of the second loss function value and the mask weight to obtain a third loss function value;

[0024] The intermediate network model is iteratively trained using the third loss function value to obtain the image reconstruction model.

[0025] Optionally, the acquiring training data includes:

[0026] Acquire multiple sample sensor data collected in the same scene, and different sample sensor data have different exposures;

[0027] Performing image signal processing on each sample sensor data to obtain processed data corresponding to each sample sensor data;

[0028] The processed data with different exposures are subjected to image fusion to obtain the standard image.

[0029] Optionally, determining a mask weight based on the processed data comprises:

[0030] The mask weight is calculated based on the pixel value of the Y channel of the processed data.

[0031] Optionally, the loss function includes a first loss function, a second loss function and a third loss function;

[0032] The first loss function is used to calculate the difference in pixel values ​​between the training result and the standard image.

[0033] The second loss function is used to reduce the difference in visual perception between the training result and the standard image.

[0034] The third loss function is used to reduce the difference in structural similarity between the training result and the standard image.

[0035] In a second aspect, an image reconstruction device is provided, the device comprising:

[0036] An image acquisition module, used to acquire the first image data and the second image data, wherein the first image data refers to image data output by an image sensor, and the second image data refers to image data obtained after image signal processing is performed on the first image data;

[0037] A model acquisition module, used to acquire the pre-trained image reconstruction model, wherein the image reconstruction model includes a first network branch, a second network branch, and an output layer connected to the first network branch and the second network branch respectively; the first network branch is used to generate an initial reconstructed image based on the first image data; the second network branch is used to extract a color correction coefficient from the second image data; and the output layer is used to adjust the initial reconstructed image using the color correction coefficient;

[0038] The image input module is used to input the first image data and the second image data into the image reconstruction model to obtain a reconstructed image.

[0039] In a third aspect, an electronic device is provided, the device comprising a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the image reconstruction method as described in the first aspect.

[0040] In a fourth aspect, a computer-readable storage medium is provided, wherein a program is stored in the storage medium, and when the program is executed by a processor, the program is used to implement the image reconstruction method as described in any one of the first aspects.

[0041] The beneficial effects of the present application are: a reconstructed image is obtained by inputting the first image data and the second image data into a pre-trained image reconstruction model; the problem that the existing image reconstruction model cannot restore the image data collected in the dark area or the bright area in the wide dynamic scene, resulting in poor image reconstruction effect, can be solved; since the image reconstruction model can collect the first image data using the image sensor through the first network branch to reconstruct the image, it can be ensured that the reconstructed image does not lose the details of the light and dark parts; at the same time, although the second image data processed by the ISP system performs poorly in the light and dark areas of the scene, the overall color information and white balance are accurate relative to the first image data. Therefore, using the second image data to extract the color correction coefficient to adjust the initial reconstructed image can make the color of the reconstructed image more accurate and improve the sensory effect of the reconstructed image.

[0042] In addition, by setting the first network sub-branch and the second network sub-branch in the second network branch, the global correction coefficient is extracted through the first network sub-branch to perform global white balance correction on the initial reconstructed image; and the local correction coefficient is extracted through the second network sub-branch to perform local color correction on the initial reconstructed image, thereby improving the white balance and color display effects of the reconstructed image.

[0043] In addition, by using a global average pooling layer in the first network sub-branch instead of the traditional fully connected neural network, the model structure can be simplified and the calculation speed of the image reconstruction network can be improved.

[0044] In addition, an image reconstruction model is obtained through two stages of training. In the first stage, the model's ability to reconstruct colors is trained, and in the second stage, the model's ability to reconstruct underexposed and overexposed areas is trained. This can improve the performance of the model in reconstructing image colors, underexposed and overexposed areas.

[0045] In addition, by determining the mask weights based on the processed data, the model can extract the mask weights that are suitable for the processed data, thereby improving the accuracy of model training.

[0046] In addition, since the image reconstruction model is trained and generated based on three loss functions, the combined loss function can narrow the difference in sensory naturalness between the training results and the standard images, and can solve the problem of unnatural sensory perception of the reconstructed images output by the image reconstruction model.

[0047] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0048] Figure 1 is a flow chart of an image reconstruction method provided by an embodiment of the present application;

[0049] Figure 2 is a structural diagram of an image reconstruction model provided by an embodiment of the present application;

[0050] Figure 3 is a flow chart of an image reconstruction model training method provided by an embodiment of the present application;

[0051] Figure 4 This is an image reconstruction model training process provided by an embodiment of the present application;

[0052] Figure 5 is a block diagram of an image reconstruction device provided by an embodiment of the present application;

[0053] Figure 6 It is a block diagram of an electronic device provided by an embodiment of the present application. [Specific implementation method]

[0054] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] In the application, unless otherwise specified, directional words such as "up, down, top, bottom" are usually used with reference to the directions shown in the drawings, or with reference to the components themselves in the vertical, perpendicular or gravity directions; similarly, for ease of understanding and description, "inside and outside" refer to the inside and outside relative to the outline of each component itself, but the above directional words are not used to limit the present application.

[0057] First, several terms involved in the embodiments of the present application are introduced.

[0058] Image Signal Processor (ISP): used to process the image data received by the image sensor in real time and convert the image data into RGB images.

[0059] Raw image file (RAW Image Format, RAW): refers to the unprocessed original image file collected by the image sensor. RAW files have higher image quality because they retain the original data.

[0060] The image reconstruction method provided by this application is introduced in detail below.

[0061] Optionally, the executor of the image reconstruction method provided in the present application is an electronic device, which may be a terminal such as a computer, a mobile phone, a tablet computer, a camera, or a server. This embodiment does not limit the implementation method of the electronic device.

[0062] This embodiment provides an image reconstruction method, such as Figure 1 As shown, the method includes at least the following steps:

[0063] Step 101, acquiring first image data and second image data, where the first image data refers to image data output by an image sensor, and the second image data refers to image data obtained after image signal processing is performed on the first image data;

[0064] In this embodiment, the first image data may be RAW image data. The second image data is obtained by performing image signal processing on the first image data using an ISP system.

[0065] Specifically, the ISP system is connected to the image sensor, and after the image sensor collects the first image data, the first image data is sent to the ISP system. Correspondingly, the ISP system receives the first image data output by the image sensor, and performs image signal processing on the first image data to obtain the second image data.

[0066] The ISP system and the image sensor are respectively connected to the electronic device, so that the electronic device can obtain the first image data and the second image data.

[0067] Optionally, the ISP system and the image sensor may be arranged in the electronic device, or may be communicatively connected to the electronic device as devices independent of the electronic device. This embodiment does not limit the relationship between the ISP system and the image sensor and the electronic device.

[0068] Optionally, the image sensor may be a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS), etc. This embodiment does not limit the type of the image sensor.

[0069] Step 102, obtaining a pre-trained image reconstruction model, the image reconstruction model includes a first network branch, a second network branch, and an output layer connected to the first network branch and the second network branch respectively; the first network branch is used to generate an initial reconstructed image based on the first image data; the second network branch is used to extract a color correction coefficient from the second image data; the output layer is used to adjust the initial reconstructed image using the color correction coefficient.

[0070] refer to Figure 2 In the image reconstruction model shown, the input of the first network branch 21 is the first image data, so that the first network branch 21 generates an initial reconstructed image based on the first image data; the input of the second network branch 22 is the second image data, so that the second network branch 22 extracts the color correction coefficient from the second image data; the output layer 23 uses the color correction coefficient to adjust the initial reconstructed image to obtain a reconstructed image.

[0071] Since the first network branch 21 uses an image sensor to collect the first image data to reconstruct the image, it can be ensured that the reconstructed image does not lose the details of the light and dark parts. At the same time, although the second image data processed by the ISP system does not perform well in the light and dark areas of the scene, the overall color information and white balance are accurate relative to the first image data. Therefore, using the second image data to extract the color correction coefficient to adjust the initial reconstructed image can make the color of the reconstructed image more accurate and improve the perceptual effect of the reconstructed image.

[0072] Optionally, the color correction coefficient includes a global correction coefficient and / or a local correction coefficient, wherein the global correction coefficient is used to perform global white balance correction on the initial reconstructed image, and the local correction coefficient is used to perform local color correction on the initial reconstructed image.

[0073] Figure 2In the example, when the color correction coefficient includes a global correction coefficient and a local correction coefficient, the second network branch 22 includes a first network sub-branch 221 and a second network sub-branch 222 respectively connected to the output layer. The first network sub-branch 221 is used to extract the global correction coefficient in the color correction coefficient from the second image data; the second network sub-branch 222 is used to extract the local correction coefficient in the color correction coefficient from the second image data. In actual implementation, the color correction coefficient may also include only one of the global correction coefficient and the local correction coefficient. This embodiment does not limit the implementation method of the color correction coefficient.

[0074] Schematically, the first network sub-branch 221 includes a convolutional layer, a rectified linear unit (ReLU) function global average pooling layer and a nonlinear activation function in sequence.

[0075] The convolution layer is used to extract feature maps from the input image. The convolution kernel size in the convolution layer can be 4X4, and the convolution kernel step size is 2. In actual implementation, the convolution kernel size and step size can also be other values. This embodiment does not limit the implementation method of the convolution layer. Optionally, the number of convolution layers can be one or more layers. This embodiment does not limit the number of convolution layers.

[0076] The ReLU function can improve the training speed of the image reconstruction model. Specifically, when the input value is less than zero, the output value is zero; when the input value is greater than or equal to zero, the output value is equal to the input value. When the input value is a positive number, the derivative is 1. Therefore, the ReLU function does not have the squeezing effect of the sigmoid function during back propagation.

[0077] The global average pooling layer is used to compress the feature map output by the convolutional layer. The global average pooling layer and the convolutional layer together replace the fully connected layer in the traditional convolutional neural network. This can reduce the network parameters in the image reconstruction model and improve the model calculation efficiency.

[0078] The non-linear activation function is used to perform classification tasks at the output. The non-linear activation function can be a sigmoid function, which can squeeze the input into the range of 0 to 1 to achieve the classification task.

[0079] In other embodiments, the network structure of the first network sub-branch 221 may also be other structures, as long as the first network sub-branch 221 can extract the global correction coefficient.

[0080] Schematically, the second network sub-branch 222 includes a convolution layer, a ReLU function, and a nonlinear activation function in sequence. The relevant description of the convolution layer, the ReLU function, and the nonlinear activation function is detailed in the relevant description of the first network sub-branch 221, and this embodiment will not be repeated here.

[0081] Optionally, refer to Figure 2 , a convolution layer and a pooling layer may also be included before the first network sub-branch 221 and the second network sub-branch 222 to pre-process the second image data. This embodiment does not limit the network structure of the second network branch.

[0082] Schematically, the first network branch 21 is constructed based on a residual network structure. Figure 2 , the residual network structure can be a Residual in Residual structure. Specifically, the residual network structure includes an input layer, a convolutional layer, a residual group, a fully connected layer, and an output layer.

[0083] The outermost layer of the Residual in Residual structure consists of multiple residual groups and a long skip connection to form coarse-grained residual learning. Accordingly, the long skip connection enables the residual network to learn residual information at a coarser level. Each residual group includes multiple residual channel attention blocks and a short skip connection. Accordingly, the short skip connection is a fine-grained identity-based skip connection, which allows a large amount of low-frequency information that is not required by the network to be filtered out.

[0084] In actual implementation, the first network branch 21 may also be other network structures as long as it can reconstruct an initial reconstructed image with good light and dark details according to the first image data.

[0085] Optionally, the image reconstruction model is pre-stored in the electronic device, or is obtained by training the electronic device. Specifically, the process of obtaining the image reconstruction model by training the electronic device is described in detail in Figure 3 The embodiment shown.

[0086] Step 103: input the first image data and the second image data into an image reconstruction model to obtain a reconstructed image.

[0087] Specifically, the first image data is input into the first network branch, and the second image data is input into the second network branch, and after calculation by the image reconstruction model, a reconstructed image corresponding to the first image data and the second image data is obtained.

[0088] In summary, the image reconstruction method provided in this embodiment obtains a reconstructed image by inputting the first image data and the second image data into a pre-trained image reconstruction model; it can solve the problem that the existing image reconstruction model cannot restore the image data collected in the dark area or the bright area in the wide dynamic scene, thereby resulting in poor image reconstruction effect; because the image reconstruction model can collect the first image data using the image sensor through the first network branch to reconstruct the image, it can be ensured that the reconstructed image does not lose the details of the light and dark parts; at the same time, although the second image data processed by the ISP system performs poorly in the light and dark areas of the scene, the overall color information and white balance are accurate relative to the first image data. Therefore, using the second image data to extract the color correction coefficient to adjust the initial reconstructed image can make the color of the reconstructed image more accurate, and can improve the sensory effect of the reconstructed image.

[0089] In addition, by setting the first network sub-branch and the second network sub-branch in the second network branch, the global correction coefficient is extracted through the first network sub-branch to perform global white balance correction on the initial reconstructed image; and the local correction coefficient is extracted through the second network sub-branch to perform local color correction on the initial reconstructed image, thereby improving the white balance and color display effects of the reconstructed image.

[0090] In addition, by using a global average pooling layer in the first network sub-branch instead of the traditional fully connected neural network, the model structure can be simplified and the calculation speed of the image reconstruction network can be improved.

[0091] refer to Figure 3 This embodiment provides a method for training an image reconstruction model, the method comprising at least the following steps:

[0092] Step 31, obtaining training data, the training data including: sample sensor data, processed data corresponding to the sample sensor data, and standard images corresponding to the sample sensor data.

[0093] Specifically, obtaining training data includes: obtaining multiple sample sensor data collected in the same scene; performing image signal processing on each sample sensor data to obtain processed data corresponding to each sample sensor data; and performing image fusion on processed data with different exposures to obtain a standard image.

[0094] The exposure of different sample sensor data is different. The training data may include sample sensor data collected in multiple scenes, and the sample sensor data collected in each scene includes sample sensor data with different exposures.

[0095] Optionally, different exposures of the sample sensing data are achieved by adjusting a shutter speed of the image sensor, and different shutter speeds correspond to different exposures.

[0096] In order to improve the accuracy of generating a standard image under the same scene, the gains of multiple sample sensor data collected under the same scene are the same. Schematically, the gains of multiple sample sensor data can be the gains in the automatic exposure mode; or the gains set by the user. This embodiment does not limit the setting method of the gains.

[0097] The manner of performing image signal processing on each piece of sample sensor data is the same as the manner of processing the first image data. Specifically, the ISP system is used to perform image signal processing on each piece of sample sensor data to obtain processed data corresponding to the sample sensor data.

[0098] Optionally, performing image fusion on processed data with different exposures to obtain a standard image includes: using a multi-exposure fusion technology to perform image fusion on processed data with different exposures to obtain a standard image. Specifically, the electronic device obtains multiple processed data with different exposures; uses the exposure value of each processed data to calculate the brightness mean weight and saturation weight of the processed data; calculates weighted data corresponding to the processed data according to the brightness mean weight and saturation weight of each processed data; and fuses the weighted data to obtain a standard image.

[0099] in,

[0100] The brightness mean weight is calculated by the following formula:

[0101]

[0102] Among them, W 1，k (i, j) is the brightness mean weight of the pixel in the i-th row and j-th column of the k-th processed data, m k The kth processed data needs to enhance the mean brightness of the area. is the adjustment coefficient, L wk (i, j) is the brightness value of the pixel in the i-th row and j-th column of the k-th processed data.

[0103] The saturation weight is expressed as follows:

[0104]

[0105] Among them, W 2，k (i, j) is the saturation weight of the pixel in the i-th row and j-th column of the k-th processed data, R(i, j) is the brightness value of the red sub-pixel in the pixel in the i-th row and j-th column of the data before processing, G(i, j) is the brightness value of the green sub-pixel in the pixel in the i-th row and j-th column of the data before processing, B(i, j) is the brightness value of the blue sub-pixel in the pixel in the i-th row and j-th column of the data before processing, M u is the average brightness value of the data before processing.

[0106] The weighted fusion process is expressed as follows:

[0107]

[0108] Among them, L Wcomb (i, j) is the brightness value of the pixel in the i-th row and j-th column of the fused image. It is the sum of the products of the fusion weight of the pixel in the i-th row and j-th column of the first processed data to the fusion weight of the pixel in the i-th row and j-th column of the k-th processed data and the brightness value of the pixel in the i-th row and j-th column of the first processed data to the brightness value of the pixel in the i-th row and j-th column of the k-th processed data. It is the sum of the fusion weights of the pixels in the i-th row and j-th column of the first processed data to the fusion weights of the pixels in the i-th row and j-th column of the k-th processed data.

[0109] Step 32, inputting the sample sensing data and the processed data into a pre-created initial network model to obtain a first training result; the model structure of the initial network model is the same as the model structure of the image reconstruction model.

[0110] The first training result is used to train the color and white balance accuracy of the sample sensing data.

[0111] Step 33: input the first training result and the standard image into a preset loss function to obtain a first loss function value.

[0112] Optionally, the preset loss function includes a first loss function, a second loss function and a third loss function. Among them, the first loss function is used to calculate the difference in pixel values ​​between the training result and the standard image. For example, the first loss function is a norm loss function (L1 LOSS). The second loss function is used to narrow the difference in visual perception between the training result and the standard image. For example, the second loss function can be a perceptual loss function (Visual Geometry Group Loss, VGGLoss). The third loss function is used to narrow the difference in structural similarity between the training result and the standard image. For example, the third loss function can be a structural loss function (Structural Similarity Loss, SSIM LOSS).

[0113] Accordingly, the preset loss function is expressed by the following formula:

[0114] Among them, L stage1 represents the loss function of the first stage of training, λ1, λ2, and λ3 are the training weights corresponding to the first loss function, the second loss function, and the third loss function, respectively.

[0115] In actual implementation, the preset loss function may also be one or two of the first loss function, the second loss function and the third loss function. This embodiment does not limit the implementation method of the preset loss function.

[0116] Step 34, iteratively training the initial network model using the first loss function value to obtain an intermediate network model;

[0117] Specifically, the initial network model is trained based on the first loss value to reduce the difference between the first training result and the standard image until the initial network model converges to obtain an intermediate network model.

[0118] Step 35, determining mask weights based on the processed data.

[0119] Due to the limitation of the image sensor's own performance, in wide dynamic scenes, the processed data may have overexposure in bright areas or too dark in dark areas. Therefore, in this embodiment, by using the brightness information of the processed data to extract masks of underexposed and overexposed areas, the training weights of bright and dark areas are increased, so that the image reconstruction model pays more attention to the learning of underexposed and overexposed areas, so that the trained image reconstruction model can restore the image data collected in bright and dark areas.

[0120] Specifically, determining the mask weight based on the processed data includes: calculating the mask weight based on the pixel value of the Y channel of the processed data.

[0121] Among them, the mask weight formula is:

[0122]

[0123] Among them, Thresh brght is the preset brightness threshold, Thresh dark is the preset darkness threshold, X i is the grayscale value of the Y channel of the i-th pixel in the processed data, γ1 and γ2 are preset weight values, γ1 and γ2 are the same or different, and this embodiment does not limit the values ​​of γ1 and γ2.

[0124] Step 36, input the sample sensing data and the processed data into the intermediate network model to obtain a second training result.

[0125] Step 37: input the second training result and the standard image into a preset first loss function to obtain a second loss function value.

[0126] Step 38, determine the product of the second loss function value and the mask weight to obtain a third loss function value.

[0127] Among them, the third loss function value is expressed by the following formula:

[0128]

[0129] Among them, w i is the mask weight, y is the output value of the image reconstruction model, is a standard image, i is the pixel position, and N is the total number of pixels.

[0130] Step 39: Use the third loss function value to iteratively train the intermediate network model to obtain an image reconstruction model.

[0131] In this step, the third loss function value is used to fine-tune the intermediate network model trained in the first stage, so that the training pays more attention to the information recovery of the underexposed and overexposed areas of the image, which can improve the performance of the model in reconstructing the underexposed and overexposed areas of the image.

[0132] Specifically, refer to Figure 4 , Figure 4 This is an embodiment of an image reconstruction model training process, and the specific steps are as follows:

[0133] The image sensor collects multiple RAW image data with different exposures (i.e., first image data), and processes the multiple different RAW image data through the ISP system to obtain multiple different RGB images (i.e., second image data); the multiple RGB images are fused into an SRGB image using an image fusion method, and the SRGB image is the standard image.

[0134] Then, the first stage of training is carried out. An initial network model is constructed. The initial network model has the same structure as the above-mentioned image reconstruction model. The first network branch is based on the Residual in Residual structure, and the second network branch includes a first network sub-branch for extracting the global correction coefficient and a second network sub-branch for extracting the local correction coefficient. The RAW image data is input into the first network branch to obtain the initial reconstructed image, and the RGB image is input into the second network branch to obtain the global correction coefficient and the local correction coefficient. The output layer uses the global correction coefficient and the local correction coefficient to adjust the initial reconstructed image to obtain the first reconstructed image. The first reconstructed image and the SRGB image are input into the loss function to obtain the first loss function value. The initial network model is trained based on the first loss function value to reduce the difference between the first reconstructed image and the SRGB image until the initial network model converges to obtain an intermediate network model.

[0135] After that, the second stage of training is carried out. The mask weight is calculated based on the pixel value of the Y channel of the RGB image; the RAW image data and the RGB image file are input into the intermediate network model to obtain the second reconstruction image, and the second reconstruction image and the SRGB image are input into the loss function to obtain the second loss function value; the product of the second loss function value and the mask weight is determined to obtain the third loss function value; the intermediate network model is iteratively trained using the third loss function value to obtain the image reconstruction model.

[0136] Optionally, during the training process, poor quality training data may be removed by manual screening, and the screened training data may be used for training to improve the performance of the trained image reconstruction model.

[0137] To summarize, in this embodiment, an image reconstruction model is obtained through two stages of training. In the first training stage, the model's ability to reconstruct colors is trained, and in the second training stage, the model's ability to reconstruct underexposed and overexposed areas is trained, which can improve the performance of the model in reconstructing image colors, underexposed and overexposed areas.

[0138] In addition, by determining the mask weights based on the processed data, the model can extract the mask weights that are suitable for the processed data, thereby improving the accuracy of model training.

[0139] In addition, since the image reconstruction model is trained and generated based on three loss functions, the combined loss function can narrow the difference in sensory naturalness between the training results and the standard images, and can solve the problem of unnatural sensory perception of the reconstructed images output by the image reconstruction model.

[0140] Figure 5 It is a block diagram of an image reconstruction device provided by an embodiment of the present application. The device includes at least the following modules: an image acquisition module 510, a model acquisition module 520 and an image input module 530.

[0141] An image acquisition module 510 is used to acquire first image data and second image data, where the first image data refers to image data output by an image sensor, and the second image data refers to image data obtained after image signal processing is performed on the first image data;

[0142] The model acquisition module 520 is used to acquire a pre-trained image reconstruction model, wherein the image reconstruction model includes a first network branch, a second network branch, and an output layer connected to the first network branch and the second network branch respectively; the first network branch is used to generate an initial reconstructed image based on the first image data; the second network branch is used to extract a color correction coefficient from the second image data; and the output layer is used to adjust the initial reconstructed image using the color correction coefficient.

[0143] The image input module 530 is used to input the first image data and the second image data into the image reconstruction model to obtain a reconstructed image.

[0144] For relevant details, refer to the above embodiments.

[0145] It should be noted that: the image reconstruction device provided in the above embodiment only uses the division of the above functional modules as an example when performing image reconstruction. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server access device is divided into different functional modules to complete all or part of the functions described above. In addition, the image reconstruction device provided in the above embodiment and the image reconstruction access method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0146] This embodiment provides an electronic device, such as Figure 6 As shown, the electronic device at least includes a processor 601 and a memory 602 .

[0147] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0148] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the image reconstruction method provided by the method embodiment of the present application.

[0149] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0150] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0151] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the image reconstruction method of the above method embodiment.

[0152] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image reconstruction method, characterized in that: The method comprises: Acquire first image data and second image data, wherein the first image data refers to image data output by an image sensor, and the second image data refers to image data obtained after image signal processing is performed on the first image data by an ISP system; Obtain a pre-trained image reconstruction model, the image reconstruction model comprising a first network branch, a second network branch, and an output layer connected to the first network branch and the second network branch respectively; the first network branch is used to generate an initial reconstructed image based on the first image data; the second network branch is used to extract a color correction coefficient from the second image data; the output layer is used to adjust the initial reconstructed image using the color correction coefficient; Inputting the first image data and the second image data into the image reconstruction model to obtain a reconstructed image; The obtaining of a pre-trained image reconstruction model comprises: Acquire training data, the training data comprising: sample sensor data, processed data corresponding to the sample sensor data, and a standard image corresponding to the sample sensor data; Inputting the sample sensing data and the processed data into a pre-created initial network model to obtain a first training result; the model structure of the initial network model is the same as the model structure of the image reconstruction model; Inputting the first training result and the standard image into a preset loss function to obtain a first loss function value; Iteratively train the initial network model using the first loss function value to obtain an intermediate network model; determining a mask weight based on the processed data; Inputting the sample sensing data and the processed data into the intermediate network model to obtain a second training result; Inputting the second training result and the standard image into a preset first loss function to obtain a second loss function value; Determine the product of the second loss function value and the mask weight to obtain a third loss function value; The intermediate network model is iteratively trained using the third loss function value to obtain the image reconstruction model.

2. The method according to claim 1, characterized in that The second network branch includes a first network sub-branch and a second network sub-branch respectively connected to the output layer; The first network sub-branch is used to extract a global correction coefficient in the color correction coefficient from the second image data, and the global correction coefficient is used to perform global white balance correction on the initial reconstructed image; The second network sub-branch is used to extract local correction coefficients in the color correction coefficients from the second image data, and the local correction coefficients are used to perform local color correction on the initial reconstructed image.

3. The method according to claim 2, characterized in that The first network sub-branch includes a convolutional layer, a rectified linear unit ReLU function, a global average pooling layer and a nonlinear activation function in sequence; The second network sub-branch includes a convolutional layer, a ReLU function and a nonlinear activation function in sequence.

4. The method according to claim 1, characterized in that: The obtaining of training data comprises: Acquire multiple sample sensor data collected in the same scene, and different sample sensor data have different exposures; Performing image signal processing on each sample sensor data to obtain processed data corresponding to each sample sensor data; The processed data with different exposures are subjected to image fusion to obtain the standard image.

5. The method according to claim 1, characterized in that The determining the mask weight based on the processed data comprises: The mask weight is calculated based on the pixel value of the Y channel of the processed data.

6. The method according to claim 1, characterized in that The loss function includes a first loss function, a second loss function and a third loss function; The first loss function is used to calculate the difference in pixel values ​​between the training result and the standard image; The second loss function is used to reduce the difference in visual perception between the training result and the standard image; The third loss function is used to reduce the difference in structural similarity between the training result and the standard image.

7. An image reconstruction device using the method according to any one of claims 1 to 6, characterized in that: The device comprises: An image acquisition module, used to acquire first image data and second image data, wherein the first image data refers to image data output by an image sensor, and the second image data refers to image data obtained after image signal processing is performed on the first image data; A model acquisition module, used to acquire a pre-trained image reconstruction model, wherein the image reconstruction model includes a first network branch, a second network branch, and an output layer connected to the first network branch and the second network branch respectively; the first network branch is used to generate an initial reconstructed image based on the first image data; the second network branch is used to extract a color correction coefficient from the second image data; and the output layer is used to adjust the initial reconstructed image using the color correction coefficient; The image input module is used to input the first image data and the second image data into the image reconstruction model to obtain a reconstructed image.

8. An electronic device, characterized in that: The device comprises a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the image reconstruction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, and when the program is executed by the processor, it is used to implement the image reconstruction method according to any one of claims 1 to 6.

Images