High-quality image reconstruction method

Abstract

The invention discloses a high-quality image reconstruction method, and the method comprises the steps: firstly constructing a convolutional neural network model which comprises an input layer, a convolutional layer, a pooling layer, a full connection layer, an output layer, a brightness enhancement module, a saturation adjustment module, and a color cast correction module; acquiring night vision image data, performing normalization and noise reduction processing on the night vision image data, dividing the night vision image data into a training set and a test set, training the convolutional neural network model by adopting the training set, and taking a mean square error (MSE) as a loss function; a test set is adopted to test and analyze the trained convolutional neural network model, and PSNR (peak signal-to-noise ratio) and structural similarity (SSIM) are adopted to evaluate the quality of the restored image, so that an optimal convolutional neural network model is obtained; and performing high-quality image reconstruction processing on the CIS image data by using the obtained optimal convolutional neural network model to obtain a high-quality image, and transmitting the high-quality image to a display device.

Description

Technical Field

[0001] The invention belongs to the technical field of image data processing, and in particular relates to a high-quality image reconstruction method. Background Art

[0002] CIS is the abbreviation of "Contact Image Sensor", which is a sensor technology used to scan documents and images. CIS is different from the CCD (charge coupled device) or CMOS (complementary metal oxide semiconductor) image sensors used in traditional flatbed scanners. CIS sensors are usually thinner, lighter, and can operate with lower power consumption. However, under extremely weak light conditions, the image data (RAW data) output by the CIS sensor will have problems such as insufficient brightness, blur, low contrast, low saturation, and color cast. In order to obtain high-quality video images, the image data needs to be further twisted for high-quality image reconstruction. For example, use histogram equalization or adaptive histogram equalization to increase the brightness of the image and enhance dark details; use sharpening filters such as Laplacian or Sobel operators to enhance edges and details; use contrast enhancement algorithms such as linear stretching or nonlinear stretching to expand the dynamic range of the image; adjust the saturation component separately through color space conversion (such as from RGB to HSL / HSV). High-quality image reconstruction requires the combination of multiple algorithms to achieve the best image enhancement effect, which makes the constructed algorithm model more complex, increases the image processing time, and affects the real-time performance of the image. Therefore, one of the current difficulties is how to design a suitable algorithm to build and optimize a high-quality image reconstruction model. In addition, with the development of artificial intelligence technology, deep learning-based methods have been widely used in video image processing. How to apply deep learning methods to high-quality image reconstruction is also one of the research focuses. Summary of the invention

[0003] In view of the above-mentioned shortcomings, the present invention discloses a high-quality image reconstruction method, which constructs a deep neural network model through a large amount of data learning and algorithm optimization, and uses it for high-quality image reconstruction of night vision image data, which can effectively solve the problems of insufficient brightness, blur, low contrast, low saturation, color cast, etc. in CIS image data.

[0004] The present invention is achieved by adopting the following technical solutions:

[0005] A high-quality image reconstruction method comprises the following steps:

[0006] (1) constructing a convolutional neural network model, the convolutional neural network model comprising an input layer, a convolutional layer, a pooling layer, a fully connected layer, an output layer, a brightness enhancement module, a saturation adjustment module, and a color cast correction module; the input layer is used to receive image data and convert it into a YCbCr color space to obtain three components of brightness Y, chroma Cb, and chroma Cr; the convolutional layer extracts features of the three components of brightness Y, chroma Cb, and chroma Cr to obtain a feature map, which includes a plurality of convolution kernels;

[0007] The brightness enhancement module is used to extract the brightness feature in the brightness Y for processing to obtain an enhanced brightness feature map. The brightness enhancement module uses logarithmic transformation to process the brightness feature. The logarithmic transformation formula is: s=c·log(1+r), where s is the pixel value of the Y channel after transformation, r is the pixel value of the original Y channel, and c is a constant;

[0008] The saturation adjustment module is used to extract features from chroma Cb and chroma Cr to perform hue conversion processing to obtain a saturation adjustment feature map after saturation adjustment. The hue conversion formula is: hue(x)=x+Δhue;

[0009] The color cast correction module is used to extract the edge, texture and color distribution features in the chromaticity CB and the chromaticity Cr for processing to obtain a color correction feature map; the color cast correction module includes several convolution layers, pooling layers and activation functions;

[0010] The fully connected layer fuses the obtained brightness feature map, saturation adjustment feature map and color correction feature map with the feature map to obtain the adjusted brightness Y, chroma Cb and chroma Cr three components, and then converts them into RGB color space through the output layer to obtain processed image data;

[0011] (2) collecting night vision image data, normalizing and denoising the data, and dividing the data into a training set and a test set. Then, the training set is used to train the convolutional neural network model in step (1), and the mean square error (MSE) is used as the loss function. The trained convolutional neural network model is tested and analyzed using the test set, wherein PSNR (peak signal-to-noise ratio) and structural similarity (SSIM) are used to evaluate the quality of the restored image, thereby obtaining the optimal convolutional neural network model.

[0012] (3) Applying the optimal convolutional neural network model obtained in step (2) to perform high-quality image reconstruction processing on the CIS image data, wherein the CIS image data is first normalized and denoised as described in step (2), and then processed using the convolutional neural network model to obtain a high-quality image, which is then transmitted to a display device.

[0013] Furthermore, in step (1), the convolutional layer uses the ReLU activation function; the activation function of the fully connected layer uses the Softplus function, whose formula is f(x)=log(1+e x ); The activation function of the output layer adopts the Softmax function, whose formula is f i (X) = e Xi / ∑ j e Xj , where i and j traverse each output unit.

[0014] Furthermore, the pooling layer in step (1) adopts a maximum pooling operation;

[0015] The maximum pooling operation is: max_pool(I,k,s)={max(I i,j ),max(I i+k / 2,j ),max(I i,j+k / 2 ),max(I i+k / 2,j+k / 2 )}, where I is the input feature map, K is the size of the pooling window and is 3×3, and s is the stride and is 1.

[0016] Furthermore, in step (2), the ratio of the training set to the test set is 8:2.

[0017] Furthermore, in step (2), during the training of the convolutional neural network model in step (1), a gradient descent optimization algorithm is used to update the network weights to minimize the loss function.

[0018] Furthermore, in step (2), the normalization is any one of batch normalization, instance normalization and group normalization.

[0019] Furthermore, in step (2), the noise reduction process adopts any one of median filtering, mean filtering and Gaussian filtering.

[0020] Compared with the prior art, this technical solution has the following beneficial effects:

[0021] 1. The present invention constructs a convolutional neural network model to reconstruct image data, solves the problems of insufficient brightness, blur, low contrast, low saturation, color cast, etc. in CIS image data, and thus obtains high-quality images. The present invention converts the collected image data into a YCbCr color space to obtain three components of brightness Y, chroma Cb and chroma Cr, and then uses a convolutional neural network model to extract features in the three components respectively, and uses a specific brightness enhancement module, a saturation adjustment module, and a color cast correction module to further process the extracted features respectively, such as the brightness enhancement module is used to extract the brightness feature in the brightness Y for processing to obtain an enhanced brightness feature map, the saturation adjustment module is used to extract the features in the chroma Cb and the chroma Cr for hue conversion processing to obtain a saturation adjustment feature map after saturation adjustment, and the color cast correction module is used to extract the edge, texture and color distribution of the chroma CB and the chroma Cr for processing to obtain a color correction feature map; then the obtained features are fused to obtain the adjusted three components of brightness Y, chroma Cb and chroma Cr, and then converted into an RGB color space to obtain processed image data, and the image data is processed by each module in a targeted manner to solve the problems of insufficient brightness, blur, low contrast, low saturation, color cast, etc. of the image data, and the processing time of the model can be improved to solve the problem of insufficient real-time performance of the image.

[0022] 2. The present invention also normalizes and de-noises the image data, making the image clearer, more detailed, and cleaner and more consistent, which helps to improve the interpretability of the reconstruction results. At the same time, since the image reconstruction algorithm may amplify the noise and errors in the input data, resulting in artifacts in the output image, these errors and artifacts can be reduced by performing de-noising in advance, thereby improving the accuracy of reconstruction. In addition, normalization and de-noising can reduce the interference of erroneous signals, thereby accelerating the convergence speed of the algorithm, reducing the number of iterations and the amount of data that needs to be processed during the reconstruction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of the high-quality image reconstruction method described in Example 1. DETAILED DESCRIPTION

[0024] The present invention is further described by the following examples, but is not intended to be limiting of the present invention. Specific experimental conditions and methods not specified in the following examples are conventional methods commonly known to those skilled in the art.

[0025] Embodiment 1: A high-quality image reconstruction method, comprising the following steps:

[0026] (1) constructing a convolutional neural network model, the convolutional neural network model comprising an input layer, a convolutional layer, a pooling layer, a fully connected layer, an output layer, a brightness enhancement module, a saturation adjustment module, and a color cast correction module; the input layer is used to receive image data and convert it into a YCbCr color space to obtain three components of brightness Y, chroma Cb, and chroma Cr; the convolutional layer extracts features of the three components of brightness Y, chroma Cb, and chroma Cr to obtain a feature map, which includes a plurality of convolution kernels;

[0027] The brightness enhancement module is used to extract the brightness feature in the brightness Y for processing to obtain an enhanced brightness feature map. The brightness enhancement module uses logarithmic transformation to process the brightness feature. The logarithmic transformation formula is: s=c·log(1+r), where s is the pixel value of the Y channel after transformation, r is the pixel value of the original Y channel, and c is a constant;

[0028] The saturation adjustment module is used to extract features from chroma Cb and chroma Cr to perform hue conversion processing to obtain a saturation adjustment feature map after saturation adjustment. The hue conversion formula is: hue(x)=x+Δhue;

[0029] The color cast correction module is used to extract the edge, texture and color distribution features in the chromaticity CB and the chromaticity Cr for processing to obtain a color correction feature map; the color cast correction module includes several convolution layers, pooling layers and activation functions;

[0030] The fully connected layer fuses the obtained brightness feature map, saturation adjustment feature map and color correction feature map with the feature map to obtain the adjusted brightness Y, chroma Cb and chroma Cr three components, and then converts them into RGB color space through the output layer to obtain processed image data;

[0031] The convolutional layer uses the ReLU activation function; the activation function of the fully connected layer uses the Softplus function, whose formula is f(x)=log(1+e x ); The activation function of the output layer adopts the Softmax function, whose formula is f i (X) = e Xi / ∑ j e Xj , where i and j traverse each output unit; the pooling layer adopts the maximum pooling operation;

[0032] The maximum pooling operation is: max_pool(I,k,s)={max(I i,j ),max(I i+k / 2,j ),max(I i,j+k / 2 ),max(I i+k / 2,j+k / 2)}, where I is the input feature map, K is the size of the pooling window and is 3×3, and s is the step size and is 1;

[0033] (2) collecting night vision image data, normalizing and denoising the data, and dividing the data into a training set and a test set, wherein the ratio of the training set to the test set is 8:2, and then using the training set to train the convolutional neural network model in step (1), using mean square error (MSE) as a loss function; using the test set to test and analyze the trained convolutional neural network model, wherein PSNR (peak signal-to-noise ratio) and structural similarity (SSIM) are used to evaluate the quality of the restored image, thereby obtaining the optimal convolutional neural network model;

[0034] During the training of the convolutional neural network model in step (1), a gradient descent optimization algorithm is used to update the network weights to minimize the loss function;

[0035] The normalization is performed by batch normalization; the noise reduction process is performed by median filtering;

[0036] (3) Applying the optimal convolutional neural network model obtained in step (2) to perform high-quality image reconstruction processing on the CIS image data, wherein the CIS image data is first normalized and denoised as described in step (2), and then processed using the convolutional neural network model to obtain a high-quality image, which is then transmitted to a display device.

[0037] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 40dB.

[0038] Embodiment 2: A high-quality image reconstruction method comprising the following steps:

[0039] (1) constructing a convolutional neural network model, the convolutional neural network model comprising an input layer, a convolutional layer, a pooling layer, a fully connected layer, an output layer, a brightness enhancement module, a saturation adjustment module, and a color cast correction module; the input layer is used to receive image data and convert it into a YCbCr color space to obtain three components of brightness Y, chroma Cb, and chroma Cr; the convolutional layer extracts features of the three components of brightness Y, chroma Cb, and chroma Cr to obtain a feature map, which includes a plurality of convolution kernels;

[0040] The brightness enhancement module is used to extract the brightness feature in the brightness Y for processing to obtain an enhanced brightness feature map. The brightness enhancement module uses logarithmic transformation to process the brightness feature. The logarithmic transformation formula is: s=c·log(1+r), where s is the pixel value of the Y channel after transformation, r is the pixel value of the original Y channel, and c is a constant;

[0041] The saturation adjustment module is used to extract features from chroma Cb and chroma Cr to perform hue conversion processing to obtain a saturation adjustment feature map after saturation adjustment. The hue conversion formula is: hue(x)=x+Δhue;

[0042] The color cast correction module is used to extract the edge, texture and color distribution features in the chromaticity CB and the chromaticity Cr for processing to obtain a color correction feature map; the color cast correction module includes several convolution layers, pooling layers and activation functions;

[0043] The fully connected layer fuses the obtained brightness feature map, saturation adjustment feature map and color correction feature map with the feature map to obtain the adjusted brightness Y, chroma Cb and chroma Cr three components, and then converts them into RGB color space through the output layer to obtain processed image data;

[0044] The convolutional layer uses the ReLU activation function; the activation function of the fully connected layer uses the Softplus function, whose formula is f(x)=log(1+e x ); The activation function of the output layer adopts the Softmax function, whose formula is f i (X) = e Xi / ∑ j e Xj , where i and j traverse each output unit; the pooling layer adopts the maximum pooling operation;

[0045] The maximum pooling operation is: max_pool(I,k,s)={max(I i,j ),max(I i+k / 2,j ),max(I i,j+k / 2 ),max(I i+k / 2,j+k / 2 )}, where I is the input feature map, K is the size of the pooling window and is 3×3, and s is the step size and is 1;

[0046] (2) collecting night vision image data, normalizing and denoising the data, and dividing the data into a training set and a test set, wherein the ratio of the training set to the test set is 8:2, and then using the training set to train the convolutional neural network model in step (1), using mean square error (MSE) as a loss function; using the test set to test and analyze the trained convolutional neural network model, wherein PSNR (peak signal-to-noise ratio) and structural similarity (SSIM) are used to evaluate the quality of the restored image, thereby obtaining the optimal convolutional neural network model;

[0047] During the training of the convolutional neural network model in step (1), a gradient descent optimization algorithm is used to update the network weights to minimize the loss function;

[0048] The normalization is performed by using instance normalization; the noise reduction process is performed by using mean filtering;

[0049] (3) Applying the optimal convolutional neural network model obtained in step (2) to perform high-quality image reconstruction processing on the CIS image data, wherein the CIS image data is first normalized and denoised as described in step (2), and then processed using the convolutional neural network model to obtain a high-quality image, which is then transmitted to a display device.

[0050] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 40dB.

[0051] Embodiment 3: A high-quality image reconstruction method comprising the following steps:

[0052] (1) constructing a convolutional neural network model, the convolutional neural network model comprising an input layer, a convolutional layer, a pooling layer, a fully connected layer, an output layer, a brightness enhancement module, a saturation adjustment module, and a color cast correction module; the input layer is used to receive image data and convert it into a YCbCr color space to obtain three components of brightness Y, chroma Cb, and chroma Cr; the convolutional layer extracts features of the three components of brightness Y, chroma Cb, and chroma Cr to obtain a feature map, which includes a plurality of convolution kernels;

[0053] The brightness enhancement module is used to extract the brightness feature in the brightness Y for processing to obtain an enhanced brightness feature map. The brightness enhancement module uses logarithmic transformation to process the brightness feature. The logarithmic transformation formula is: s=c·log(1+r), where s is the pixel value of the Y channel after transformation, r is the pixel value of the original Y channel, and c is a constant;

[0054] The saturation adjustment module is used to extract features from chroma Cb and chroma Cr to perform hue conversion processing to obtain a saturation adjustment feature map after saturation adjustment. The hue conversion formula is: hue(x)=x+Δhue;

[0055] The color cast correction module is used to extract the edge, texture and color distribution features in the chromaticity CB and the chromaticity Cr for processing to obtain a color correction feature map; the color cast correction module includes several convolution layers, pooling layers and activation functions;

[0056] The fully connected layer fuses the obtained brightness feature map, saturation adjustment feature map and color correction feature map with the feature map to obtain the adjusted brightness Y, chroma Cb and chroma Cr three components, and then converts them into RGB color space through the output layer to obtain processed image data;

[0057] The convolutional layer uses the ReLU activation function; the activation function of the fully connected layer uses the Softplus function, whose formula is f(x)=log(1+e x ); The activation function of the output layer adopts the Softmax function, whose formula is f i (X) = e Xi / ∑ j e Xj , where i and j traverse each output unit; the pooling layer adopts the maximum pooling operation;

[0058] The maximum pooling operation is: max_pool(I,k,s)={max(I i,j ),max(I i+k / 2,j ),max(I i,j+k / 2 ),max(I i+k / 2,j+k / 2 )}, where I is the input feature map, K is the size of the pooling window and is 3×3, and s is the step size and is 1;

[0059] (2) collecting night vision image data, normalizing and denoising the data, and dividing the data into a training set and a test set, wherein the ratio of the training set to the test set is 8:2, and then using the training set to train the convolutional neural network model in step (1), using mean square error (MSE) as a loss function; using the test set to test and analyze the trained convolutional neural network model, wherein PSNR (peak signal-to-noise ratio) and structural similarity (SSIM) are used to evaluate the quality of the restored image, thereby obtaining the optimal convolutional neural network model;

[0060] During the training of the convolutional neural network model in step (1), a gradient descent optimization algorithm is used to update the network weights to minimize the loss function;

[0061] The normalization is performed by group normalization; the noise reduction process is performed by Gaussian filtering;

[0062] (3) Applying the optimal convolutional neural network model obtained in step (2) to perform high-quality image reconstruction processing on the CIS image data, wherein the CIS image data is first normalized and denoised as described in step (2), and then processed using the convolutional neural network model to obtain a high-quality image, which is then transmitted to a display device.

[0063] The signal-to-noise ratio of the night vision video image processed by the method described in this embodiment can reach more than 40dB.

[0064] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A high-quality image reconstruction method, characterized in that: The following steps are involved: (1) constructing a convolutional neural network model, the convolutional neural network model comprising an input layer, a convolutional layer, a pooling layer, a fully connected layer, an output layer, a brightness enhancement module, a saturation adjustment module, and a color cast correction module; the input layer is used to receive image data and convert it into a YCbCr color space to obtain three components of brightness Y, chroma Cb, and chroma Cr; the convolutional layer extracts features of the three components of brightness Y, chroma Cb, and chroma Cr to obtain a feature map, which includes a plurality of convolution kernels; The brightness enhancement module is used to extract the brightness feature in the brightness Y for processing to obtain an enhanced brightness feature map. The brightness enhancement module uses logarithmic transformation to process the brightness feature. The logarithmic transformation formula is: s=c·log(1+r), where s is the pixel value of the Y channel after transformation, r is the pixel value of the original Y channel, and c is a constant; The saturation adjustment module is used to extract features from chroma Cb and chroma Cr to perform hue conversion processing to obtain a saturation adjustment feature map after saturation adjustment. The hue conversion formula is: hue(x)=x+Δhue; The color cast correction module is used to extract the edge, texture and color distribution features in the chromaticity CB and the chromaticity Cr for processing to obtain a color correction feature map; the color cast correction module includes several convolution layers, pooling layers and activation functions; The fully connected layer fuses the obtained brightness feature map, saturation adjustment feature map and color correction feature map with the feature map to obtain the adjusted brightness Y, chroma Cb and chroma Cr three components, and then converts them into RGB color space through the output layer to obtain processed image data; (2) collecting night vision image data, normalizing and denoising the data, and dividing the data into a training set and a test set, then using the training set to train the convolutional neural network model in step (1), using mean square error as a loss function; using the test set to test and analyze the trained convolutional neural network model, wherein PSNR and structural similarity are used to evaluate the quality of the restored image, thereby obtaining an optimal convolutional neural network model; (3) Applying the optimal convolutional neural network model obtained in step (2) to perform high-quality image reconstruction processing on the CIS image data, wherein the CIS image data is first normalized and denoised as described in step (2), and then processed using the convolutional neural network model to obtain a high-quality image, which is then transmitted to a display device.

2. The high-quality image reconstruction method according to claim 1, characterized in that: In step (1), the convolutional layer uses the ReLU activation function; the activation function of the fully connected layer uses the Softplus function, whose formula is f(x)=log(1+e x ); The activation function of the output layer adopts the Softmax function, whose formula is f i (X) = e Xi / ∑ j e Xj , where i and j traverse each output unit.

3. The high-quality image reconstruction method according to claim 1, characterized in that: The pooling layer in step (1) adopts a maximum pooling operation; The maximum pooling operation is: max_pool(I,k,s)={max(I i,j ),max(I i+k / 2,j ),max(I i,j+k / 2 ),max(I i+k / 2,j+k / 2 )}, where I is the input feature map, K is the size of the pooling window and is 3×3, and s is the stride and is 1.

4. The high-quality image reconstruction method according to claim 1, characterized in that: In step (2), the ratio of the training set to the test set is 8:

2.

5. The high-quality image reconstruction method according to claim 1, characterized in that: In step (2), during the training process of the convolutional neural network model described in step (1), a gradient descent optimization algorithm is used to update the network weights to minimize the loss function.

6. The high-quality image reconstruction method according to claim 1, characterized in that: In step (2), the normalization is any one of batch normalization, instance normalization and group normalization.

7. The high-quality image reconstruction method according to claim 1, characterized in that: In step (2), the noise reduction process is performed by using any one of median filtering, mean filtering and Gaussian filtering.

Images