Checkerboard image restoration method based on deep neural network

Abstract

The present invention proposes a checkerboard image restoration method based on a deep neural network. First, the original image in the data set is preprocessed by chessboard pixel division to form a training sample. Afterwards, it is trained in a proposed deep neural network to complete the checkerboard image and generate a restored image. After the training is completed, the high-definition image can be automatically restored by simply inputting the checkerboard image into the pre-trained model. The deep neural network eliminated by the present invention mainly adopts a convolution block series strategy, and combines residual connections and dense connections to obtain high-dimensional image features, which are used to enrich the texture information required to complete the image. The present invention can not only realize the prediction and restoration of checkerboard images, but also be used in the field of reversible information hiding technology to improve embedding efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision, and specifically relates to a checkerboard image restoration method based on a deep learning network. The deep learning network is trained on a large number of data sets, and the learned features are used to restore high-quality images. Background Art

[0002] With the rapid development of artificial intelligence (AI), an increasing number of applications have become integrated into our daily lives, such as facial recognition, natural language processing, and autonomous driving. It can be said that AI is at the forefront of global change. Deep learning, a new and versatile class of AI algorithms, learns to solve complex tasks by observing large amounts of data, and has brought breakthroughs to many branches of science. Deep learning is based on deep neural networks, which, through multiple layers of artificial neurons, can discover complex structures in high-dimensional data. Using deep learning to solve multimedia problems, particularly in computer vision, has become a significant research hotspot. While deep learning has been widely applied in computer vision and image processing, many research topics remain to be explored.

[0003] A checkerboard image is an image in which the pixels of the image are divided into two categories according to the parity property of the sum of their coordinates, similar to the black and white squares on a chessboard. Afterwards, all white area pixels are used to predict the black area pixels to achieve image recovery. In recent years, checkerboard pixel prediction and image recovery have been applied to reversible information hiding. The error between the predicted pixels and the original pixels is used to achieve information hiding by shifting the prediction error histogram [2]. In addition, checkerboard pixel prediction can also be used to obtain a high-definition magnified image of an image, that is, a small-size image is sequentially expanded into a checkerboard-shaped double-magnified image, and then the surrounding pixels are used to predict and restore the image, thereby generating a double-magnified high-definition image.

[0004] In traditional checkerboard pixel prediction methods, the white pixels surrounding the black pixels are mainly used to predict the average value, gradient value, etc. Among them, the most classic method is to calculate the average value of the four surrounding white pixels as the predicted pixel value. Other related methods such as nearest-neighbor interpolation [3], median-edge detector [4], gradient adjustment [5] and bilinear interpolation [6] have also been proposed to improve the accuracy of prediction. Although these traditional methods using surrounding pixel prediction can achieve good prediction results in smooth areas of the image, they often cannot accurately predict areas with complex textures. Therefore, in recent years, methods using deep learning for checkerboard image prediction have been proposed. After training on a large number of data sets, the learned features can be used to better predict checkerboard pixels, especially in areas with complex textures, achieving better results than traditional methods. However, there are currently few methods that use deep learning technology to restore checkerboard images. There are only two related papers, MS-CNN[7] and MemNet[8]. The deep learning networks used in these papers are relatively simple, and the improvement in effect is limited, which needs to be further improved.

[0005] Related references are as follows:

[0006] [2]X.Li, W.Zhang,

[0007] 9,pp.2016–2067,2015.

[0008] [3]J.Tian,'Reversible data embedding using a difference expansion,'IEEE Trans.

[0009] Circuits Syst. Video Technol., vol.13, no.8, pp.890–896, 2003.

[0010] [4]DMThodi and JJRodriguez,'Expansion embedding techniques forreversiblewatermarking,'IEEE Trans.Image Process., vol.16, no.3, pp.721–730, 2007.

[0011] [5] M.Fallahpour,'Reversible image data hiding based on gradientadjustedprediction,'IEICE Electron.Exp., vol.5, no.20, pp.870–876, 2008.

[0012] [6] L.Luo, Z.Chen, M.Chen, X.Zeng, and Z.Xiong, 'Reversibleimagewatermarking using interpolation technique,' IEEE Trans.Inf.ForensicsSecur.,

[0013] vol.5,no.1,pp.187–193,2010.

[0014] [7] R.Hu and S.Xiang, "CNN prediction based reversible data hiding," IEEE SignalProcess.Lett., vol.28, pp.464–468, 2021.

[0015] [8]C.-C.Chang, “Neural reversible steganography with long short-termmemory,”

[0016] Secur.Commun.Netw.,vol.2021,pp.5 580 272:1–14,2021. Summary of the Invention

[0017] After comprehensively analyzing the relevant technologies of checkerboard image prediction methods, in order to solve the problem of poor checkerboard image restoration effect, the present invention proposes a checkerboard image restoration method based on deep neural network, which can more accurately predict image pixel values ​​and thus improve the quality of restored images.

[0018] The inventive concept of the present invention is: first, the original image in the data set is pre-processed by chessboard pixel division to form a training sample. After that, it is trained in a proposed deep neural network to complete the chessboard image and generate a restored image. After the training is completed, the high-definition image can be automatically restored by simply inputting the chessboard image into the pre-trained model. The proposed deep neural network mainly adopts a convolution module series connection strategy, and combines residual connection and dense connection to obtain high-dimensional image features, which are used to enrich the texture information required to complete the image. The present invention can not only realize the binary prediction and restoration of chessboard images, but also be used in the field of reversible information hiding technology to improve embedding efficiency.

[0019] The technical solutions specifically adopted in the present invention are as follows:

[0020] A checkerboard image restoration method based on a deep neural network, the steps are as follows:

[0021] S1: Divide each original grayscale image in the grayscale image dataset into two categories: black pixels and white pixels. Set all black pixels to 0 and keep white pixels unchanged, thereby converting the grayscale image into a checkerboard image. A training sample consists of a single grayscale image and the corresponding checkerboard image.

[0022] S2: Using the dataset consisting of the training samples, a deep neural network is trained using a pixel-level loss function to obtain a checkerboard image restoration model;

[0023] The deep neural network takes a checkerboard image as input and an original grayscale image as output. The network includes a first convolutional layer, a second convolutional layer, multiple convolutional modules, a fourth convolutional layer, a fifth convolutional layer and a sixth convolutional layer, and the front and rear convolutional modules are connected by a dense network. In the convolutional module, the front part is a cascade of multiple unit structures consisting of a convolutional layer plus a ReLU activation function, and each unit structure has a residual connection and a dense network connection. The rear part is connected to a convolutional layer, and the output of the convolutional layer is element-wise added to the original input of the convolutional module as the final output of the convolutional module. The input image of the deep neural network first undergoes two convolution feature extractions through the first convolutional layer and the second convolutional layer, and the extracted feature map is further extracted through multiple convolutional modules in turn. The deep features are then passed through the fourth convolutional layer and the fifth convolutional layer to output deep features. The deep features are fused with the previously obtained shallow features in the form of residual connections. The fused features are finally input into the sixth convolutional layer with a convolution kernel of 1×1 to obtain the original grayscale image finally output by the deep neural network.

[0024] S3. Input the checkerboard image to be restored into a checkerboard image restoration model to obtain a corresponding original grayscale image.

[0025] Preferably, in S1, the specific method of performing checkerboard division on the original grayscale image I is:

[0026]

[0027] Among them, (i, j) is the horizontal and vertical coordinates of the pixel, I (i，j) is the pixel at coordinate (i, j) in the original grayscale image.

[0028] Preferably, the grayscale image dataset is a BOSSbase dataset.

[0029] Preferably, in the deep neural network, the number of convolution modules m=1-3, and the number of convolution layers contained in each convolution module n=1-5.

[0030] Preferably, in the deep neural network, the number of convolution modules m=3, and the number of convolution layers contained in each convolution module n=5.

[0031] Preferably, in the deep neural network, except for the last sixth convolutional layer whose convolution kernel size is 1×1, the convolution kernel size in all other convolutional layers including the convolution module is 3×3.

[0032] Preferably, the pixel level loss function is an L1 norm loss function or an L2 norm loss function.

[0033] As an advantage, when training the deep neural network, the learning rate is set to 10 -4 .

[0034] Preferably, the Adam optimizer is selected during the training of the deep neural network.

[0035] Preferably, when training the deep neural network, the training needs to be repeated multiple times and the network model with the smallest pixel-level loss function among all training results is selected as the final checkerboard image restoration model.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention proposes a checkerboard image restoration method based on a deep neural network. First, the original image in the data set is preprocessed by checkerboard pixel division and divided into a training set and a test set. Afterwards, it is trained in a proposed deep neural network to complete the checkerboard image and generate a restored image. After the training is completed, the high-definition image can be automatically restored by simply inputting the checkerboard image into the pre-trained model. The proposed deep neural network mainly adopts a convolution block series strategy, and combines residual connections and dense connections to obtain high-dimensional image features, which are used to enrich the texture information required to complete the image. The present invention can not only realize the binary prediction restoration of checkerboard images, but also be used in the field of reversible information hiding technology to improve embedding efficiency. Compared with other deep learning related methods, the present invention proposes a deeper network structure, thereby obtaining better restored image quality and better practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the steps of the checkerboard image restoration method based on deep neural network.

[0039] Figure 2 Schematic diagram of checkerboard image distribution.

[0040] Figure 3 Figure 2 is the structural diagram of the proposed deep learning network.

[0041] Figure 4 Test images used in experiments.

[0042] Figure 5 and Figure 6 Comparison graphs are shown for image details restored using different methods.

[0043] Figure 7 Comparison of the average restored image quality of different methods on the test set. DETAILED DESCRIPTION

[0044] The specific embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

[0045] In a preferred embodiment of the present invention, a checkerboard image restoration method based on a deep neural network is provided, and the specific steps are described as follows: Figure 1 The specific implementation of steps S1 to S3 is described in detail below.

[0046] S1: Divide each original grayscale image in the grayscale image dataset into two categories: black pixels and white pixels; set all black pixels to 0 and keep white pixels unchanged, thereby converting the grayscale image into a checkerboard image. A training sample consists of a single grayscale image and the corresponding checkerboard image.

[0047] In an embodiment of the present invention, the original grayscale image I is divided into a checkerboard pattern. The specific method that can be used is:

[0048]

[0049] Among them, (i, j) is the horizontal and vertical coordinates of the pixel, I (i，j) is the pixel at coordinate (i, j) in the original grayscale image I.

[0050] Finally, the black and white pixels of the checkerboard image are distributed as follows Figure 3 As shown in the middle left figure, the middle and right figures show specific examples of the original grayscale image and the processed checkerboard image.

[0051] In this embodiment of the present invention, the grayscale image dataset used is the BOSSbase grayscale image dataset, which contains 10,000 grayscale images of size 512×512. Due to the limited memory of the graphics card used in the experiment, the original images were first downsampled using a Lanczos filter to generate 256×256 grayscale images. Each image was then processed in a loop through step S1 above, and 8,000 of them were randomly selected as the training set, and the remaining 2,000 were used as the test set.

[0052] S2: Using the dataset consisting of the training samples obtained in S1 above, the deep neural network is trained through a pixel-level loss function to obtain a checkerboard image restoration model.

[0053] The deep neural network proposed in this invention is as follows Figure 3As shown in the figure, it is abbreviated as DCRN (Deep Chequerboard Image Recovery Net). The deep neural network takes a checkerboard image as input and outputs the original grayscale image. The backbone network includes the first convolutional layer, the second convolutional layer, multiple convolutional modules, the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer. The two convolutional modules are connected by a dense network. In the convolution module, the front part is a cascade of multiple unit structures consisting of convolution layers plus ReLU activation functions, and each unit structure has residual connections and dense network connections. The back part is connected to a convolution layer, and the output of the convolution layer is element-wise added to the original input of the convolution module as the final output of the convolution module; the input image of the deep neural network first passes through the first convolution layer and the second convolution layer for two convolution feature extractions, and the extracted feature map is further extracted through multiple convolution modules in turn, and then the deep features are output after passing through the fourth convolution layer and the fifth convolution layer. The deep features are fused with the previously obtained shallow features in the form of residual connections, and the fused features are finally input into the sixth convolution layer with a convolution kernel of 1×1 to obtain the original grayscale image finally output by the deep neural network.

[0054] In the above deep neural network, in order to deepen the number of network layers, a convolutional module (BLOCK) is introduced. The front part of each convolutional module is divided into a unit structure of multiple convolutional layers plus ReLU activation function, and the back part is connected to a convolutional layer for feature fusion. The residual network structure (Resnet) is used for residual connection between each part. Figure 3 The dotted arrows in the middle indicate that a dense network structure (Densenet) is used for connection. The two connection modes of residual connection and dense network connection can be expressed as follows:

[0055] x l =H l (x l-1 )+x l-1 , (2)

[0056] x l =H l ([x0, x1, ..., x l-1 ]), (3)

[0057] Where l represents the layer, x l represents the output of the lth layer, H l Represents a convolutional layer or an activation function layer.

[0058] In the above-mentioned deep neural network of this embodiment, except for the convolution kernel size of the last sixth convolution layer which is 1×1, the convolution kernel size of all other convolution layers including the convolution module is 3×3.

[0059] The training of the above-mentioned deep neural network belongs to the prior art. The training process in this embodiment is as follows:

[0060] 1: Set the number of convolutional modules to m and the number of convolutional layers in each convolutional module to n. In this example, m = 1 to 3 convolutional modules and n = 1 to 5 convolutional layers in each convolutional module. Use different combinations of (m, n) to find the optimal hyperparameters.

[0061] 2: Let the restored image output by the deep neural network be R, and set two pixel-level loss functions l1 and l2 to evaluate the difference between the restored image and the original image. Among them, l1 is the L1 norm loss function, l2 is the L2 norm loss function, and the calculation formulas are as follows

[0062] R=DCRN(I), (4)

[0063] l1=||IR||1, (5)

[0064] l2=||IR||2. (6)

[0065] Where: DCRN represents the deep neural network, I is the network input, and R is the network output.

[0066] 3: Set the learning rate, number of training times and optimizer. In this example, the learning rate is set to 10 -4 , set the number of training times to 100, and select Adam optimizer for iterative training.

[0067] 4: Use Pytorch to implement relevant codes and train on GTX3090 graphics card;

[0068] 5: Repeat the training 100 times and save the network model with the smallest loss function among the 100 training results as the final checkerboard image restoration model for subsequent experimental results testing.

[0069] S3: After the test images in the test set are processed by S1, a checkerboard distribution image is generated. The checkerboard distribution image is input into the checkerboard image restoration model saved in S2, and the restored image is output to evaluate its performance.

[0070] Thus, after the above steps, checkerboard image restoration can be achieved based on a deep neural network. Because setting different numbers of convolutional blocks and convolutional layers will cause the depth of the network model to change, this example first fixed the loss function to l1 and tested the quality of the restoration of 2000 test set images under different network parameters, as shown in Table 1.

[0071] Table 1 Average image quality of the test set under different network parameters (PSNR: dB)

[0072]

[0073] As can be seen, as the parameter settings increase and the number of network layers deepens, the model can learn more image features to restore pixels, thereby improving the quality of the restored image. Therefore, this example primarily uses parameters (3, 5) for subsequent testing. Those skilled in the relevant technical field can achieve even better restoration results by setting a deeper network layer based on graphics card performance. After fixing the network parameters, the results of testing the two loss functions are shown in Table 2.

[0074] Table 2 Average image quality of the test set under different loss functions (PSNR: dB)

[0075]

[0076] It can be seen that the l1 loss function can achieve better results for the restoration of checkerboard images and the proposed network model. In addition, this embodiment also conducts comparative tests with two existing technical solutions, MS-CNN and MemNet. First, for example Figure 4 The two commonly used test images and the two images in the test set are shown to show the restoration details. Figure 5 and Figure 6 It can be seen that the method proposed in the present invention is more excellent in restoring texture details. In addition, this embodiment also compares the average restoration image quality of different schemes on a test set of 2000 images. The experimental results refer to Figure 7 This method achieves better image restoration quality than other deep learning-based methods, with an improvement of nearly 0.7dB on the test set.

[0077] Of course, it should be noted that in the above step S3, the test set can be replaced by the checkerboard image to be restored in actual application.

[0078] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A checkerboard image restoration method based on deep neural network, characterized in that: Here are the steps: S1: Divide each original grayscale image in the grayscale image dataset into two categories: black pixels and white pixels; All black pixels are set to 0, and white pixels remain unchanged, thereby converting the grayscale image into a checkerboard image. A training sample consists of a single grayscale image and the corresponding checkerboard image. S2: Using the dataset consisting of the training samples, a deep neural network is trained using a pixel-level loss function to obtain a checkerboard image restoration model; The deep neural network takes a checkerboard image as input and an original grayscale image as output. The network includes a first convolutional layer, a second convolutional layer, multiple convolutional modules, a fourth convolutional layer, a fifth convolutional layer and a sixth convolutional layer, and the front and rear convolutional modules are connected by a dense network. In the convolutional module, the front part is a cascade of multiple unit structures consisting of a convolutional layer plus a ReLU activation function, and each unit structure has a residual connection and a dense network connection. The rear part is connected to a convolutional layer, and the output of the convolutional layer is element-wise added to the original input of the convolutional module as the final output of the convolutional module. The input image of the deep neural network first undergoes two convolution feature extractions through the first convolutional layer and the second convolutional layer, and the extracted feature map is further extracted through multiple convolutional modules in turn. The deep features are then passed through the fourth convolutional layer and the fifth convolutional layer to output deep features. The deep features are fused with the previously obtained shallow features in the form of residual connections. The fused features are finally input into the sixth convolutional layer with a convolution kernel of 1×1 to obtain the original grayscale image finally output by the deep neural network. S3. Input the checkerboard image to be restored into a checkerboard image restoration model to obtain a corresponding original grayscale image.

2. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: In S1, the specific method of performing chessboard division on the original grayscale image I is as follows: Among them, (i, j) is the horizontal and vertical coordinates of the pixel, I (i,) is the pixel at coordinate (i, j) in the original grayscale image I.

3. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: The grayscale image dataset is the BOSSbase dataset.

4. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: In the deep neural network, the number of convolution modules m=1-3, and the number of convolution layers contained in each convolution module n=1-5.

5. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: In the deep neural network, the number of convolution modules m=3, and the number of convolution layers contained in each convolution module n=5.

6. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: In the deep neural network, except for the last sixth convolutional layer whose convolution kernel size is 1×1, the convolution kernel size in all other convolutional layers including the convolution module is 3×3.

7. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: The pixel level loss function is an L1 norm loss function or an L2 norm loss function.

8. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: When training the deep neural network, the learning rate is set to 10 -4 .

9. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: When training the deep neural network, the Adam optimizer is selected.

10. The checkerboard image restoration method based on deep neural network according to claim 1, characterized in that: When training the deep neural network, the training needs to be repeated multiple times and the network model with the smallest pixel-level loss function among all training results is selected as the final checkerboard image restoration model.

Images