Water meter compressed image artifact removal method based on large-kernel depth separable convolution

Abstract

The invention provides a water meter compressed image artifact removal method based on large-kernel depth separable convolution. The method is composed of a compression quality estimation module (CQE Block) and a recovery sub-network (RSNet). The former estimates the quality of the input image and the latter processes the image based on the estimated quality representation. A plurality of RSNet parts are connected in a cascading manner for sequential processing, so that different features can be efficiently processed. Each RSNet part is composed of a plurality of large-kernel depth separable convolution blocks (LKDSC Blocks), and the convolution blocks can effectively improve the image quality. According to the method, artifact removal can be fully carried out on water meter compressed images with different compression qualities, and good performance is shown in the aspect of removing compression artifacts from multiple compressed data sets.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for removing artifacts in compressed images, and in particular to a method for removing artifacts in compressed water meter images based on large kernel depth separable convolution. Background Art

[0002] In water management, the transmission of water meter images is often limited by bandwidth and storage capacity, so the water meter images need to be compressed before being uploaded to the smart water system. Currently, JPEG compression, as a common lossy compression method, is widely used in the storage and transmission of water meter images. However, this compression method loses high-frequency information in the water meter image during the quantization stage, resulting in compression artifacts such as block effects, ringing effects, and blurring in the image. These artifacts not only affect the visual quality of the image, but also affect the accuracy of subsequent water meter readings.

[0003] In order to solve the artifact problem generated in compressed images, the existing solutions are mainly divided into methods based on traditional image processing and methods based on deep learning. Traditional image processing methods mostly rely on filtering algorithms to reduce the impact of artifacts by smoothing images. They can achieve good results when processing some simple images, but their effects are often limited in complex image degradation. In addition, traditional methods usually rely on fixed rules and parameters, lack sufficient adaptability and flexibility, and are difficult to effectively deal with the diversity of different compressed image artifact types. Therefore, traditional image processing methods have certain limitations in the task of removing artifacts from water meter compressed images. With the rapid development of deep neural networks, compressed image artifact removal methods based on deep learning have gradually become mainstream. Deep learning methods can effectively handle complex image degradation problems through their powerful feature extraction and nonlinear mapping capabilities, and show superior performance than traditional methods in removing compressed artifacts. Compared with traditional methods, deep learning methods can automatically learn the potential features in images, thereby more effectively removing artifacts from water meter compressed images and improving recognition accuracy.

[0004] Although deep learning-based compressed image artifact removal methods have achieved remarkable results, there are still some problems. Many methods require training multiple specific networks for images of different compression qualities, which limits the practicality of the methods. Although some methods use a single model to process water meter images of different compression qualities, these methods often ignore the effective use of compression quality information and cannot accurately reflect the degree of degradation of water meter images. Therefore, there is an urgent need for a new method that can effectively remove artifacts from water meter compressed images, improve visual effects, and increase processing flexibility and efficiency, thereby improving the accuracy of water meter readings. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned proposed method, the present invention proposes a method for removing artifacts from compressed water meter images based on large kernel depthwise separable convolution, in order to make full use of the quality estimation score obtained by the compression quality estimation module (CQE Block) and the image features extracted by the restoration subnetwork (RSNet) for multi-stage image restoration. The image restoration network of this method connects multiple RSNets in a cascade manner for sequential processing, which can restore degraded images of different compression qualities, realize flexible and efficient removal of artifacts from compressed water meter images, and thus improve the accuracy of water meter readings.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The invention is a method for removing artifacts from compressed water meter images based on large kernel depth-separable convolution, which is characterized by the following steps:

[0008] Step 1, data preparation: water meter image acquisition and production;

[0009] Step 2, constructing a water meter compressed image artifact removal network consisting of a shallow feature extraction module, a compression quality estimation module, and an image restoration module;

[0010] Step 3: Build a shallow feature extraction module:

[0011] The shallow feature extraction module mainly consists of a convolutional layer with a convolution kernel of size 3x3, a stride of 1 and a padding of 1;

[0012] Step 3.1, receiving a water meter compressed image, usually a color image or a grayscale image. The number of channels C of a color image is 3, and the number of channels C of a grayscale image is 1;

[0013] Step 3.2: The input image passes through the convolution layer to extract the initial low-level features, and finally obtains the shallow feature F0, F0 = H SF (I L )(1), where H SF (·) is the shallow feature extraction module;

[0014] Step 4: Build the compression quality estimation module:

[0015] The compression quality estimation module is composed of a global feature extraction module, an adaptive average pooling layer, multiple convolutional layers and a Sigmoid activation function;

[0016] Step 4.1, receiving the feature F0 from the shallow feature extraction module, further extracting and enhancing the input image through the global feature extraction module, and finally obtaining the global feature F GF , F GF =H GF(F0)(2), where H GF (·) is the global feature extraction module;

[0017] Step 4.2, F GF Input to the adaptive average pooling layer to compress the global feature map into an output of a specific size, with an output shape of (batch_size, 64, 1, 1);

[0018] In step 4.3, multiple convolutional layers are used to gradually reduce the number of channels of the global feature map. In this way, the network gradually reduces complexity and focuses on the quality features of the image. The number of output channels of the convolutional layer is gradually reduced, and the last layer outputs a single channel, which represents the quality estimation of the image;

[0019] In step 4.4, the output of the quality estimation is compressed into the range of [0, 1] through the Sigmoid activation function, which represents the quality estimation score of the image.

[0020] Step 5: Build the image restoration module:

[0021] The module includes: two head layers, which are composed of two convolutional layers; the recovery sub-network part, which is composed of RSNet1 to RSNet6, each RSNet is composed of a small U-Net network composed of multiple large-core depth-separable convolutional blocks; two tail layers, which are composed of two convolutional layers;

[0022] Step 5.1: The two head layers receive shallow features, then process the input features in parallel, and fuse the processed features to obtain the fused features F MF ;

[0023] Step 5.2, enter F MF After being processed by RSNet1 and RSNet2, the obtained features are passed to the tail layer for fusion with the original input image. Among them, the quality estimation score (QES) from the compression quality estimation module is used to weight the features. Specifically, QES is multiplied with the feature map extracted by the head layer for weighting;

[0024] In step 5.3, the feature map obtained in the first stage is weighted with the quality estimation score (QES), and then the weighted features are processed by RSNet3 and RSNet4, and then fused with the original input image through the tail layer. The weighting operation of QES on the feature map is similar to the weighting method in step 5.2;

[0025] In step 5.4, the feature map obtained in the second stage is weightedly fused with the quality estimation score (QES), and then the weighted features are processed by RSNet5 and RSNet6, and finally weighted fused with the original input image through the tail layer to obtain the final output.

[0026] Step 5.5, construct the total loss function L total :L total =L IR +λL QES (3)

[0027] In formula (3), λ is the hyperparameter in the objective loss function L, λ = 0.005;

[0028]

[0029] In formula (4), K is the total number of stages of model output; y k is the real image (ground truth) of the kth stage; is the model prediction output of the kth stage. ① Stage loss: Part 1 Calculate the mean square error between the predicted output and the true value of each stage. ② Adjacent stage loss: Part 2 Compute the mean squared error between adjacent stage outputs.

[0030]

[0031] In formula (5), represents the true compression quality score of the label image corresponding to the i-th compressed image in the batch data, Represents the compression quality score predicted for the i-th compressed image in the batch data.

[0032] Step 6, network training: Move the constructed network model to GPU training. Use Adam optimizer to optimize the image restoration network, calculate the total loss function and then back propagate it to the network for optimization. Stop training when the total loss function converges. After training is completed, save the model.pth weight file for subsequent testing.

[0033] Step 7, network testing: Use the water meter compression dataset to test the model and evaluate the performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0035] Figure 1 It is a schematic diagram of the structure of the network of the present invention.

[0036] Figure 2 Schematic diagram of the compression quality estimation module.

[0037] Figure 3 Schematic diagram of the image restoration module. DETAILED DESCRIPTION

[0038] In the present invention, a method for removing artifacts from water meter compressed images based on large kernel depthwise separable convolution is proposed, aiming to use a single model to achieve artifact removal of water meter images with different compression qualities, and use the quality estimation score obtained by the compression quality estimation module to guide multiple image restoration sub-networks to achieve sequentially processed compression artifact removal.

[0039] Figure 1 The structural diagram of the network of the present invention is shown below, and the specific implementation of the present invention will be described.

[0040] Step 1, data preparation: water meter image acquisition and production, use standard JPEG compression to preprocess the image in MATLA, and obtain water meter images with different compression qualities after preprocessing. L ∈R C×H×W . Where C represents the number of channels of the image, H and W represent the height and width of the image respectively;

[0041] Step 2, constructing a JPEG artifact removal network consisting of a shallow feature extraction module, a compression quality estimation module, and an image restoration module;

[0042] Step 3: Build a shallow feature extraction module:

[0043] The shallow feature extraction module mainly consists of a convolutional layer with a convolution kernel of size 3x3, a stride of 1 and a padding of 1;

[0044] Step 3.1, receiving a water meter compressed image, usually a color image or a grayscale image. The number of channels C of a color image is 3, and the number of channels C of a grayscale image is 1;

[0045] Step 3.2: The input image passes through the convolution layer to extract the initial low-level features, and finally obtains the shallow feature F0, F0 = H SF (I L )(1), where H SF (·) is the shallow feature extraction module;

[0046] Step 4: Build the compression quality estimation module:

[0047] The compression quality estimation module is composed of a global feature extraction module, an adaptive average pooling layer, multiple convolutional layers and a Sigmoid activation function;

[0048] Step 4.1, receiving the feature F0 from the shallow feature extraction module, further extracting and enhancing the input image through the global feature extraction module, and finally obtaining the global feature F GF , F GF =H GF (F0)(2), where H GF(·) is the global feature extraction module;

[0049] Step 4.2, F GF Input to the adaptive average pooling layer to compress the global feature map into an output of a specific size. The output shape is (batch_size, 64, 1, 1). Through this pooling layer, the network can obtain global statistical information of the image and generate a single scalar value.

[0050] In step 4.3, multiple convolutional layers are used to gradually reduce the number of channels of the feature map. In this way, the network gradually reduces complexity and focuses on the quality features of the image. The number of output channels of the convolutional layer is gradually reduced, and the last layer outputs a single channel, which represents the quality estimation of the image;

[0051] In step 4.4, the output of the quality estimation is compressed into the range of [0, 1] through the Sigmoid activation function, which represents the quality estimation score of the image.

[0052] Step 5: construct the image restoration module: Figure 3 As shown;

[0053] The module includes: two head layers, which are composed of two convolutional layers; the recovery sub-network part, which is composed of RSNet1 to RSNet6, each RSNet is composed of a small U-Net network composed of multiple large kernel depth separable convolution blocks (LKDSC Block); two tail layers, the first tail layer is composed of convolution, normalization and activation function, and the second tail layer is composed of convolution and normalization;

[0054] Step 5.1: The two head layers receive shallow features, then process the input features in parallel, and fuse the processed features to obtain the fused features F MF ;

[0055] Step 5.2, enter F MF After being processed by RSNet1 and RSNet2, the obtained features are passed to the tail layer for fusion with the original input image. Among them, the quality estimation score (QES) from the compression quality estimation module is used to weight the features. Specifically, QES is multiplied with the feature map extracted by the head layer for weighting;

[0056] In step 5.3, the feature map obtained in the first stage is weighted with the quality estimation score (QES), and then the weighted features are processed by RSNet3 and RSNet4, and then fused with the original input image through the tail layer. The weighting operation of QES on the feature map is similar to the weighting method in step 5.2;

[0057] In step 5.4, the feature map obtained in the second stage is weightedly fused with the quality estimation score (QES), and then the weighted features are deeply processed by RSNet5 and RSNet6, and finally weighted fused with the original input image through the tail layer to obtain the final output.

[0058] Step 5.5, the shallow feature extraction module constructed in step 3, the compression quality estimation module constructed in step 4, and the image restoration module constructed in step 5 together constitute the total water meter compressed image artifact removal network, and construct the total loss function L total :L total =L IR +λL QES (3)

[0059] In formula (3), λ is the hyperparameter in the objective loss function L, λ = 0.005;

[0060]

[0061] In formula (4), K is the total number of stages of model output; y k is the real image (ground truth) of the kth stage; is the model prediction output of the kth stage. Among them, the stage loss is: the first part Compute the mean squared error between the predicted output and the true value at each stage; Adjacent stage losses: Part II Compute the mean squared error between adjacent stage outputs.

[0062]

[0063] In formula (5), represents the true compression quality score of the label image corresponding to the i-th compressed image in the batch data, Represents the compression quality score predicted for the i-th compressed image in the batch data.

[0064] Step 6, network training: First, select the training set and divide the validation data set. Then, move the constructed network model to GPU training. Use Adam optimizer to optimize the image restoration network, calculate the total loss function and back propagate it to the network for optimization. Stop training when the total loss function converges. After training, save the model.pth weight file for subsequent testing.

[0065] Step 7, network testing:

[0066] Step 7.1, with reference images: Select Classic5 (5 images) and LIVE (29 images) to test the algorithm performance. The specific method is: first compress the selected test data set to different degrees, then use different network models to remove artifacts from the compressed images, and finally calculate the PSNR (peak signal-to-noise ratio) and SSIM (structural similarity distance) of the restored image and the real image to test the network performance.

[0067] Step 7.2, No-reference images: Select images that have been compressed multiple times in reality and create a dataset Real (10 images) for algorithm performance testing. Use different network models to remove artifacts from the above Real dataset, and finally calculate the Natural Image Quality Evaluation (NIQE) and No-reference Image Spatial Quality Evaluation (BRISQUE) scores to test the network performance.

[0068] Although the specific embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments. The above embodiments are only instructive and illustrative, and not restrictive. Under the guidance of this specification, those skilled in the art can also make many kinds of water meter compressed image artifact removal methods without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A method for removing artifacts from water surface compressed images based on large kernel depthwise separable convolution, characterized in that: The steps are as follows: Step 1, data preparation: water meter image acquisition and production; Step 2, constructing a JPEG artifact removal network consisting of a shallow feature extraction module, a compression quality estimation module, and an image restoration module; Step 3: Build a shallow feature extraction module: The shallow feature extraction module mainly consists of a convolutional layer with a convolution kernel of size 3x3, a stride of 1 and a padding of 1; Step 3.1, receive a compressed image of a water meter, which is usually a color image or a grayscale image. The number of channels C of a color image is 3, and the number of channels C of a grayscale image is 1; Step 3.2: The compressed image of the water meter passes through the convolution layer to extract the initial low-level features, and finally obtains the shallow feature F0, F0 = H SF (I L )(1), where H SF (·) is the shallow feature extraction module; Step 4: Build the compression quality estimation module: The compression quality estimation module is composed of a global feature extraction module, an adaptive average pooling layer, multiple convolutional layers and a Sigmoid activation function; Step 4.1, receiving the feature F0 from the shallow feature extraction module, further extracting and enhancing the input image through the global feature extraction module, and finally obtaining the global feature F GF , F GF =H GF (F0)(2), where H GF (·) is the global feature extraction module; Step 4.2, F GF Input to the adaptive average pooling layer to compress the global feature map into an output of a specific size; In step 4.3, multiple convolutional layers are used to gradually reduce the number of channels of the feature map. In this way, the network gradually reduces complexity and focuses on the quality features of the image. The number of output channels of the convolutional layer is gradually reduced, and the last layer outputs a single channel, which represents the quality estimation score of the image; In step 4.4, the output of the quality estimation is compressed into the range of [0, 1] through the Sigmoid activation function to represent the quality score of the image. Step 5: Build the image restoration module: The module includes: two head layers, which are composed of two convolutional layers; the recovery sub-network part, which is composed of RSNet1 to RSNet6, each RSNet is composed of a small U-Net network composed of multiple large kernel depth separable convolutional blocks (LKDSC Block); two tail layers, which are composed of two convolutional layers; Step 5.1: The two head layers receive shallow features, then process the input features in parallel, and fuse the processed features to obtain the fused features F MF ; Step 5.2, enter F MF After being processed by RSNet1 and RSNet2, the obtained features are passed to the tail layer for fusion with the original input image. Among them, the quality estimation score (QES) from the compression quality estimation module is used to weight the features. Specifically, QES is multiplied with the feature map extracted by the head layer for weighting; In step 5.3, the feature map obtained in the first stage is weighted with the quality estimation score (QES), and then the weighted features are processed by RSNet3 and RSNet4, and then fused with the original input image through the tail layer. The weighting operation of QES on the feature map is similar to the weighting method in step 5.2; In step 5.4, the feature map obtained in the second stage is weightedly fused with the quality estimation score (QES), and then the weighted features are deeply processed by RSNet5 and RSNet6, and finally weighted fused with the original input image through the tail layer to obtain the final output. Step 5.5, the shallow feature extraction module constructed in step 3, the compression quality estimation module constructed in step 4, and the image restoration module constructed in step 5 together constitute a total water meter compressed image artifact removal network, and construct a total loss function; Step 6, network training: Move the constructed network model to GPU training. Use Adam optimizer to optimize the image restoration network, calculate the total loss function and then back propagate it to the network for optimization. Stop training when the total loss function converges. After training is completed, save the model.pth weight file for subsequent testing. Step 7, network testing: Use water meter compressed images to test the model and evaluate the performance of the network.

2. The method for removing artifacts from water surface compressed images based on large kernel depthwise separable convolution as claimed in claim 1, characterized in that In step 5.5, the total loss function is calculated as follows: total =L IR +λL QES (3); In formula (3), λ is the total loss function L total The hyperparameter in is λ=0.005; In formula (4), K is the total number of stages of model output; y k is the real image of the kth stage; is the model prediction output of the kth stage. ① Stage loss: Part 1 Calculate the mean square error between the predicted output and the true value of each stage. ② Adjacent stage loss: Part 2 Compute the mean squared error between adjacent stage outputs. In formula (5), represents the true compression quality score of the label image corresponding to the i-th JPEG image in the batch data, Represents the predicted compression quality score of the i-th JPEG image in the batch data.

Images