A lightweight neural network deblocking method based on scale compression
By introducing a lightweight neural network based on scale compression and a Transformer structure with multiple attention mechanisms in image processing, the block effect problem in image compression is solved, the image quality improvement and computational complexity reduction are achieved, and the resource limitation is suitable for environment.
Patent Information
- Application Number
- CN202411268673.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-11
AI Technical Summary
The prior art has severe block effect of low-quality images during image compression, resulting in extremely poor visual effects of the decoded image. The commonly used Transformer structure has high computational complexity in image processing, making it difficult to meet the limitations of computing resources.
A deblocking effect method of lightweight neural network based on scale compression is proposed. The Transformer structure module with multiple attention mechanisms is adopted, including a self-attention module based on window division and a self-attention module based on scale compression is constructed. The level deblocking effect enhances neural network, and achieves image quality improvement through feature extraction, feature enhancement and image reconstruction modules.
The block effect of removing compressed images at lower computing costs and higher efficiency is achieved, which significantly reduces the computational complexity, ensures improvement in image quality, and has extremely small parameters, making it suitable for resource-constrained environments.
Smart Images

Figure CN118799212B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image compression and quality enhancement, relates to an image deblocking method, and specifically provides a deblocking method of a lightweight neural network based on scale compression. Background Art
[0002] As a carrier of visual information, images have become one of the main media for humans to obtain information from the outside world. In particular, with the development of science and technology and emerging digital communication media, digital images have gradually occupied an important position in people's production and life. Image compression, as an indispensable part of digital image transmission and storage, is used to convert the original image data into a portable and circulated bit stream. Among them, the commonly used static image compression methods include JPEG, JPEG-XR and PNG. The main steps of the image compression encoding process are color conversion, frequency domain transformation, quantization and encoding, and the image decoding process is the corresponding inverse operation. Among them, the key factor affecting image quality is quantization. The larger the quantization step size, the less bit data is used, but the greater the data error generated, the greater the difference between the image obtained after decoding and the original image, and the worse the quality; conversely, the more bit data is used, the richer the details and texture of the decoded image, and the better its quality. In real application scenarios, there are often limitations on transmission bandwidth and memory space, so it is necessary to adjust the quantization step size to use a limited bit data amount to transmit and store images, which leads to severe block effects of low-quality images, making the human eye visual effect of the decoded image extremely poor. In order to balance the limitations of compressed transmission and visual effects, quality enhancement post-processing of image encoding and decoding is of great practical significance.
[0003] In recent years, deep learning theory has shown amazing potential in low-level image vision tasks such as denoising, deblocking and super-resolution. With the upgrading of technology, more and more enhanced networks choose to build with larger parameters and more resource consumption in exchange for better enhancement effects, which puts higher requirements on computer computing power and video memory. In particular, the Transformer structure neural network, which is different from the convolutional neural network mechanism, has become popular. The core mechanism of Transformer is self-attention calculation, which was born in the field of natural language processing, but because of its natural long-range and large receptive field properties, it is widely used in the field of image processing. In the self-attention calculation of the image, each pixel should correspond to a token, while in the field of natural language processing, each word corresponds to a token. Since the scale of the three key matrices (Q, K, V) involved in the self-attention calculation is consistent with the number of tokens, the computational complexity of the matrix multiplication of the self-attention calculation in the image field is extremely high, and is positively correlated with the length-width product of the image (the number of pixels). It can be seen that it is very necessary to reduce the computational complexity of the Transformer structure network in the image field. Summary of the invention
[0004] The purpose of the present invention is to provide a deblocking method for a lightweight neural network based on scale compression, propose a Transformer structure module based on multiple attention mechanisms, and then construct a lightweight deblocking enhancement neural network, so as to remove the block effect of the compressed image with lower computational cost and higher efficiency, and achieve image quality enhancement.
[0005] To achieve the above object, the technical solution adopted by the present invention is:
[0006] A method for removing blocking effects of a lightweight neural network based on scale compression includes the following steps:
[0007] Step 1. Use JPEG compression as the image codec to compress the original RGB image or the original grayscale image into a low-quality image, take the low-quality image as input, and take the original RGB image or the original grayscale image as a label to construct a training dataset;
[0008] Step 2. Construct a lightweight deblocking effect enhancement neural network and complete the training based on the training data set; the lightweight deblocking effect enhancement neural network includes: a feature extraction module, a feature enhancement module and an image reconstruction module, wherein the feature enhancement module is composed of four Transformer structure modules based on multiple attention mechanisms;
[0009] Step 3. Input the compressed image to be processed as input data into the trained lightweight deblocking effect enhancement neural network, and the lightweight deblocking effect enhancement neural network outputs a high-quality enhanced image.
[0010] Furthermore, in the lightweight deblocking effect enhancement neural network, the feature extraction module is composed of a convolutional layer, which is used to obtain the shallow features of the input image and input it into the feature enhancement module; the image reconstruction module is composed of a convolutional layer, which is used to reconstruct a high-quality enhanced image according to the output features of the feature enhancement module.
[0011] Furthermore, in the lightweight deblocking effect enhancement neural network, the output features of the fourth Transformer structure module based on multiple attention mechanisms of the feature enhancement module are added to the input features of the feature enhancement module to obtain the output features of the feature enhancement module; the Transformer structure module based on multiple attention mechanisms includes: a self-attention module based on window partitioning, a self-attention module based on scale compression and a multi-layer perceptron, and the input feature F in Evenly split into [F Win ,F Cin ], feature F Win With feature F Cin They are used as the input of the self-attention module based on window partitioning and the self-attention module based on scale compression, respectively. The self-attention module based on window partitioning and the self-attention module based on scale compression output the feature F Wout With feature F Cout , feature F Wout With feature F Cout Concatenate to get feature F mid , feature F mid With feature F in After addition, it is input to the multi-layer perceptron, and the multi-layer perceptron outputs the feature F out .
[0012] Furthermore, in the window-based self-attention module, feature F Win After a layer of standardization module is used for standardization, the standardized processing result is stretched to obtain a two-dimensional feature F Wmid0 , and then divided into several local window features through local continuous window division operation, all local window features are sent to the window self-attention module in parallel; all output features of the window self-attention module are restored according to the original spatial position to obtain feature F Wmid1 , and then through dimension conversion to obtain the three-dimensional feature F Wmid2 ; Feature F Wmid2 With feature F Win Add together to get feature F Wmid3 , feature F Wmid3After passing through another layer standardization module and multi-layer perceptron, feature F is obtained. Wmid4 , feature F Wmid4 With feature F Wmid3 Add the output feature F Wout .
[0013] Furthermore, in the window self-attention module, the input local window features are mapped into key matrices Q, K and V through a linear layer, and the self-attention calculation results are obtained through matrix multiplication:
[0014] ,
[0015] Among them, Attention represents the result of self-attention calculation, d is the dimension of feature channel, B is the spatial position encoding matrix, Softmax is the activation function, represents matrix multiplication and T represents the transpose symbol.
[0016] Furthermore, in the self-attention module based on scale compression, feature F Cin After a layer of standardization module, the standardized processing result is decomposed into I by using uniform downsampling method with a ratio of I 2 sparse non-local sub-features, and stretch them to obtain two-dimensional sparse non-local sub-features, and then send them to the channel compression self-attention module in parallel; all output features of the channel compression self-attention module are restored according to the original spatial position to obtain feature F Cmid1 , and then through dimension conversion to obtain the three-dimensional feature F Cmid2 ; Feature F Cmid2 With feature F Cin Add together to get feature F Cmid3 , feature F Cmid3 After passing through another layer standardization module and multi-layer perceptron, feature F is obtained. Cmid4 , feature F Cmid4 With feature F Cmid3 Add the output feature F Cout .
[0017] Furthermore, in the channel compression self-attention module, the input sparse non-local sub-features are mapped into key matrices Q, K, and V through a linear layer; at the same time, the input sparse non-local sub-features are passed through another linear layer to obtain the compression matrix A; the compression matrix A is used as an intermediate medium for self-attention calculation to participate in the matrix multiplication operation, and the self-attention calculation result is obtained:
[0018] ,
[0019] ,
[0020] Among them, Attention represents the result of self-attention calculation, d is the dimension of feature channel, B is the spatial position encoding matrix, Softmax is the activation function, represents matrix multiplication and T represents the transpose symbol.
[0021] Furthermore, the training adopts MSELoss as the loss function to complete the training of the lightweight deblocking effect enhanced neural network.
[0022] Based on the above technical solution, the beneficial effects of the present invention are:
[0023] The present invention provides a deblocking method for a lightweight neural network based on scale compression, and creatively proposes a window-partitioned self-attention module (WSAB) for obtaining local information and a scale-compression-based self-attention module (CSAB) for obtaining non-local information. Compared with conventional self-attention modules, the computational complexity of both modules is significantly reduced. On this basis, the present invention constructs a module (MATL) of a Transformer structure based on multiple attention mechanisms, and adopts four MATLs to form a feature enhancement module, which finally forms a lightweight deblocking enhancement neural network together with a feature extraction module and an image reconstruction module, thereby realizing low-cost enhancement of RGB images and grayscale images. In summary, the lightweight deblocking enhancement neural network proposed in the present invention has the advantages of simple structure, extremely small number of parameters and extremely low computational complexity, and can ensure a good image deblocking and repairing effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a structural schematic diagram of a lightweight deblocking effect enhancement neural network based on scale compression of the present invention.
[0025] Figure 2 It is a structural schematic diagram of the multi-attention neural unit (MATU) in the Transformer structure module (MATL) based on multiple attention mechanisms of the present invention.
[0026] Figure 3 It is a structural schematic diagram of the window partitioning-based self-attention module (WSAB) and the scale compression-based self-attention module (CSAB) in the multi-attention neural unit (MATU) of the present invention.
[0027] Figure 4 It is a structural diagram of the window self-attention module (Window Attention) in the window partitioning based self-attention module (WSAB) of the present invention.
[0028] Figure 5Schematic diagram of the downsampling and restoration process in the scale compression based self-attention module (CSAB) of the present invention.
[0029] Figure 6 It is a structural schematic diagram of the channel compression self-attention module (Anchor Attention) in the scale compression based self-attention module (CSAB) of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0031] This embodiment provides a lightweight neural network deblocking method based on texture and structure decomposition, which is used to quickly and efficiently remove the block effect of a compressed image. The method specifically includes the following steps:
[0032] Step 1. Data preprocessing;
[0033] Step 1.1. Use JPEG compression as the image codec to compress the original RGB image or grayscale image into a low-quality image under randomly generated compression qualities. The specific process can be expressed as:
[0034] I com = JPEG ( I ori , QF ), QF ∈ [ 10 , 20 , 30 , 40 ] ,
[0035] Among them, QF is the set compression quality, I ori is the original image, I com is the compressed image generated; for color training and test data sets, the original color image can be directly compressed by JPEG; for grayscale training and test data sets, the original grayscale image can be directly compressed, or the brightness channel data of the color image can be compressed and saved as a grayscale image;
[0036] Take the low-quality image as input, the original RGB image or the original grayscale image as the label, construct the training sample, and then construct the training data set;
[0037] Step 2. Build a lightweight deblocking enhancement neural network and complete the training;
[0038] Step 2.1. Construct a lightweight deblocking effect enhancement neural network, including: a feature extraction module, a feature enhancement module and an image reconstruction module. The input of the deblocking effect enhancement neural network is a three-channel RGB color compressed image or a single-channel grayscale compressed image, and the output of the lightweight deblocking effect enhancement neural network corresponds to an RGB enhanced image or a grayscale enhanced image.
[0039] The light computational complexity of the deblocking effect enhancement neural network is as follows Figure 1 As shown in the figure, the feature extraction module for obtaining shallow features of the image and the image reconstruction module for outputting high-quality enhanced images are both composed of a convolution layer Conv3×3. The only difference is that the number of input and output channels of the two is different, that is, the input image obtains a feature with a scale of 2d×H×W after passing through the feature extraction module, and the 2d×H×W feature obtains a reconstructed high-quality image with the same scale as the original input image after passing through the image reconstruction module; d is the feature channel dimension, H is the length of the feature space dimension, and W is the width of the feature space dimension;
[0040] As the core module of the lightweight deblocking effect enhancement neural network, the feature enhancement module consists of four Transformer structure modules (Multi-Attention Transformer Layer, MATL) based on multiple attention mechanisms. The output features of the fourth MATL are added to the input features of the feature enhancement module to obtain the output features of the feature enhancement module; the MATL module is mainly composed of three Multi-Attention Transformer Units (MATU) and a convolutional layer Conv3×3; Figure 2 As shown in the figure, MATU consists of two different attention mechanisms, namely, a window-based self-attention block (WSAB) for obtaining local information and a compression-based self-attention block (CSAB) for obtaining non-local information. At the same time, MATU also contains a multi-layer perceptron (MLP) to fuse the results of the two different self-attentions. In MATU, the input feature F with a scale of 2d×H×W is first split in the channel dimension by a split operation (Split). in Evenly split into two parts [F Win , F Cin ], as the input of WSAB and CSAB respectively, WSAB and CSAB output features F Wout With F Cout , feature F Wout With F Cout Concatenate (Cat) in the channel dimension to get feature F mid , feature F mid With feature F in After addition, it is input to MLP, and MLP outputs feature F out , which is the output feature of MATU, with a scale of 2d×H×W;
[0041] Furthermore, the window partitioning based self-attention module (WSAB) and the scale compression based self-attention module (CSAB) are as follows: Figure 3 As shown;
[0042] The window-based self-attention module (WSAB) mainly includes the window self-attention module (WindowAttention), the feature F with a scale of d×H×W Win After inputting WSAB, it is normalized by a layer normalization module (LayerNorm) and then stretched in the spatial dimension to obtain a feature F with a scale of d×HW Wmid0 , and then divided into (H / W in )·(W / W in ) size is d×W in W in The local window feature, W in is the local window size, and all local window features are sent to the window self-attention module (Window Attention) in parallel; the window self-attention module is as follows Figure 4 As shown in the figure, the input local window features are mapped into three key matrices Q (Query), K (Key) and V (Value) through a linear layer. The spatial dimensions of the three matrices are consistent with the input local window features, that is, d×W in W in ; Then use the self-attention calculation formula to perform matrix multiplication on the three matrices in turn to obtain the result of self-attention calculation, which is expressed as:
[0043] ,
[0044] Among them, Attention represents the result of self-attention calculation, B is the spatial position encoding matrix, and Softmax is the activation function that normalizes the matrix to a sum of elements equal to 1. Represents matrix multiplication;
[0045] All output features of the window self-attention module are restored according to the original spatial position to obtain the feature F of scale d×HW Wmid1 , and then after dimension conversion, we get the feature F with a scale of d×H×W Wmid2 ; Feature F Wmid2 With feature F Win Add together to get feature F Wmid3 , feature F Wmid3 Then, after passing through the layer normalization module (LayerNorm) and the multi-layer perceptron (MLP), the feature F is obtained. Wmid4 , feature F Wmid4 With feature F Wmid3Add together to get the output feature F of WSAB Wout ;
[0046] The scale compression based self-attention module (CSAB) mainly includes the channel compression self-attention module with joint downsampling. Before the self-attention calculation, two different scale compressions are required, namely spatial dimension compression and feature channel dimension compression. First, a layer normalization module (LayerNorm) is used to convert the input feature F with a scale of d×H×W into Cin Standardization processing, such as Figure 5 As shown, the standardized processing result is uniformly downsampled at a ratio of I and decomposed into I 2 The sparse non-local sub-features are d×h×w in scale, where h=H / I and w=W / I. The sparse non-local sub-features are then stretched in spatial dimension to obtain sparse non-local sub-features of scale d×hw, and then sent to the channel compression self-attention module (Anchor Attention) in parallel. The channel compression self-attention module (Anchor Attention) is as follows Figure 6 As shown in the figure, the input sparse non-local sub-features are mapped through a linear layer (linear) to obtain three key matrices Q (Query), K (Key) and V (Value), whose spatial dimensions are consistent with the input sparse non-local sub-features, that is, d×hw; at the same time, the input sparse non-local sub-features are compressed to 1 / s in the channel dimension through another linear layer (linear) where the weights are not shared. 2 The compression matrix A(Anchor) has a scale of (d / s 2 )×hw, s is the compression ratio. Through dimensional transformation, the scale of the compression matrix Anchor(A) becomes d×(hw / s 2 ); A is used as an intermediate medium for self-attention calculation to participate in matrix multiplication operation. At this time, the self-attention calculation is expressed as:
[0047] ,
[0048] ,
[0049] All output features of the channel compression self-attention module are restored according to the original spatial position to obtain the feature F of scale d×HW Cmid1 , and then after dimension conversion, we get the feature F with a scale of d×H×W Cmid2 ; Feature F Cmid2 With feature F Cin Add together to get feature F Cmid3 , feature F Cmid3 Then, after passing through the layer normalization module (LayerNorm) and the multi-layer perceptron (MLP), the feature F is obtained.Cmid4 , feature F Cmid4 With feature F Cmid3 Add together to get the output feature F of CSAB Cout ;
[0050] In terms of computational complexity, the computational complexity of the conventional self-attention mechanism is mainly composed of two matrix multiplications, namely:
[0051] ,
[0052] Where N is the product of the length and width of the input feature;
[0053] Since the window self-attention mechanism is performed on independent local window features when calculating self-attention, the computational complexity of window self-attention is expressed as:
[0054] ,
[0055] The computational complexity of the channel compression self-attention mechanism is the computational complexity of four matrix multiplications, expressed as:
[0056] ,
[0057] Therefore, it can be seen that the overall computational complexity of the self-attention mechanism based on scale compression is:
[0058] ,
[0059] According to the above computational complexity reasoning, due to and Therefore, both the window-based attention module and the scale-compression-based self-attention module can reduce the computational complexity of the original conventional self-attention module to 1 / 2 of the original computational complexity. and ;
[0060] Step 2.2. Train a lightweight deblocking enhancement neural network;
[0061] Set the training parameters, initialize the model weights, and use the Adam optimizer to train the lightweight deblocking effect enhanced neural network. The initial learning rate lr is 2e-4, and the optimizer is adjusted by gradually decaying the learning rate. The learning rate decays by 50% every 100,000 training times, and a total of 600,000 iterative trainings are performed; the loss function is MSELoss, specifically:
[0062]
[0063] Among them, I recTo reconstruct the image, I is the original image, (i, j) represents the pixel coordinates, H and W are the length and width of the image respectively;
[0064] Step 3. The compressed image to be processed is input as input data to the trained lightweight deblocking effect enhancement neural network, and a high-quality enhanced image is reconstructed by the lightweight deblocking effect enhancement neural network.
[0065] The beneficial effects of the present invention are verified by combining simulation tests below:
[0066] This embodiment uses DIV2K and Flickr2K datasets as training sets to train the lightweight deblocking enhancement neural network, uses Classic5 and LIVE1 image sets as grayscale compression image test sets, and uses LIVE1 and BSDS500 image sets as color image test sets to verify the enhancement effect of the model, and uses PSNR value and SSIM value as the evaluation criteria for the enhancement effect. In the deblocking performance test, the commonly used grayscale image enhancement models DnCNN, QGAC, FBCNN and DRUnet convolutional neural networks are used as comparative examples for grayscale compression image testing, and the color image enhancement models QGAC and FBCNN with the best convolutional neural network structure are used as comparative examples for color compression image testing. The enhancement effects of the present invention and the comparative examples on grayscale images and color images are shown in Tables 1 and 2, respectively, where the best PSNR performance is marked in bold;
[0067] Table 1
[0068]
[0069] Table 2
[0070]
[0071] It can be seen from Table 1 and Table 2 that for compressed images with compression qualities of 10, 20, 30 and 40, the restoration effect of the present invention is good and better than most of the comparative examples.
[0072] In addition, the neural network models SwinIR and ART of the Transformer structure are further used as comparative examples. As shown in Table 3, the parameter amount and Flops of the present invention and the comparative example with closer performance are statistically analyzed. The two represent the size and computational complexity of the model respectively. The smaller the parameter amount and the smaller the Flops, the less computing resources the neural network model consumes and the faster the running speed.
[0073] Table 3
[0074]
[0075] Among them, Flops is statistically calculated under the input data size of 160x160, and both SwinIR and ART in the comparative example are neural networks of Transformer structure; as can be seen from Table 3, although the present invention is a Transformer structure, its parameter amount and Flops value are smaller, which effectively reduces the training and operation cost of the neural network model of the Transformer structure, especially for ART, the Flops value is reduced by 84.13%; for the PSNR test results of the grayscale image with QF of 10 in Table 3, the performance of the present invention is similar to that of the comparative example of the convolutional neural network structure, but the parameter amount is greatly reduced, especially compared with FBCNN, the parameter amount is reduced by 96.48%. It should be noted that since the number of training data sets of the comparative examples (SwinIR and ART) of the Transformer structure in Table 3 is much larger than the number of data sets used in the present invention, and generally, the larger the training data set, the better the training effect of the model, then SwinIR and ART have natural performance advantages, and it is not fair to compare with the present invention, so the two are not included in the performance test comparative examples of Tables 1 and 2.
[0076] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.
Claims
1. A method for removing blocking effects of a lightweight neural network based on scale compression, characterized in that: The following steps are involved: Step 1. Use JPEG compression as the image codec to compress the original RGB image or the original grayscale image into a low-quality image, take the low-quality image as input, and take the original RGB image or the original grayscale image as a label to construct a training dataset; Step 2. Build a lightweight deblocking enhancement neural network and complete the training based on the training data set; The lightweight deblocking effect enhancement neural network includes: a feature extraction module, a feature enhancement module and an image reconstruction module, wherein the feature enhancement module is composed of four Transformer structure modules based on multiple attention mechanisms; In the lightweight deblocking effect enhancement neural network, the output features of the fourth Transformer structure module based on multiple attention mechanisms of the feature enhancement module are added to the input features of the feature enhancement module to obtain the output features of the feature enhancement module; the Transformer structure module based on multiple attention mechanisms includes: a self-attention module based on window partitioning, a self-attention module based on scale compression and a multi-layer perceptron, and the input feature F in Evenly split into [F Win ,F Cin ], feature F Win With feature F Cin They are used as the input of the self-attention module based on window partitioning and the self-attention module based on scale compression, respectively. The self-attention module based on window partitioning and the self-attention module based on scale compression output the feature F Wout With feature F Cout , feature F Wout With feature F Cout Concatenate to get feature F mid , feature F mid With feature F in After addition, it is input to the multi-layer perceptron, and the multi-layer perceptron outputs the feature F out ; In the window-based self-attention module, feature F Win After a layer of standardization module is used for standardization, the standardized processing result is stretched to obtain a two-dimensional feature F Wmid0 , and then divided into several local window features through local continuous window division operation, all local window features are sent to the window self-attention module in parallel; all output features of the window self-attention module are restored according to the original spatial position to obtain feature F Wmid1 , and then through dimension conversion to obtain the three-dimensional feature F Wmid2 ; Feature F Wmid2 With feature F Win Add together to get feature F Wmid3 , feature F Wmid3 After passing through another layer standardization module and multi-layer perceptron, feature F is obtained. Wmid4 , feature F Wmid4 With feature F Wmid3 Add the output feature F Wout ; In the window self-attention module, the input local window features are mapped into key matrices Q, K, and V through a linear layer, and the self-attention calculation results are obtained through matrix multiplication: Among them, Attention represents the result of self-attention calculation, d represents the dimension of feature channel, B represents the spatial position encoding matrix, Softmax represents the activation function, * represents matrix multiplication, and T represents the transposition symbol; In the self-attention module based on scale compression, feature F Cin After a layer of standardization module, the standardized processing result is decomposed into I by using uniform downsampling method with a ratio of I 2 sparse non-local sub-features, and stretch them to obtain two-dimensional sparse non-local sub-features, and then send them to the channel compression self-attention module in parallel; all output features of the channel compression self-attention module are restored according to the original spatial position to obtain feature F Cmid1 , and then through dimension conversion to obtain the three-dimensional feature F Cmid2 ; Feature F Cmid2 With feature F Cin Add together to get feature F Cmid3 , feature F Cmid3 After passing through another layer standardization module and multi-layer perceptron, feature F is obtained. Cmid4 , feature F Cmid4 With feature F Cmid3 Add the output feature F Cout ; In the channel compression self-attention module, the input sparse non-local sub-features are mapped into key matrices Q, K, and V through a linear layer; at the same time, the input sparse non-local sub-features are passed through another linear layer to obtain the compression matrix A; the compression matrix A is used as an intermediate medium for self-attention calculation to participate in the matrix multiplication operation, and the self-attention calculation result is obtained: Among them, Attention represents the result of self-attention calculation, d represents the dimension of feature channel, B represents the spatial position encoding matrix, Softmax represents the activation function, * represents matrix multiplication, and T represents the transposition symbol; Step 3. Input the compressed image to be processed as input data to the trained lightweight deblocking effect enhancement neural network, and the lightweight deblocking effect enhancement neural network outputs a high-quality enhanced image.
2. The method for removing blocking effects of a lightweight neural network based on scale compression according to claim 1, characterized in that: In the lightweight deblocking enhancement neural network, the feature extraction module consists of a convolutional layer to obtain the shallow features of the input image and input them into the feature enhancement module; The image reconstruction module consists of a convolutional layer to reconstruct a high-quality enhanced image based on the output features of the feature enhancement module.
3. The method for removing blocking effects of a lightweight neural network based on scale compression according to claim 1, characterized in that: The training uses MSELoss as the loss function to complete the training of the lightweight deblocking effect enhanced neural network.
