Distributed region-of-interest image compression method

Abstract

The invention discloses a distributed region-of-interest image compression method, and mainly solves the problems that the training speed of an existing image compression algorithm is low and distinguished feature extraction is not carried out according to the importance of different region features in the prior art. According to the implementation scheme, the method comprises the following steps: selecting a landsat remote sensing image data set from a public website, and dividing the landsat remote sensing image data set into a training set and a test set; a distributed training strategy is introduced into an image compression algorithm; constructing a multi-layer hyper-prior image compression and decompression network comprising a region-of-interest detection module, an analysis transformation sub-network, a synthesis transformation sub-network, a hyper-prior analysis sub-network and a hyper-prior synthesis transformation sub-network; training a multi-layer hyper-prior image compression and decompression network by using the training set; and inputting the test set into the trained network to obtain a compressed and decompressed image. Reasonable and efficient compression can be performed according to the feature importance of different regions of the image, the training speed is improved, the code rate is saved, and the method can be used for a scene of performing fine compression on important regions in the image.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to a method for compressing an image of a region of interest, which can be used for finely compressing an important region in an image. Background Art

[0002] With the continuous reform of parallel computing of hardware devices such as GPU, the field of artificial intelligence has achieved new development, and image processing technology based on deep learning has also become an important method for multimedia data processing. Image compression technology based on region of interest ROI can better meet people's visual needs. When there are target individuals in the image, different compression strategies are adopted in different regions to achieve higher compression efficiency and better visual reconstruction effects. The amount of information contained in the detailed information of the image is very large, and it is difficult to have good compression performance when the wireless channel environment is relatively limited. Therefore, in order to improve people's visual sensory experience of compressed and reconstructed images, it is necessary to perform differential compression on the region of interest and the background area. In addition, with the continuous increase of existing image compression network training data sets and model parameter quantities, the network training speed is limited by hardware resources. The use of distributed training strategies can greatly improve the network training speed and solve the problem of slow training speed of a single node.

[0003] Cai et al. proposed a method for ROI image compression in their paper "End-to-End Optimized ROI Image Compression" (Published as a conference paper at CVPR 2021). The implementation steps are: (1) Obtain a binary mask of the image ROI through an offline network; (2) Multiply the original image with the obtained binary mask of the ROI to obtain the ROI image; (3) Input the ROI image into the analysis transformation module to obtain the latent map of the image; (2) Input the latent features into the super analysis module to obtain the distribution of the image, and the output defaults to the standard Gaussian distribution; (3) Input the latent map of the image into the entropy model, and encode and decode according to the image distribution obtained by the super analysis transformation module; (4) Pass the decompressed feature map through the synthesis transformation module to obtain the restored image. The disadvantage of this method is that since the binary mask of the ROI needs to be generated by an offline network first, and then the ROI image is obtained for image compression, it cannot realize adaptive binary masking, resulting in a low network training speed.

[0004] Patent document with application number CN202410318128.6 discloses an "image compression system, image compression method, storage medium and chip", and the implementation steps of the image compression method are as follows: (1) inputting the image into a nonlinear transformation network to extract the potential representation features of the image; (2) inputting the potential representation features into a super priori transformation network to obtain the side information of the features; (3) using a second encoder to encode the side information to generate a second code stream; (4) inputting the second code stream into a second decoder to obtain reconstructed feature side information; (5) inputting the feature side information into a super priori inverse transformation network to obtain the variance parameter and mean parameter of the potential representation features; (6) inputting the variance parameter, mean parameter and potential representation features into a context model to reconstruct the Gaussian distribution parameters of the potential representation features of the image; (7) using a first encoder to encode the potential representation features according to the Gaussian distribution parameters to generate a first code stream; (8) decoding the first code stream through a first decoder to obtain the potential representation features; (9) inputting the potential representation features into a nonlinear inverse transformation network to obtain a reconstructed image. Since this method treats all features equally when using the hyper-prior transform network for image feature extraction, it does not introduce a modeling mechanism for the importance of features in different regions. In reality, the impact of image features in different regions on visual perception and the demand for bit rate allocation are different, which leads to unreasonable bit rate allocation and affects the image compression quality. Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the present invention proposes a method for compressing image regions of interest based on a distributed multi-layer hyper-prior network. One of its purposes is to improve the network training speed by jointly compressing the region of interest detection module and the image compression network; the second purpose is to save bit rate and improve image compression quality by reasonably and efficiently compressing the importance of features in different regions of the image.

[0006] To achieve the above object, the technical solution of the present invention comprises the following steps:

[0007] (1) Select the Landsat remote sensing image dataset from the public website and divide it into a training set and a test set in a ratio of 8:1;

[0008] (2) Construct a multi-layer super-prior image compression and decompression network:

[0009] (2a) establishing a region of interest detection module including a residual submodule, a square and strip pooling submodule, and a quantization submodule;

[0010] (2b) Selecting the existing analysis transformation sub-network, synthesis transformation sub-network, two super-prior analysis sub-networks, and two super-prior synthesis transformation sub-networks;

[0011] (2c) The region of interest detection module is connected in parallel with the analysis transformation sub-network, and then cascaded with the first super-prior analysis sub-network, the second super-prior analysis sub-network, the first super-prior synthesis transformation sub-network, the second super-prior synthesis transformation sub-network, and the synthesis transformation sub-network in sequence to form a multi-layer super-prior image compression and decompression network, and the existing super-prior image compression network loss function is used as the loss function Loss of the network;

[0012] (3) Training of multi-layer hyper-prior image compression and decompression networks:

[0013] (3a) Introducing a distributed training strategy, that is, using two nodes, selecting one of the nodes as the main node, and setting up a complete multi-layer hyper-prior image compression and decompression network on each node;

[0014] (3b) Input the Landsat remote sensing image training set into each multi-layer hyper-prior image compression and decompression network, and set the training data batch Distribute the training tasks to the four GPUs of the two nodes to parallelly calculate the gradients of each node, then send the gradients of each node to the master node for accumulation, and then send the accumulated gradients of the master node to other nodes;

[0015] (3c) Each node uses the gradient descent method to iteratively update the parameters of each layer of the multi-layer super-prior image compression and decompression network in the reverse direction of the gradient to reduce the loss function Loss;

[0016] (3d) Repeat steps (3b) to (3c) until the loss function Loss of the multi-layer super-prior image compression and decompression network converges, thereby obtaining a trained multi-layer super-prior image compression and decompression network;

[0017] (4) The image to be compressed is input into a trained multi-layer super-prior image compression and decompression network for compression and decompression to obtain a decompressed restored image.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] First, the present invention introduces a distributed training strategy and utilizes the storage and computing resources of two nodes to jointly train the network model, which can effectively improve the training speed while ensuring the quality of the reconstructed image, overcome the problem of slow training speed of the existing image compression algorithm, and make the present invention have a wider application scenario for remote sensing image compression.

[0020] Second, the present invention constructs a multi-layer super-prior image compression and decompression network region of interest detection module, which can

[0021] By reasonably and efficiently compressing the importance of features in different areas of the image, the bit rate is saved, thereby achieving better reconstruction performance of the region of interest and improving the image compression quality. The problem in the prior art that the system performs unified feature extraction on the entire image when there is no region of interest is overcome, so that the present invention can show advantages when different areas in the image have different compression quality requirements, and can also maintain good performance when compressing complex images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the implementation of the present invention;

[0023] Figure 2 It is a schematic diagram of the structure of a multi-layer super-prior image compression and decompression network constructed in the present invention;

[0024] Figure 3 yes Figure 2 Schematic diagram of the structure of the region of interest detection module in .

[0025] Figure 4 yes Figure 3 Schematic diagram of the residual submodule structure in .

[0026] Figure 5 yes Figure 3 Schematic diagram of the structure of the square and strip pooling submodules in .

[0027] Figure 6 yes Figure 3 Schematic diagram of the quantization submodule structure in . DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0029] Reference Figure 1 , the implementation steps of this example are as follows:

[0030] Step 1: Select the data set and divide it into training set and test set.

[0031] 1.1) Obtain at least 5,000 visible color remote sensing images taken by the Landsat-8OLI / TIRS satellite with a resolution of 1,000 × 1,000 from public websites. In order to make the network generalizable, select images of water, ice and snow, land, clouds, etc. from the dataset;

[0032] 1.2) Divide the selected dataset into training set and test set in a ratio of 8:1.

[0033] Step 2: Establish a region of interest detection module including a residual submodule, a square and strip pooling submodule, and a quantization submodule.

[0034] Reference Figure 3 , the implementation of this step is as follows:

[0035] 2.1) Establish a residual submodule consisting of three residual blocks connected in series, such as Figure 4 As shown in the figure, each residual block includes two convolutional layers and one activation layer, where the first convolutional layer, the activation layer, and the second convolutional layer are connected in series in sequence, and the input of the first convolutional layer is added to the output of the second convolutional layer to obtain the output of the residual submodule; the convolution kernel size of each convolutional layer is 3*3, and the activation layer is implemented using the ReLU function to detect the features of the input;

[0036] 2.2) Establish square and strip pooling submodules:

[0037] 2.2.1) Establish a feature extraction layer consisting of a horizontal strip pooling layer, a vertical strip pooling layer, and an expansion convolution layer in parallel, where:

[0038] The horizontal strip pooling layer is used to capture the long-range context along the horizontal space. It uses a strip pooling window to perform pooling along the vertical direction, and then uses 1D convolution to expand the feature map in the left and right directions respectively. Given a 2D input tensor X∈R H×W ,The horizontal pooling formula is as follows:

[0039]

[0040] Among them, x i,j Represents the element in the i-th row and j-th column of the two-dimensional input tensor X, H represents the height of the tensor, W represents the width of the tensor, 0≤i <H,0≤j<W, Indicates the result of horizontal strip pooling;

[0041] The vertical strip pooling layer is used to capture the remote context along the vertical space. It uses a strip pooling window to perform pooling along the horizontal direction, and then uses 1D convolution to expand the feature map in the up and down directions respectively. Given a 2D input tensor X∈R H×W ,The vertical pooling formula is as follows:

[0042] in, Represents the result of vertical strip pooling;

[0043] The dilated convolution layer has a convolution kernel size of 3*3 and a dilation rate of 2;

[0044] 2.2.2) Establish a convolution layer with a convolution kernel size of 1*1;

[0045] 2.2.3) Establish an activation layer, which is implemented using the sigmoid function to calculate the probability of the input sample being detected as each class.

[0046] 2.2.4) Connect the feature extraction layer, convolution layer, and activation layer in series, and then multiply the input of the feature extraction layer by the output of the activation layer to obtain the square and strip pooling submodules, such as Figure 5 As shown;

[0047] 2.3) Establish a quantization submodule consisting of a 3*3 convolutional layer, an activation layer, and a rounding function layer connected in series, such as Figure 6 As shown, the activation layer uses the sigmoid function to achieve normalization, the rounding function layer rounds the values ​​between 0 and 1 to obtain a binary feature map, and the convolution kernel size of the convolution layer is;

[0048] 2.4) A convolution layer with a convolution kernel size of 2*2, a square and strip pooling submodule, a convolution layer with a convolution kernel size of 5*5, and a residual submodule are connected in series in sequence, then connected in parallel with a convolution layer with a convolution kernel size of 1*1, and then connected in series with the quantization submodule to form a region of interest detection module.

[0049] Step 3: Build the analysis transformation sub-network.

[0050] 3.1) Select four convolutional layers with a kernel size of 5*5 and a convolution stride of 2, one boundary padding layer with a padding size of 2, three boundary padding layers with a padding size of 1, and three GDN layers implemented by generalized normalization;

[0051] 3.2) Connect the boundary filling layer with a filling size of 2, the first convolutional layer, the first GDN layer, the first boundary filling layer with a filling size of 1, the second convolutional layer, the second GDN layer, the second boundary filling layer with a filling size of 1, the third convolutional layer, the third GDN layer, the third boundary filling layer with a filling size of 1, and the fourth convolutional layer in sequence to form an analysis transformation subnetwork.

[0052] Step 4: Build the synthetic transformation sub-network.

[0053] 4.1) Select three deconvolution layers with a kernel size of 5*5 and a convolution step size of 2, three boundary padding layers with a padding size of 2, and three IGDN layers implemented by inverse generalized normalization;

[0054] 4.2) The first boundary filling layer, the first deconvolution layer, the first IGND layer, the second boundary filling layer, the second deconvolution layer, the second IGDN layer, the third boundary filling layer, the third deconvolution layer, and the third IGDN layer are sequentially connected to form a synthetic transformation subnetwork.

[0055] Step 5: Build two identical hyper-prior analysis transformation sub-networks.

[0056] 5.1) Select a convolutional layer with a kernel size of 3*3 and a convolution step size of 1, three convolutional layers with a kernel size of 1*1 and a step size of 1, and two ReLU activation layers;

[0057] 5.2) The convolution layer with a convolution kernel size of 3*3, the first convolution layer with a convolution kernel size of 1*1, the first ReLU activation layer, the second convolution layer with a convolution kernel size of 1*1, the second ReLU activation layer, and the third convolution layer with a convolution kernel size of 1*1 are connected in sequence to form a super-prior analysis transformation subnetwork.

[0058] Step 6: Build two identical super-prior synthesis transformation sub-networks.

[0059] 6.1) Select three deconvolution layers with a convolution kernel size of 1*1, one deconvolution layer with a convolution kernel size of 3*3, and two ReLU activation layers;

[0060] 6.2) Connect the first deconvolution layer with a convolution kernel size of 1*1, the second deconvolution layer with a convolution kernel size of 1*1, the ReLU activation layer, the third deconvolution layer with a convolution kernel size of 1*1, the ReLU activation layer, and the deconvolution layer with a convolution kernel size of 3*3 in sequence to form a super-prior synthetic transformation sub-network.

[0061] Step 7: construct a multi-layer hyper-prior image compression and decompression network.

[0062] 7.1) The region of interest detection module constructed in step 2 is connected in parallel with the analysis transformation sub-network constructed in step 3, and then connected in cascade with the two super-prior analysis sub-networks constructed in step 5, the two super-prior synthesis transformation sub-networks constructed in step 6, and the synthesis transformation sub-network constructed in step 4 to form a multi-layer super-prior image compression and decompression network, such as Figure 2 As shown;

[0063] 7.2) The existing super-prior image compression network loss function is used as the loss function Loss of the network, which is expressed as follows:

[0064]

[0065] Among them, R represents the compression bit rate of the overall framework, D represents the distortion between the reconstructed image and the original image, and λ is a hyperparameter used to weigh the bit rate and distortion. is the quantized potential representation, are the modeling parameters output by the first hyper-prior analysis subnetwork, represents the modeling parameters of the second super-prior analysis subnetwork output, d(·) represents the distortion metric, which is measured using mean square error, x represents the original image, represents the reconstructed image, E represents the calculation of cross entropy, p x is the distribution of the potential representation of the image.

[0066] Step 8: Train the multi-layer hyper-prior image compression and decompression network.

[0067] 8.1) Adopt a distributed training strategy, use two nodes for training, select one of the nodes as the main node, and set up a complete multi-layer hyper-prior image compression and decompression network on each node;

[0068] 8.2) Input the training set images into the multi-layer hyper-prior image compression and decompression network, and set the training data batch

[0069] Distribute the training task to the four GPUs of the two nodes and parallelize the gradients of each node. On the working nodes n=1,2, use the current model parameters w t and from the training data batch The data sampled in b n The calculated loss function For each sample data b n , calculate the gradient of the loss function Then All sample data in b n The gradients of the tth iteration are averaged to obtain the local gradient of the working node n

[0070]

[0071] 8.3) Send each gradient to the master node for accumulation, and then send the accumulated gradient of the master node to other nodes. Each node iteratively updates the parameters of each layer of the multi-layer super-prior image compression and decompression network in the opposite direction of the gradient to reduce the loss function Loss. The parameter update formula is as follows:

[0072]

[0073] in is the gradient, is the network model parameter of the tth iteration, is the network model parameter of the t+1 iteration, n is the node, and α is the set learning rate. Model parameters of node n Based on the current gradient Adjustment, α controls the magnitude of each update. By subtracting a value proportional to the gradient from the current parameters, the model parameters are updated in the direction of reducing the loss function, thereby gradually optimizing the network performance.

[0074] 8.4) Repeat steps 8.2) and 8.3) until the loss function Loss of the multi-layer super-prior image compression and decompression network converges to obtain a trained multi-layer super-prior image compression and decompression network.

[0075] Step 9: compress and decompress the remote sensing image.

[0076] The test set images are input into the trained multi-layer super-prior image compression and decompression network. The entropy model parameters are obtained through two super-prior analysis transformation sub-networks, and the compressed code stream of the image is output. Then, the pixels of the original remote sensing image are restored through the synthesis transformation sub-network to complete the image compression and decompression.

[0077] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0078] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the order of the step numbers is not limited.

Claims

1. A method for compressing region of interest images based on a distributed multi-layer hyper-prior network, characterized in that: The steps include: (1) Select the Landsat remote sensing image dataset from the public website and divide it into a training set and a test set in a ratio of 8:1; (2) Construct a multi-layer super-prior image compression and decompression network: (2a) establishing a region of interest detection module including a residual submodule, a square and strip pooling submodule, and a quantization submodule; (2b) Selecting the existing analysis transformation sub-network, synthesis transformation sub-network, two super-prior analysis sub-networks, and two super-prior synthesis transformation sub-networks; (2c) The region of interest detection module is connected in parallel with the analysis transformation sub-network, and then cascaded with the first super-prior analysis sub-network, the second super-prior analysis sub-network, the first super-prior synthesis transformation sub-network, the second super-prior synthesis transformation sub-network, and the synthesis transformation sub-network in sequence to form a multi-layer super-prior image compression and decompression network, and the existing super-prior image compression network loss function is used as the loss function Loss of the network; (3) Training of multi-layer hyper-prior image compression and decompression networks: (3a) Introducing a distributed training strategy, that is, using two nodes, selecting one of the nodes as the main node, and setting up a complete multi-layer hyper-prior image compression and decompression network on each node; (3b) Input the Landsat remote sensing image training set into each multi-layer hyper-prior image compression and decompression network, and set the training data batch Distribute the training tasks to the four GPUs of the two nodes to parallelly calculate the gradients of each node, then send the gradients of each node to the master node for accumulation, and then send the accumulated gradients of the master node to other nodes; (3c) Each node uses the gradient descent method to iteratively update the parameters of each layer of the multi-layer super-prior image compression and decompression network in the reverse direction of the gradient to reduce the loss function Loss; (3d) Repeat steps (3b) to (3c) until the loss function Loss of the multi-layer super-prior image compression and decompression network converges, thereby obtaining a trained multi-layer super-prior image compression and decompression network; (4) The images of the test set are input into the trained multi-layer super-prior image compression and decompression network for compression and decompression to obtain the decompressed restored image.

2. The method according to claim 1, characterized in that The structural parameters of each submodule constituting the region of interest detection module in step (2a) are as follows: The residual submodule is used to improve the feature extraction capability, and includes three residual blocks connected in series. Each residual block includes two convolutional layers and an activation layer. The first convolutional layer, the activation layer, and the second convolutional layer are connected in series in series. The input of the first convolutional layer is then added to the output of the second convolutional layer to obtain the output of the residual submodule. The convolution kernel size of each convolutional layer is 3*3, and the activation layer is implemented using the ReLU function. The square and strip pooling submodule is used to aggregate global and local context information and effectively extract features of the region of interest. It includes a feature extraction layer, a convolution layer, and an activation layer. The feature extraction layer is composed of a horizontal strip pooling layer, a vertical strip pooling layer, and an expansion convolution layer in parallel. The convolution kernel size of the convolution layer is 1*1, and the activation layer is implemented using a sigmoid function. The feature extraction layer is connected in series with the convolution layer and the activation layer in sequence, and the input of the feature extraction layer is added to the output of the activation layer to obtain the output of the square and strip pooling submodule; The quantization submodule is used to obtain a binary mask of the region of interest, which is composed of a convolution layer, an activation layer, and a rounding function layer connected in series in sequence. The convolution kernel size of the convolution layer is 3*3, and the activation layer is implemented using a sigmoid function.

3. The method according to claim 1, characterized in that The analysis transformation sub-network, synthesis transformation sub-network, two super-prior analysis sub-networks, and two super-prior synthesis transformation sub-networks selected in step (2b) have the following structural parameters: The analysis transformation subnetwork is used to extract the low-dimensional potential representation of the image. It includes four boundary filling layers, four convolution layers and three GDN layers. Its structural relationship is: first boundary filling layer→first convolution layer→first GDN layer→second boundary filling layer→second convolution layer→second GDN layer→third boundary filling layer→third convolution layer→third GDN layer→fourth boundary filling layer→fourth convolution layer; the size of the convolution kernel of the first to fourth convolution layers is 5*5, and the convolution step is 2; the first to third GDN layers are all implemented by generalized normalization; The synthetic transformation subnetwork is used to restore the potential representation to an approximate original image. It includes three boundary filling layers, three deconvolution layers and three IGDN layers. Its structural relationship is the first boundary filling layer→the first deconvolution layer→the first IGDN layer→the second boundary filling layer→the second deconvolution layer→the second IGDN layer→the third boundary filling layer→the third deconvolution layer→the third IGDN layer; the convolution kernel size of the first to third deconvolution layers is 5*5, and the step size of the deconvolution layer is 2; Each of the super-prior analysis sub-networks is used to model the statistical characteristics of the potential representation, and includes four convolutional layers and two ReLU activation layers, and its structural relationship is: the first convolutional layer → the second convolutional layer → the first ReLU activation layer → the third convolutional layer → the second ReLU activation layer → the fourth convolutional layer; the size of the convolution kernel of the first convolutional layer is 3*3, and the sizes of the convolution kernels of the second to fourth convolutional layers are all 1*1; Each of the super-prior synthetic transformation sub-networks is used to further model the statistical characteristics of the potential representation during decoding. It includes four deconvolution layers and two ReLU activation layers, and its structural relationship is: first deconvolution layer→second deconvolution layer→ReLU activation layer→third deconvolution layer→ReLU activation layer→fourth deconvolution layer; the convolution kernel size of the first to third deconvolution layers is 1*1, and the deconvolution kernel size of the fourth deconvolution layer is 3*3.

4. The method according to claim 1, characterized in that The loss function Loss of the multi-layer super-prior image compression and decompression network set in step (2c) is expressed as follows: Among them, R represents the compression bit rate of the overall framework, D represents the distortion between the reconstructed image and the original image, and λ is a hyperparameter used to weigh the bit rate and distortion. is the quantized potential representation, are the modeling parameters output by the first hyper-prior analysis subnetwork, represents the modeling parameters of the second super-prior analysis subnetwork output, d(·) represents the distortion metric, which is measured using mean square error, x represents the original image, represents the reconstructed image, E represents the calculation of cross entropy, p x is the distribution of the potential representation of the image.

5. The method according to claim 1, characterized in that: In step (3b), the gradient of each node is calculated in parallel on the working nodes n=1,2, using the current model parameters w t and from the training data batch The data sampled in b n The calculated loss function For each sample data b n , calculate the gradient of the loss function Then All sample data in b n The gradients of the tth iteration are averaged to obtain the local gradient of the working node n 6. The method according to claim 1, characterized in that In step (3c), each node uses the gradient descent method to iteratively update the parameters of each layer of the multi-layer super-prior image compression and decompression network in the reverse direction of the gradient. The formula is as follows: in is the gradient, is the network model parameter of the tth iteration, is the network model parameter for t+1 iterations, n is the node, and α is the set learning rate.

7. The method according to claim 1, characterized in that In step (4), the image to be compressed is input into a trained multi-layer super-prior image compression and decompression network for compression and decompression. The entropy model parameters are first obtained through two super-prior analysis transformation sub-networks in the network, and the compressed code stream of the image is output. The compressed code stream is then restored to the pixels of the original remote sensing image through the synthesis transformation sub-network to complete the image compression and decompression.

Images