Image compression method based on distributed region of interest

Abstract

This invention discloses a distributed region-of-interest (ROI) image compression method, primarily addressing the slow training speed of existing image compression algorithms and the lack of differentiated feature extraction based on the importance of different regions. The implementation involves: selecting the Landsat remote sensing image dataset from a publicly available website and dividing it into training and test sets; introducing a distributed training strategy into the image compression algorithm; constructing a multi-layered hyperprior image compression and decompression network comprising a ROI detection module, an analysis transform sub-network, a synthesis transform sub-network, a hyperprior analysis sub-network, and a hyperprior synthesis transform sub-network; training the multi-layered hyperprior image compression and decompression network using the training set; and inputting the test set into the trained network to obtain the compressed and decompressed images. This invention enables reasonable and efficient compression based on the importance of different regions of the image, improving training speed and saving bitrate, and can be used for scenarios requiring fine-grained compression of important regions in images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to a method for region-of-interest (ROI) image compression. It can be used for scenarios requiring fine compression of important regions in an image. Background Technology

[0002] With the continuous advancements in parallel computing technology using hardware such as GPUs, the field of artificial intelligence has seen new developments, and deep learning-based image processing technology has become an important method for multimedia data processing. Image compression technology based on Regions of Interest (ROIs) can better meet people's visual needs. When different regions contain target individuals in the image, different compression strategies are adopted to achieve higher compression efficiency and better visual reconstruction results. Images contain a vast amount of detailed information, and in relatively limited wireless channel environments, achieving good compression performance is difficult. Therefore, to improve the visual experience of compressed and reconstructed images, differentiated compression of ROIs and background regions is necessary. Furthermore, as the training datasets and model parameters of existing image compression networks continue to increase, the training speed of the network is limited by hardware resources. Adopting a distributed training strategy can greatly improve the network training speed and solve the problem of slow training speed of individual nodes.

[0003] Cai et al. proposed a region-of-interest (ROI) image compression method in their paper "End-to-End Optimized ROI Image Compression" (Published as a conference paper at CVPR 2021). The implementation steps are: (1) Obtain the binary mask of the ROI through an offline network; (2) Multiply the original image with the obtained ROI binary mask to obtain the ROI image; (3) Input the ROI image into the analysis and transformation module to obtain the latent mapping of the image; (2) Input the latent features into the hyper-analysis module to obtain the image distribution, with the output defaulting to a standard Gaussian distribution; (3) Input the latent mapping of the image into the entropy model, and encode and decode according to the image distribution obtained from the hyper-analysis transformation module; (4) Use the synthesis transformation module to obtain the restored image from the decompressed feature map. The drawback of this method is that, since obtaining the ROI binary mask requires first generating it using an offline network, and then compressing the ROI image, adaptive binary masks cannot be achieved, resulting in a low network training speed.

[0004] Patent document with application number CN202410318128.6 discloses an "image compression system, image compression method, storage medium and chip". The implementation steps of the image compression method are as follows: (1) Input the image into a nonlinear transform network to extract the latent representation features of the image; (2) Input the latent representation features into a super-prior transform network to obtain the side information of the features; (3) Encode the side information using a second encoder to generate a second bitstream; (4) Input the second bitstream into a second decoder to obtain the reconstructed feature side information; (5) Input the feature side information into a super-prior inverse transform network to obtain the variance parameter and mean parameter of the latent representation features; (6) Input the variance parameter, mean parameter and latent representation features into a context model to reconstruct the Gaussian distribution parameters of the latent representation features of the image; (7) Encode the latent representation features using a first encoder according to the Gaussian distribution parameters to generate a first bitstream; (8) Decode the first bitstream through a first decoder to obtain the latent representation features; (9) Input the latent representation features into a nonlinear inverse transform network to obtain the reconstructed image. This method treats all features equally when using the super prior transform network for image feature extraction, and does not introduce a modeling mechanism for the importance of features in different regions. In reality, the impact of image features in different regions on visual perception and the requirements for bitrate allocation are different, which leads to unreasonable bitrate allocation and affects the image compression quality. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention proposes a region-of-interest (ROI) image compression method based on a distributed multi-layer super-prior network. One objective is to improve the network training speed by jointly compressing the ROI detection module and the image compression network. Another objective is to save bitrate and improve image compression quality by rationally and efficiently compressing the features of different regions of the image according to their importance.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0007] (1) Select the Landsat remote sensing image dataset from a public website and divide it into training and test sets in an 8:1 ratio;

[0008] (2) Constructing a multi-layer hyperprior image compression and decompression network:

[0009] (2a) Establish a region of interest detection module including a residual submodule, a square and bar pooling submodule, and a quantization submodule;

[0010] (2b) Select the existing analytical transform subnetwork, synthetic transform subnetwork, two prior analytical subnetworks, and two prior synthetic transform subnetworks;

[0011] (2c) Connect the region of interest detection module in parallel with the analysis and transformation sub-network, and then cascade it with the first super-prior analysis sub-network, the second super-prior analysis sub-network, the first super-prior synthesis transformation sub-network, the second super-prior synthesis transformation sub-network, and the synthesis transformation sub-network in sequence to form a multi-layer super-prior image compression and decompression network, and use the existing super-prior image compression network loss function as the loss function Loss of this network.

[0012] (3) Training the multi-layer hyperprior image compression and decompression network:

[0013] (3a) Introduce a distributed training strategy, that is, use two nodes, select one of the nodes as the master node, and set up a complete multi-layer super-prior image compression and decompression network for each node;

[0014] (3b) Input the Landsat remote sensing image training set into each multi-layer hyperprior image compression and decompression network, and set the training data batch size. The training task is distributed to four GPUs on two nodes to compute the gradient of each node in parallel. The gradients of each node are then sent to the master node for accumulation. Finally, the gradients accumulated by the master node are sent to the other nodes.

[0015] (3c) Each node uses the gradient descent method to iteratively update the parameters of each layer of the multilayer super-prior image compression and decompression network along the opposite direction of the gradient, so as to reduce the loss function Loss.

[0016] (3d) Repeat steps (3b) to (3c) until the loss function Loss of the multilayer super prior image compression and decompression network converges, and the trained multilayer super prior image compression and decompression network is obtained.

[0017] (4) Input the image to be compressed into the trained multilayer super-prior image compression and decompression network for compression and decompression to obtain the decompressed restored image.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] First, by introducing a distributed training strategy, this invention utilizes the storage and computing resources of two nodes to jointly train the network model. This effectively improves the training speed while ensuring the quality of the reconstructed image, overcoming the problem of slow training speed in existing image compression algorithms. This makes the application scenarios of this invention for remote sensing image compression more extensive.

[0020] Secondly, this invention, by constructing a multi-layered hyper-prior image compression and decompression network region of interest detection module, can...

[0021] By rationally and efficiently compressing the features of different regions of an image according to their importance, the bit rate is saved, thereby achieving better region of interest reconstruction performance and improving image compression quality. This overcomes the problem in the prior art that the system performs uniform feature extraction on the entire image when there is no region of interest. As a result, this invention can show advantages when different regions of an image have different compression quality requirements, and can also maintain good performance when compressing complex images. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0023] Figure 2 This is a schematic diagram of the multi-layer super-prior image compression and decompression network structure constructed in this invention;

[0024] Figure 3 yes Figure 2 A schematic diagram of the region of interest detection module in the image.

[0025] Figure 4 yes Figure 3 A schematic diagram of the residual submodule structure.

[0026] Figure 5 yes Figure 3 The diagram shows the structure of the square and bar pooling submodules.

[0027] Figure 6 yes Figure 3 A schematic diagram of the quantization submodule structure. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] Reference Figure 1 The implementation steps for this example are as follows:

[0030] Step 1: Select the dataset and divide it into training and test sets.

[0031] 1.1) Select at least 5,000 visible light color remote sensing images taken by Landsat-8 OLI / TIRS satellite with a resolution of 1,000×1,000 from public websites. In order to make the network generalizable, select images of types such as water, ice and snow, land, and clouds from the dataset.

[0032] 1.2) Divide the selected dataset into training and test sets in an 8:1 ratio.

[0033] Step 2: Establish a region of interest detection module that includes a residual submodule, square and bar pooling submodules, and a quantization submodule.

[0034] Reference Figure 3 The implementation of this step is as follows:

[0035] 2.1) Establish a residual submodule consisting of three residual blocks connected in series, such as... Figure 4 As shown, each residual block includes two convolutional layers and one activation layer. The first convolutional layer, the activation layer, and the second convolutional layer are connected in series. The input of the first convolutional layer is then added to the output of the second convolutional layer to obtain the output of the residual sub-module. The kernel size of each convolutional layer is 3*3. The activation layer is implemented using the ReLU function to detect the input features.

[0036] 2.2) Establish square and bar pooling submodules:

[0037] 2.2.1) Construct a feature extraction layer consisting of horizontal strip pooling layers, vertical strip pooling layers, and dilated convolutional layers connected in parallel, wherein:

[0038] The horizontal strip pooling layer is used to capture remote context along the horizontal space. It performs pooling along the vertical direction using a strip pooling window, and then uses 1D convolution to expand the feature map in the left and right directions, given a two-dimensional input tensor X∈R. H×W The horizontal pooling formula is as follows:

[0039]

[0040] Where, x i,j Let H represent the element in the i-th row and j-th column of the two-dimensional input tensor X, and let W represent the height and width of the tensor, where 0 ≤ i ≤ j. <H,0≤j<W, This represents the result of horizontal strip pooling;

[0041] The vertical strip pooling layer is used to capture long-range context along the vertical space. It performs pooling along the horizontal direction using a strip pooling window, and then uses 1D convolution to expand the feature map in the vertical direction, given a 2D input tensor X∈R. H×W The vertical pooling formula is as follows:

[0042] in, This represents the result of vertical strip pooling;

[0043] The dilated convolutional layer has a kernel size of 3*3 and a dilation rate of 2.

[0044] 2.2.2) Convolutional layers are constructed with kernels of size 1*1;

[0045] 2.2.3) Establish an activation layer, which is implemented using the sigmoid function to calculate the probability that the input sample is detected as each class.

[0046] 2.2.4) The feature extraction layer, convolutional layer, and activation layer are sequentially connected in series. Then, the input of the feature extraction layer is multiplied by the output of the activation layer to obtain square and bar pooling sub-modules, as shown below. Figure 5 As shown;

[0047] 2.3) Establish a quantization submodule consisting of a 3x3 convolutional layer, an activation layer, and a rounding function layer connected in series, such as... Figure 6 As shown, the activation layer uses the sigmoid function for normalization, the floor function layer rounds the values ​​between 0 and 1 to obtain a binary feature map, and the kernel size of the convolutional layer is ;

[0048] 2.4) A convolutional layer with a kernel size of 2*2, square and bar pooling sub-modules, a convolutional layer with a kernel size of 5*5, and a residual sub-module are connected in series, then connected in parallel with a convolutional layer with a kernel size of 1*1, and then connected in series with a quantization sub-module to form a region of interest detection module.

[0049] Step 3: Build and analyze the transformation subnetwork.

[0050] 3.1) Select four convolutional layers with a kernel size of 5*5 and a stride of 2, one boundary padding layer with a padding size of 2, three boundary padding layers with a padding size of 1, and three GDN layers implemented by generalized normalization.

[0051] 3.2) The boundary padding layer with a padding size of 2, the first convolutional layer, the first GDN layer, the first boundary padding layer with a padding size of 1, the second convolutional layer, the second GDN layer, the second boundary padding layer with a padding size of 1, the third convolutional layer, the third GDN layer, the third boundary padding layer with a padding size of 1, and the fourth convolutional layer are connected in sequence to form the analysis transformation sub-network.

[0052] Step 4: Construct the synthetic transformation subnetwork.

[0053] 4.1) Select three deconvolutional layers with a kernel size of 5*5 and a stride of 2, three boundary padding layers with a padding size of 2, and three IGDN layers implemented by inverse generalized normalization.

[0054] 4.2) The first boundary padding layer, the first deconvolution layer, the first IGND layer, the second boundary padding layer, the second deconvolution layer, the second IGDN layer, the third boundary padding layer, the third deconvolution layer, and the third IGDN layer are sequentially connected to form a synthetic transform subnetwork.

[0055] Step 5: Construct two identical super-prior analysis transformation subnetworks.

[0056] 5.1) Select one convolutional layer with a kernel size of 3*3 and a stride of 1, three convolutional layers with a kernel size of 1*1 and a stride of 1, and two ReLU activation layers;

[0057] 5.2) A super-prior analysis transform subnetwork is formed by sequentially connecting a convolutional layer with a kernel size of 3*3, a first convolutional layer with a kernel size of 1*1, a first ReLU activation layer, a second convolutional layer with a kernel size of 1*1, a second ReLU activation layer, and a third convolutional layer with a kernel size of 1*1.

[0058] Step 6: Construct two identical super-prior synthetic transform subnetworks.

[0059] 6.1) Select three deconvolutional layers with a kernel size of 1*1, one deconvolutional layer with a kernel size of 3*3, and two ReLU activation layers;

[0060] 6.2) The first deconvolutional layer with a kernel size of 1*1, the second deconvolutional layer with a kernel size of 1*1, the ReLU activation layer, the third deconvolutional layer with a kernel size of 1*1, the ReLU activation layer, and the deconvolutional layer with a kernel size of 3*3 are sequentially connected to form a super-prior synthesis transform subnetwork.

[0061] Step 7: Construct a multi-layer hyperprior image compression and decompression network.

[0062] 7.1) The region of interest detection module constructed in step 2 is connected in parallel with the analysis and transformation sub-network built in step 3, and then cascaded with the two advanced prior analysis sub-networks built in step 5, the two advanced prior synthesis transformation sub-networks built in step 6, and the synthesis transformation sub-network built in step 4 to form a multi-layer advanced prior image compression and decompression network, as follows: Figure 2 As shown;

[0063] 7.2) The existing loss function of the hyperprior image compression network is used as the loss function Loss of this network, as follows:

[0064]

[0065] Where R represents the overall compression bitrate, D represents the distortion between the reconstructed image and the original image, and λ is a hyperparameter used to balance bitrate and distortion. For the quantized potential representation, The modeling parameters output by the first hyperprior analysis subnetwork are... Let represent the modeling parameters output by the second hyperprior analysis subnetwork, d(·) represent the distortion metric, which is measured using mean squared error, and x represent the original image. Let p represent the reconstructed image, E represent the calculation of cross-entropy, and p represent the reconstructed image. x represents the distribution of the latent representation of the image.

[0066] Step 8: Train the multi-layer hyperprior image compression and decompression network.

[0067] 8.1) A distributed training strategy is adopted, using two nodes for training, selecting one node as the master node, and setting up a complete multi-layer super-prior image compression and decompression network for each node;

[0068] 8.2) Input the images from the training set into a multi-layer hyperprior image compression and decompression network, and set the training data batch size.

[0069] The training task is distributed to four GPUs on two nodes, which compute the gradient in parallel for each node. On worker nodes n=1,2, the current model parameters w are used. t and from training data batches Data b from the middle sampling n The calculated loss function For each sampled data b n Calculate the gradient of the loss function. Then All sampled data b n The gradient is averaged to obtain the local gradient of the working node n in the t-th iteration.

[0070]

[0071] 8.3) Each gradient is sent to the master node for accumulation, and then the gradient accumulated by the master node is sent to other nodes. Each node iteratively updates the parameters of each layer of the multilayer hyperprior image compression and decompression network along the opposite direction of the gradient to reduce the loss function. The parameter update formula is as follows:

[0072]

[0073] in For gradient, Let be the network model parameters for the t-th iteration. Here are the network model parameters for iteration t+1, where n is the number of nodes and α is the learning rate. The model parameters for node n... Based on the current gradient Adjustments are made, with α controlling the magnitude of each update. By subtracting a value proportional to the gradient from the current parameters, the model parameters are updated in the direction of reducing the loss function, thereby gradually optimizing the network performance.

[0074] 8.4) Repeat steps 8.2) and 8.3) until the loss function Loss of the multilayer hyperprior image compression and decompression network converges, and the trained multilayer hyperprior image compression and decompression network is obtained.

[0075] Step 9: Compress and decompress the remote sensing image.

[0076] The test set images are input into a trained multi-layer hyperprior image compression and decompression network. The entropy model parameters are obtained through two hyperprior analysis transformation sub-networks, and the compressed bitstream of the output image is then restored to the pixels of the original remote sensing image through a synthesis transformation sub-network, thus completing image compression and decompression.

[0077] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0078] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A method for region-of-interest image compression based on a distributed multi-layer hyperprior network, characterized in that, Includes the following steps: (1) Select the Landsat remote sensing image dataset from a public website and divide it into training and test sets in an 8:1 ratio; (2) Constructing a multi-layer hyperprior image compression and decompression network: (2a) Establish a region of interest detection module including a residual submodule, a square and bar pooling submodule, and a quantization submodule; (2b) Select the existing analytical transform subnetwork, synthetic transform subnetwork, two prior analytical subnetworks, and two prior synthetic transform subnetworks; (2c) Connect the region of interest detection module in parallel with the analysis and transformation sub-network, and then cascade it with the first super-prior analysis sub-network, the second super-prior analysis sub-network, the first super-prior synthesis transformation sub-network, the second super-prior synthesis transformation sub-network, and the synthesis transformation sub-network in sequence to form a multi-layer super-prior image compression and decompression network, and use the loss function of the super-prior image compression network as the loss function Loss of this network. (3) Training the multi-layer hyperprior image compression and decompression network: (3a) Introduce a distributed training strategy, that is, use two nodes, select one of the nodes as the master node, and set up a complete multi-layer super-prior image compression and decompression network for each node; (3b) Input the Landsat remote sensing image training set into each multi-layer hyperprior image compression and decompression network, and set the training data batch. The training task is distributed to four GPUs on two nodes to compute the gradient of each node in parallel. The gradients of each node are then sent to the master node for accumulation. Finally, the gradients accumulated by the master node are sent to the other nodes. (3c) Each node uses the gradient descent method to iteratively update the parameters of each layer of the multilayer super-prior image compression and decompression network along the opposite direction of the gradient, so as to reduce the loss function Loss. (3d) Repeat steps (3b) to (3c) until the loss function Loss of the multilayer super prior image compression and decompression network converges, and the trained multilayer super prior image compression and decompression network is obtained. (4) Input the images of the test set into the trained multi-layer super-prior image compression and decompression network for compression and decompression to obtain the decompressed restored images.

2. The method according to claim 1, characterized in that, The structural parameters of each sub-module constituting the region of interest detection module in step (2a) are as follows: The residual submodule is used to improve feature extraction capability. It includes three residual blocks connected in series. Each residual block includes two convolutional layers and one activation layer. The first convolutional layer, the activation layer, and the second convolutional layer are connected in series. The input of the first convolutional layer is added to the output of the second convolutional layer to obtain the output of the residual submodule. The kernel size of each convolutional layer is 3*3. The activation layer is implemented using the ReLU function. The square and bar pooling submodules are used to aggregate global and local contextual information and effectively extract features of the region of interest. It includes a feature extraction layer, a convolutional layer, and an activation layer. The feature extraction layer consists of a horizontal bar pooling layer, a vertical bar pooling layer, and a dilated convolutional layer connected in parallel. The kernel size of this convolutional layer is 1*1. The activation layer uses the sigmoid function. This feature extraction layer is connected in series with the convolutional layer and the activation layer, and the input of the feature extraction layer is added to the output of the activation layer to obtain the output of the square and bar pooling submodules. The quantization submodule is used to obtain a binary mask of the region of interest. It consists of a convolutional layer, an activation layer, and a rounding function layer connected in series. The convolutional kernel size of the convolutional layer is 3*3, and the activation layer is implemented using the sigmoid function.

3. The method according to claim 1, characterized in that, The structural parameters of the analytical transform subnetwork, the synthetic transform subnetwork, the two prior analytical subnetworks, and the two prior synthetic transform subnetworks selected in step (2b) are as follows: The analytical transform subnetwork is used to extract the low-dimensional latent representation of the image. It includes four boundary padding layers, four convolutional layers, and three GDN layers, with the following structural relationship: first boundary padding layer → first convolutional layer → first GDN layer → second boundary padding layer → second convolutional layer → second GDN layer → third boundary padding layer → third convolutional layer → third GDN layer → fourth boundary padding layer → fourth convolutional layer. The kernel size of the first to fourth convolutional layers is 5*5, and the convolution stride is 2. The first to third GDN layers are all implemented by generalized normalization. The synthetic transform subnetwork is used to restore the latent representation to an approximate original image. It includes three boundary padding layers, three deconvolution layers, and three IGDN layers, with the following structural relationship: first boundary padding layer → first deconvolution layer → first IGDN layer → second boundary padding layer → second deconvolution layer → second IGDN layer → third boundary padding layer → third deconvolution layer → third IGDN layer. The kernel size of the first to third deconvolution layers is 5*5, and the stride of the deconvolution layer is 2. Each of the super-prior analysis subnetworks is used to model the statistical properties of the latent representation. It includes four convolutional layers and two ReLU activation layers, with the following structural relationship: first convolutional layer → second convolutional layer → first ReLU activation layer → third convolutional layer → second ReLU activation layer → fourth convolutional layer. The kernel size of the first convolutional layer is 3*3, and the kernel size of the second to fourth convolutional layers is 1*1. Each of the super-prior synthetic transform subnetworks is used to further model the statistical properties of the latent representation during decoding. It includes four deconvolutional layers and two ReLU activation layers, with the following structural relationship: first deconvolutional layer → second deconvolutional layer → ReLU activation layer → third deconvolutional layer → ReLU activation layer → fourth deconvolutional layer. The kernel size of the first to third deconvolutional layers is 1*1, and the kernel size of the fourth deconvolutional layer is 3*3.

4. The method according to claim 1, characterized in that, The loss function Loss of the multi-layer hyperprior image compression and decompression network set in step (2c) is expressed as follows: Where R represents the overall compression bitrate, D represents the distortion between the reconstructed image and the original image, and λ is a hyperparameter used to balance bitrate and distortion. For the quantized potential representation, The modeling parameters output by the first hyperprior analysis subnetwork are... Let represent the modeling parameters output by the second hyperprior analysis subnetwork, d(·) represent the distortion metric, which is measured using mean squared error, and x represent the original image. Let p represent the reconstructed image, E represent the calculation of cross-entropy, and p represent the reconstructed image. xx represents the distribution of the latent representation of the image.

5. The method according to claim 1, characterized in that, In step (3b), the gradient of each node is computed in parallel, using the current model parameters w at the working nodes n=1,2. tt and from training data batches Data b from the middle sampling nn The calculated loss function For each sampled data b nn Calculate the gradient of the loss function. Then All sampled data b nn The gradient is averaged to obtain the local gradient g of the working node n in the t-th iteration. tt nn :

6. The method according to claim 1, characterized in that, In step (3c), each node uses the gradient descent method to iteratively update the parameters of each layer of the multilayer hyperprior image compression and decompression network along the opposite direction of the gradient, as shown in the following formula: in For gradient, Let be the network model parameters for the t-th iteration. Here are the network model parameters for iteration t+1, where n is the number of nodes and α is the learning rate.

7. The method according to claim 1, characterized in that, In step (4), the image to be compressed is input into the trained multi-layer super-prior image compression and decompression network for compression and decompression. First, the entropy model parameters are obtained through two super-prior analysis transformation sub-networks in the network, and the compressed bitstream of the image is output. The compressed bitstream is then restored to the pixels of the original remote sensing image through the synthesis transformation sub-network, thus completing the image compression and decompression.

Images