A multispectral image compression system based on multi-directional convolutional neural network

Abstract

The invention discloses a multispectral image compression system based on a multi-directional convolutional neural network, the system includes: a forward coding network, a quantization module, an entropy coding module, an entropy decoding module, an inverse quantization module, and an inverse decoding network; the method is specifically: constructing a multispectral image compression network and training it to obtain an optimal multispectral image compression network model; sending the multispectral image to be compressed into the multispectral image compression network, extracting the inter-spectral spatial features of the image by multi-directional convolution, using downsampling to reduce the size of the feature map after dimensionality reduction and fusion, removing data redundancy by quantization, and obtaining a compressed code stream for transmission and storage by lossless entropy coding; performing entropy decoding and inverse quantization on the received compressed code stream to obtain multispectral image spectral spatial feature data, inputting the inverse decoding network, and obtaining a reconstructed multispectral image. The invention can realize multi-rate compression of multispectral images and effectively improve the compression performance of multispectral images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and deep learning, and in particular to a multi-spectral image compression system based on a multi-directional convolutional neural network. Background Art

[0002] With the rapid development of multispectral imaging technology, encoding rich spectral spatial features into several narrow and continuous spectral bands can generate multispectral images that can better reflect the characteristics of the same scene than ordinary visible light images. Multispectral images have rich spectral and spatial information and have been widely used in environmental monitoring, crop status assessment, military reconnaissance, target monitoring and other fields. However, due to the large amount of information it contains, the amount of data in multispectral images has increased dramatically compared to visible light images. Considering that massive amounts of data will bring tremendous pressure to the transmission, storage and application of images, especially when the channel capacity is limited, it is necessary to effectively compress the data.

[0003] For the compression of visible light images, in most cases, only spatial correlation needs to be considered, so many effective traditional compression methods have emerged, such as JPEG2000 and 3D-SPIHT. These algorithms usually transform in the spatial dimension to eliminate spatial redundancy to obtain a higher compression ratio. Directly applying JPEG2000 or other traditional methods to multispectral image compression can also achieve good results, but given the characteristics of multispectral images, it is necessary to study a compression algorithm that focuses on the simultaneous extraction of spectral and spatial features to better retain the information contained in multispectral images.

[0004] Image data can be compressed because there are various redundant components between the data. The redundancy of image data is mainly divided into: spatial redundancy caused by the correlation between adjacent pixels, temporal redundancy caused by the correlation between adjacent frames in the image sequence, and spectral redundancy caused by the correlation between different color channels or spectral bands. In the imaging process of multispectral images, due to the small band interval, there is a strong correlation between the data of each band, that is, inter-spectral redundancy; and for a single image of each band, it is equivalent to a two-dimensional static image, that is, there is spatial redundancy. However, most of the existing multispectral image compression algorithms focus on removing spatial redundancy, while ignoring inter-spectral redundancy.

[0005] Traditional multispectral image compression algorithms are mainly divided into three categories: (1) algorithms based on predictive coding; (2) algorithms based on vector quantization coding; (3) algorithms based on transform coding. All three algorithms have obvious shortcomings: although the algorithm based on predictive coding can achieve lossless compression, the compression rate is low, and the quality of the predictor design is the main factor affecting the compression performance of the method; the algorithm based on vector quantization coding has a high computational complexity, and its compression performance is directly related to the size of the codebook, that is, the larger the codebook, the better the compression performance, but when the codebook is too large, it will greatly increase the amount of calculation; when the compression rate is high, the algorithm based on transform coding will have block effects and edge Gibbs effects, which seriously affect the compression performance. Although traditional multispectral image compression methods can also achieve good compression effects, they have failed to fully utilize the rich spectral and spatial features of multispectral images.

[0006] In recent years, deep learning technology has developed rapidly, and its application in the field of image processing has become more and more extensive. Therefore, combining deep learning technology with image compression is an inevitable trend in research. Deep learning technology combined with compression technology mainly utilizes the multi-parameter and learnable characteristics of deep learning, and uses frameworks such as convolutional neural networks or recurrent neural networks to form different network structures to extract image data features. The advantage of deep learning technology is that it can extract deep information in the image and retain the essential characteristics of the object. Applying this advantage to image compression can effectively make up for the shortcomings of traditional compression technology such as incomplete spectral and spatial feature extraction. Convolutional neural networks are the most widely used today, but ordinary convolutional networks usually ignore the inter-spectral information of multispectral images, which will cause a lot of information loss. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a multispectral image compression system based on a multi-directional convolutional neural network, which can realize multi-rate compression of multispectral images and effectively improve the compression performance of multispectral images.

[0008] In order to solve the above technical problems, the present invention provides a multispectral image compression method based on a multi-directional convolutional neural network, comprising the following steps:

[0009] (1) Construct a multispectral image compression network and train it, optimize the network parameters, and obtain the optimal multispectral image compression network model;

[0010] (2) The multispectral image to be compressed is sent to the multispectral image compression network, and the spectral features and spatial features of the image are extracted from different directions through multi-directional convolution. After dimension reduction and fusion, the size of the feature map is reduced by downsampling. The data redundancy is removed by the quantization module, and the quantized feature data is losslessly entropy encoded to obtain a compressed code stream for transmission and storage;

[0011] (3) The received compressed code stream is entropy decoded and inverse quantized to obtain the spectral and spatial feature data of the multispectral image, which is then input into the reverse decoding network. The feature map size is first restored by upsampling, and then the corresponding spectral and spatial features are restored through the spectral / spatial multi-directional convolution module in the decoding network. The reconstructed multispectral image is obtained after dimensionality reduction and fusion. Different network models are obtained by changing parameter training to achieve multi-rate compression of multispectral images.

[0012] Preferably, in step (1), a linear rectification function is used in the multispectral image compression network, and the linear rectification function expression is:

[0013] ReLU(x i )=max(0,x i )

[0014] where x i is the data of the i-th channel. The function maps the input into two segments. When the input value is less than 0, the original value is mapped to 0. If the input value is greater than 0, it is passed according to the original value. It can be seen from the derivative that there is no gradient loss during the reverse calculation.

[0015] Preferably, in step (2), the quantization function is approximated, and the formula is as follows:

[0016] X q =round[(2 q -1)×X s ]

[0017] Where X s It is the intermediate feature data obtained after being extracted by the forward encoding network and normalized by the Sigmoid function. q is the quantization level. The approximated quantization function rounds the data during forward propagation, and skips the quantization layer during back propagation, passing the gradient directly to the previous layer.

[0018] Preferably, in step (3), there are two ways to achieve multi-rate compression of multispectral images:

[0019] (a) The penalty weight λ in the rate-distortion optimization is fixed. By changing the number of neurons in the intermediate convolutional layer, the compression network with different bit rates is trained. The smaller the number of neurons, the smaller the bit rate.

[0020] (b) The number of neurons in the middle convolutional layer is fixed, and the value of the penalty weight λ in the rate-distortion optimization is changed to train compression networks with different bit rates. The larger the λ, the smaller the bit rate.

[0021] Accordingly, a multi-spectral image compression system based on a multi-directional convolutional neural network includes: a forward coding network, a quantization module, an entropy coding module, an entropy decoding module, an inverse quantization module, and a reverse decoding network; the forward coding network includes an inter-spectral / spatial multi-directional convolution extraction module, an extraction feature fusion module, and a downsampling module; the reverse coding network includes an upsampling module, an inter-spectral / spatial multi-directional convolution restoration module, and a restoration feature fusion module;

[0022] The inter-spectral / spatial multi-directional convolution extraction module extracts the inter-spectral features and spatial features of the multi-spectral image from different direction convolutions respectively; the extracted feature fusion module fuses the extracted spatial features pixel by pixel and reduces the dimension, and then connects them in parallel with the inter-spectral features after the dimension reduction to obtain a feature map; the downsampling module reduces the size of the feature map; the quantization module and the entropy coding module obtain a compressed bit stream for transmission and storage by quantizing and entropy coding the feature map; the entropy decoding module and the inverse quantization module entropy decode and inverse quantize the compressed bit stream to obtain the feature data of the multi-spectral image; the upsampling module restores the size of the feature map; the inter-spectral / spatial multi-directional convolution restoration module restores the corresponding features of the feature map; the restored feature fusion module fuses the spatial features pixel by pixel and reduces the dimension, and then connects them in parallel with the inter-spectral features after the dimension reduction to obtain a reconstructed multi-spectral image.

[0023] Preferably, the forward encoding network includes 8 inter-spectral convolution modules, 8 spatial direction 1 convolution modules, 8 spatial direction 2 convolution modules, and 3 downsampling modules; the downsampling operation uses a convolution layer with a step size of 2 and a convolution kernel size of 4×4.

[0024] Preferably, the multispectral image compression network also includes a rate-distortion optimization module; the rate-distortion optimization module introduces the entropy value of the intermediate feature data into the loss function calculation of the compression network, and continuously optimizes the distribution of spectral features and spatial feature data through training to make it more compact. The loss function is expressed as:

[0025] L=L d +λL r

[0026] Among them, L d is the distortion loss, expressed by mean square error (MSE); λ is the penalty weight, used to explicitly control the bit rate; L r It is the entropy approximation of the quantized feature map data, reflecting the distribution concentration of the intermediate feature data;

[0027] L d The calculation formula is:

[0028]

[0029] Where I represents the input image, represents the compressed network restored image, N represents the batch data size, H, W, and C represent the height, width, and number of channels of the input image respectively;

[0030] L r The calculation formula is:

[0031] L r = -Ε[log2P q ]

[0032] in,

[0033]

[0034] P d (x) is the probability density function of the intermediate data.

[0035] The beneficial effects of the present invention are as follows: (1) the network model adopts the spectral / spatial multi-directional convolution extraction module to independently extract corresponding features from different directions. The extracted feature fusion module first only fuses the features in the spatial direction pixel by pixel, and connects the spectral and spatial features in parallel, thereby ensuring that the spectral and spatial features are completely separated while also ensuring the integrity of the spatial spectral features without introducing new parameters; when training the network, the data to be trained can be directly input into the network, and the training results can be obtained at the output end, thereby reducing the intermediate independent learning steps and greatly improving the learning efficiency; (2) the spectral / spatial multi-directional convolution extraction module is used to extract the corresponding features from different directions, and the extracted feature fusion module first only fuses the features in the spatial direction pixel by pixel, and connects the spectral and spatial features in parallel, thereby ensuring that the spectral and spatial features are completely separated while also ensuring the integrity of the spatial spectral features, and no new parameters need to be introduced; when training the network, the data to be trained can be directly input into the network, and the training results can be obtained at the output end, thereby reducing the intermediate independent learning steps and greatly improving the learning efficiency; The convolution modules are cascaded separately and connected in parallel to form a forward encoding network and a reverse decoding network together with the feature fusion module to improve the training efficiency. At the same time, the inter-spectral / spatial convolution units contain short-circuit connections and refer to the residual network units, which can speed up the training and avoid a series of problems such as gradient explosion and network degradation caused by deep networks. (3) Rate-distortion optimization is added to the loss function to make the data distribution of inter-spectral features and spatial features more compact and maintain a trade-off balance between bit rate and loss distortion. While retaining more complete inter-spectral feature information of the image, the reconstructed image is closer to the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the compression system structure of the present invention.

[0037] Figure 2 It is a schematic diagram of the structure of the inter-spectral directional convolution unit of the present invention.

[0038] Figure 3 It is a schematic diagram of the structure of the spatial direction convolution unit of the present invention.

[0039] Figure 4 Schematic diagram of the forward coding network structure of the present invention.

[0040] Figure 5 This is a schematic diagram of the reverse decoding network structure of the present invention.

[0041] FIG6( a ) is a graph showing the average PSNR of a 7-band test set at different bit rates according to the present invention.

[0042] FIG6( b ) is a graph showing the average PSNR of an 8-band test set at different bit rates according to the present invention.

[0043] Figure 7 This is a display diagram of the 7-band restored image of the present invention.

[0044] Figure 8 This is a display diagram of the 8-band restored image of the present invention.

[0045] FIG9( a ) is a spectral similarity curve diagram of the 7-band test set of the present invention.

[0046] FIG9( b ) is a spectral similarity curve diagram of the 8-band test set of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 As shown, a multi-spectral image compression system based on a multi-directional convolutional neural network includes: a forward coding network, a quantization module, an entropy coding module, an entropy decoding module, an inverse quantization module, and a reverse decoding network; wherein the forward coding network includes an inter-spectral / spatial multi-directional convolution extraction module, an extraction feature fusion module, and a downsampling module; the reverse coding network includes an upsampling module, an inter-spectral / spatial multi-directional convolution restoration module, and a restoration feature fusion module; the inter-spectral / spatial multi-directional convolution extraction module extracts inter-spectral features and spatial features of a multi-spectral image from different direction convolutions respectively; the extraction feature fusion module performs pixel-by-pixel fusion on the extracted spatial features. After merging and reducing the dimension, the feature map is connected in parallel with the inter-spectral features after the dimension reduction to obtain the feature map; the downsampling module reduces the size of the feature map; the quantization module and the entropy coding module obtain a compressed code stream for transmission and storage by quantizing and entropy coding the feature map; the entropy decoding module and the inverse quantization module perform entropy decoding and inverse quantization on the compressed code stream to obtain the feature data of the multispectral image; the upsampling module restores the size of the feature map; the inter-spectral / spatial multi-directional convolution restoration module restores the corresponding features of the feature map; the restored feature fusion module fuses the spatial features pixel by pixel and reduces the dimension, and then connects them in parallel with the inter-spectral features after the dimension reduction to obtain the reconstructed multispectral image.

[0048] The present invention proposes a multispectral image compression method based on a multi-directional convolutional neural network, which specifically comprises the following steps:

[0049] Step S1: construct a multispectral image compression network, input training set data to train the network to optimize network parameters, until the optimal multispectral image compression network model is obtained.

[0050] Step S11: Multiple inter-spectral / spatial convolution units in different directions are cascaded and stacked as basic unit structures to form a corresponding inter-spectral / spatial multi-directional convolution extraction (or restoration) module, downsampling (or upsampling) is added to form a forward encoding network and a reverse decoding network respectively, and rate-distortion optimization, quantization, entropy coding, entropy decoding, and inverse quantization modules are added to form an end-to-end multispectral image compression network.

[0051] Step S12: The structures of the inter-spectral convolution unit and the spatial convolution unit are respectively as follows: Figure 2 and Figure 3 As shown in the figure, Conv represents convolution, ReLU represents linear rectification function, and the use of short-circuit connection to learn residual can speed up training. The inter-spectral convolution unit uses convolution with a kernel size of 1×1×3 to extract features, and only calculates on the inter-spectral dimension to extract complete inter-spectral feature data. The non-linear activation function ReLU is added to learn non-linear information. The spatial convolution unit uses convolution with a kernel size of 3×3×1 to extract features. Multiple inter-spectral / spatial convolution units are stacked and cascaded to form the corresponding inter-spectral / spatial multi-directional convolution extraction module. After the feature extraction is completed, they are fused and down-sampled together to form a forward encoding network.

[0052] Step S13: The forward encoding network is mainly composed of a cascade of convolutional layers, linear rectification units, multi-directional convolutional networks, feature fusion networks, and downsampling layers, and its function is to extract the complete spectral and spatial features of multispectral images.

[0053] Step S14: After the forward coding network structure, there is a quantization layer and entropy coding, which implements the function of quantizing and entropy coding the inter-spectral spatial fusion features extracted by the forward coding network. The purpose of quantization is to reduce the redundant information in the data, but it is also the main source of image compression distortion. Entropy coding is to encode the quantized data under lossless conditions to obtain a compressed binary code stream, further removing the redundancy between data statistics. The quantization layer, entropy coding and forward coding network constitute the compression model in the network.

[0054] Step S15: Inverse quantization, entropy decoding, and reverse decoding networks constitute a decompression model in the network. The inverse quantization, entropy decoding, and reverse decoding networks in the network structure correspond one to one with the quantization, entropy coding, and forward coding network functions. The entire network has a symmetrical structure and together constitutes a complete compression network structure.

[0055] Step S16: The entire compression network model adopts an end-to-end training method: at the input end, the preprocessed multispectral image is directly input into the network model to start training, and a batch processing method is adopted, that is, multiple images are read at one time to improve the network training efficiency; the image data passes through the compression model and is transmitted to the decompression model, and the data obtained at the output end is compared with the original data to calculate the distortion error, and the network parameters are optimized by minimizing the loss function, and the network parameters are updated by back propagation until the parameters reach the optimal value.

[0056] Step S2: The multispectral image to be compressed is sent to the multispectral image compression network, and the inter-spectral features and spatial features of the multispectral image are extracted respectively through the inter-spectral / spatial multi-directional convolution extraction module, and the fused spectral-spatial features are obtained through the extraction feature fusion module. Downsampling is used to reduce the size of the feature map, and data redundancy is removed through the quantization layer. The quantized intermediate feature data is losslessly entropy encoded to obtain a compressed code stream for transmission and storage. Its structure mainly includes a forward coding network, a quantization layer, and entropy coding.

[0057] Step S21: The forward encoding network is used to extract the feature data of the input multispectral image, retain the spectral information and spatial information of the multispectral image, and help reconstruct a high-quality image. Its structure is as follows: Figure 4 As shown in the figure, Conv represents the convolution layer, and the parameters in the brackets after Conv represent the number of input channels, the number of output channels, the convolution kernel size, the step size, and the padding. In addition, the downsampling operation is implemented by using a convolution with a step size of 2, and the parameters in the brackets after it represent the convolution kernel size, the step size, and the padding. ReLU represents the rectified linear unit, and the spectral and spatial convolution units are Figure 2 and Figure 3 The structure shown, the specific process of the forward encoding network is as follows:

[0058] (1) The multispectral data of size H×W×C are input into the inter-spectral and spatial multi-directional convolution modules respectively. In the inter-spectral direction convolution module, the convolution kernel size is 1×1×3, the step size is (1,1,1), and the padding is (0,0,1) (the dimension with padding 1 corresponds to the inter-spectral dimension), and the input and output channels are both 1; in the spatial direction convolution module, the convolution kernel size is 3×3×1, the step size is (1,1,1), and the padding is (1,1,0) (the 1 in padding corresponds to the spatial dimension). The number of input channels of the first convolution layer is 7 or 8, which is determined by the number of channels of the input multispectral image itself, and the number of output channels is 56. The input and output channels of the intermediate units are both 56. The input channels of the last convolution layer are 56 and the output is 7 or 8. The ReLU activation function is used to increase the nonlinear relationship between the layers of the neural network.

[0059] (2) After the features are extracted by the multi-directional convolution module, the spatial features in different directions are first added pixel by pixel and fused. The convolution with 7 or 8 input channels (determined by the number of channels of the input multispectral image itself), 4 output channels, 1×1 kernel size, 1 step size, and 0 padding is used for dimensionality reduction. The convolution with 7 or 8 input channels (determined by the number of channels of the input multispectral image itself), 3 or 4 output channels (corresponding to 7 or 8), 1×1 kernel size, 1 step size, and 0 padding is used for dimensionality reduction. The reduced inter-spectral features are connected in parallel to obtain fused spectral-spatial features, which are then passed through a layer of convolution and a layer of ReLU activation function with a convolution kernel size of 3×3, a step size of 1, and a padding of 1. Downsampling is then performed, i.e., a convolution layer with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, to obtain 64 feature maps of size H / 2×W / 2. The same process is repeated twice to obtain 64 feature maps of size H / 8×W / 8, reducing the spatial resolution of the image to 1 / 64 of the original.

[0060] (3) Finally, through the convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, the number of input and output channels is (64, 48) respectively, and 48 intermediate feature map data of size M / 8×N / 8 are finally extracted. After passing through the Sigmoid function, the network output value is between [0, 1].

[0061] Step S22: The quantization layer and entropy coding layer mainly remove data redundancy. The encoded and quantized data is a compressed code stream. Since the derivative of the quantization function is discontinuous, it cannot be directly derived and added to the network. The gradient will disappear during back propagation, which hinders the update of network parameters. Therefore, the quantization function needs to be approximated. The formula is as follows:

[0062] X q =round[(2 q -1)×X s ]

[0063] Where X s It is the intermediate feature data obtained after being extracted by the convolution layer and passed through the Sigmoid function, and q is the quantization level.

[0064] The approximated quantization function rounds the data during forward propagation and skips the quantization layer during backward propagation, passing the gradient directly to the previous layer. q ZPAQ lossless entropy coding is used to generate a binary bitstream. The bitstream is entropy decoded and restored to obtain the quantized intermediate feature data X q , and then the inverse quantized intermediate feature data X q / (2 q-1) is input into the reverse decoding network.

[0065] Step S3: perform entropy decoding and inverse quantization on the received bitstream to obtain spectral and spatial feature data of the multispectral image, which is sent to the reverse decoding network to reconstruct the multispectral image, and different compression network models are trained by changing parameters to achieve multi-rate compression of the multispectral image.

[0066] Step S31: The reverse decoding network is used to reconstruct the intermediate feature image into a multispectral image, and its structure is as follows: Figure 5 As shown in the figure, Conv represents the convolutional layer, which is symmetrical with the forward encoding network structure. The parameters in the brackets after Conv represent the number of input channels, the number of output channels, the convolution kernel size, the step size, and the padding. ReLU represents the rectified linear unit, and the spectral and spatial convolution units are Figure 2 and Figure 3 The structure shown. PixelShuffle represents an upsampling function used to restore the feature map size. The specific decoding process of the reverse decoding network is as follows:

[0067] (1) 48 feature map data of size H / 8×W / 8 are first passed through a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, resulting in 64 feature maps of size H / 8×W / 8;

[0068] (2) 16 feature maps of size H / 4×W / 4 are obtained through the Pixelshuffle layer; then, a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 and a ReLU activation function are used to obtain 64 feature maps of size H / 4×W / 4. Then, a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1 is used to obtain 64 feature maps of size M×N.

[0069] (3) Then, the data is input into the inter-spectral convolution module and the spatial convolution module respectively. In the inter-spectral convolution module, the data passes through a convolution layer with a kernel size of 1×1×3, a step size of (1,1,1), and a padding of (0,0,1) (1 in padding corresponds to the inter-spectral dimension), and the input and output channels are both 1. In the spatial convolution module, the kernel size is 3×3×1, the step size is (1,1,1), and the padding is (1,1,0) (1 in padding corresponds to the spatial dimension). The first convolution layer has 7 or 8 input channels and 56 output channels, which is determined by the number of bands of the input image. The input and output channels of the middle unit are both 56. The input channel of the last convolution layer is 56, and the output is 7 or 8. The whole process is symmetrical with the forward encoding network, which is equivalent to performing a deconvolution operation on the feature map data.

[0070] (4) Finally, the data obtained by the spatial convolution module are added and connected in parallel with the inter-spectral data after dimension reduction to reconstruct a restored image of the same size as the original image.

[0071] Step S32: In order to further optimize the performance of the multi-spectral compression network and reduce the bit rate while maintaining the restored image quality as much as possible, it is necessary to balance the bit rate and image quality loss. At the same time, in order to obtain models under different bit rates, the loss function adopted in this paper is expressed by the following formula:

[0072] L=L d +λL r

[0073] Where L d is the distortion loss, which uses the mean square error (MSE); λ is the penalty weight, which is used to explicitly control the bit rate, that is, the larger the value, the smaller the bit rate; L r It is the entropy approximation of the quantized feature map data, which can reflect the distribution concentration of the intermediate feature data;

[0074] L d The calculation formula is:

[0075]

[0076] Among them, H, W, C represent the height, width and number of channels of the image respectively, N represents the batch size, I(x, y, z) represents the pixel value at the spatial position (x, y, z) of the input image, Indicates that the compression network restores the pixel value at the image spatial position (x, y, z).

[0077] L r The calculation formula is:

[0078] L r = -Ε[log2P q ]

[0079] in,

[0080]

[0081] Here P d (x) is the probability density function of the intermediate data. Introducing the entropy of the intermediate feature data into the loss function allows the network to continuously optimize the distribution of the intermediate feature data and concentrate it during the learning process, thereby improving the compression performance of the network.

[0082] Step S33: The goal of the multispectral image compression network is to make the restored image as close to the input image as possible and retain the original image information. In the network, the optimal network parameters are obtained through learning and training to minimize the loss function value, which is expressed as:

[0083]

[0084] in,

[0085] x m (θ)=Se(θ1,x)+Sa1(θ2,x)+Sa2(θ3,x)

[0086] θ=(θ1,θ2,θ3)

[0087] Among them, x represents the input image, θ represents the parameters of the network, θ1, θ2, θ3 correspond to the parameters of the convolutional network in each direction, Se(), Sa1() and Sa2() represent the inter-spectral convolutional network, spatial direction 1 convolutional network and spatial direction 2 convolutional network respectively, Qu() represents quantization coding, and Re() represents decoding and reconstruction network. The specific process of network parameter optimization is:

[0088] At the beginning of network training, the network parameters are randomly initialized, quantized by the forward encoding network, and enter the decoding network; the decoding end reconstructs the image according to the parameters; the pixel values ​​of the reconstructed image and the pixel values ​​of the original image are sent to the loss function, the error between the two is calculated, and the parameters are updated through the back propagation algorithm until the loss function reaches the minimum, where L r The input is the unquantized output of the encoding network. Adding information entropy to the loss function can make the image data distribution more compact and effectively reduce the bit rate.

[0089] The effect of the present invention is further described below in conjunction with simulation experiments.

[0090] The hardware test platform of the present invention is: GPU (NVIDIA GeForce GTX 2080TI), memory (32GB), hard disk (Samsung SSD SM871 2.5 7mm 256GB / solid state hard disk).

[0091] The software platform is: Windows7 64-bit operating system, pytorch1.2.0, matlab.

[0092] The training set and test set of the multispectral image compression network used in the present invention are derived from the multispectral images of Landsat8 and WorldView-3 satellites, respectively containing 7 or 8 multispectral bands. In order to prevent the network from overfitting, the training set of the network selects multispectral images under different seasons, multiple weather conditions, and multiple terrain conditions, so that it contains relatively rich and diverse features, and cuts it into blocks of 128×128 size, wherein about 80,000 pieces of 7-band data (Landsat8) are used as training sets, and about 17,000 pieces of 8-band data (WorldView-3) are used as training sets; the test set of the network is selected by the same standard, the 7-band test set contains a total of 17 images, the size of which is 512×512, and the 8-band test set contains 14 images, the size of which is 512×512; the training set and the test set of the network have no repeated images, that is, the multispectral image data selected as the test set does not participate in the training.

[0093] Adam Optimizer is used for network training. The initial learning rate is set to 0.0001 for rapid network convergence and generation of pre-trained models. When the network convergence speed slows down, the learning rate is reduced to 0.00001 and the required model is obtained through training. For rate distortion optimization, according to the easy-to-hard learning idea, in the initial stage L r Set it to 0, and gradually increase L after the network fully converges. r The value of the weight λ makes the distribution of the intermediate feature map data gradually concentrated, significantly improving the compression performance.

[0094] The present invention is applied to compress multispectral images at different bit rates. The image compression ratio can be changed by changing the penalty weight λ in the trained model. The restoration effect of the image is tested using 8 different bit rates in simulation, and the performance is compared with that of the existing JPEG2000 and 3D-SPIHT methods. In order to verify the restoration of spectral information before and after image compression, spectral similarity is introduced as a reference.

[0095] FIG6(a) and FIG6(b) show the average PSNR comparison of the present invention with JPEG2000 and 3D-SPIHT on two data sets at different bit rates; Figure 7 A comparison diagram showing the restoration of four selected 7-band multispectral images using the present invention and the two algorithms of JPEG2000 and 3D-SPIHT; Figure 8 Figure 9(a) and Figure 9(b) show the comparison of the spectral similarity curves of the present invention, JPEG2000 and 3D-SPIHT on two data sets.

[0096] Peak signal-to-noise ratio (PSNR) is the most widely used objective evaluation index for traditional image compression algorithms. It is mainly measured by the change in pixels before and after compression. Its calculation formula is:

[0097]

[0098]

[0099] Where n represents the number of bits of the image, H, W, and C represent the height, width, and number of channels of the image, respectively, and I(x, y, z) represents the pixel value at the spatial position (x, y, z) of the input image. Indicates the pixel value at the spatial position (x, y, z) of the compressed network restored image. As can be seen from Figure 6, with the increase of the bit rate, the PSNR of the reconstructed image and the original image both show an upward trend. At the same bit rate, the average PSNR between the reconstructed image and the original image of the present invention is higher than that of JPEG2000 and 3D-SPIHT. Comparing the results of Figure 6 (a) and Figure 6 (b), it can be further explained that with the increase of the number of bands, the advantage of the present invention is more obvious; comprehensive comparison results at 8 different bit rates show that the PSNR of the present invention on the 7-band data set is about 3-4dB higher than that of JPEG2000 on average, and 1-2dB higher than that of 3D-SPIHT. On the 8-band data set, the PSNR of the present invention is 6-8dB higher than that of JPEG2000 on average, and 3-5dB higher than that of 3D-SPIHT; when the bit rate is 0.3 to 0.4, the PSNR advantage of the present invention is most obvious.

[0100] Figure 7 and Figure 8 The four figures respectively show the visual effect comparison of the reconstructed images of the present invention and the original images with the JPEG2000 and 3D-SPIHT algorithms on the 7-band and 8-band data sets, which more intuitively demonstrates the pros and cons of the restoration effect. It can be seen from the four comparison figures that the present invention is obviously superior to the JPEG2000 and 3D-SPIHT algorithms in the quality of the reconstructed images. Many texture details of the restored images of the latter two algorithms are blurred, especially the restored images of the JPEG2000 algorithm produce obvious block effects, while the restored images of the present invention are clearer and the texture details are more completely preserved.

[0101] Figure 9(a) and Figure 9(b) show the spectral similarity curves on the two data sets. The spectral similarity curve of the multispectral image restored by the present invention has always remained below the spectral similarity curves of JPEG2000 and 3D-SPIHT. The average spectral similarity of the multispectral image compression network at different bit rates is less than the average spectral similarity of JPEG2000 and 3D-SPIHT. The smaller the spectral similarity value, the more similar the spectral curves are and the closer they are to the original spectrum.

[0102] Table 1 Average spectral similarity of 7-band test data sets at different bit rates

[0103]

[0104] Table 2 Average spectral similarity of 8-band test data sets at different bit rates

[0105]

[0106] Table 1 shows the average spectral similarity of the 7-band test data set at different bit rates, and Table 2 shows the average spectral similarity of the 8-band test data set at different bit rates. The overall experimental data show that the algorithm of the present invention has better effect than JPEG2000 and 3D-SPIHT in terms of spectral similarity index, that is, it retains the inter-spectral information more effectively and completely.

Claims

1. A multi-spectral image compression system based on a multi-directional convolutional neural network, characterized in that: include: Forward coding network, quantization module, entropy coding module, entropy decoding module, inverse quantization module, reverse decoding network; the forward coding network includes inter-spectral / spatial multi-directional convolution extraction module, extraction feature fusion module, down-sampling module; the reverse coding network includes up-sampling module, inter-spectral / spatial multi-directional convolution restoration module and restoration feature fusion module; The inter-spectral / spatial multi-directional convolution extraction module extracts the inter-spectral features and spatial features of the multi-spectral image from different directional convolutions respectively; the extracted feature fusion module fuses the extracted spatial features pixel by pixel and reduces the dimension, and then connects them in parallel with the inter-spectral features after the dimension reduction to obtain the feature map; the downsampling module reduces the size of the feature map; the quantization module and the entropy coding module obtain the compressed code stream for transmission and storage by quantizing and entropy coding the feature map; the entropy decoding module and the inverse quantization module perform entropy decoding and inverse quantization on the compressed code stream to obtain the feature data of the multi-spectral image; The upsampling module restores the size of the feature map; the inter-spectral / spatial multi-directional convolution restoration module restores the corresponding features of the feature map; the restored feature fusion module fuses the spatial features pixel by pixel and reduces their dimensions, and then connects them in parallel with the inter-spectral features that have also been reduced in dimension to obtain a reconstructed multispectral image.

2. The multi-spectral image compression system based on a multi-directional convolutional neural network as claimed in claim 1, characterized in that: The forward encoding network contains 8 inter-spectral convolution modules, 8 spatial direction 1 convolution modules, 8 spatial direction 2 convolution modules, and 3 downsampling modules; the downsampling operation uses a convolution layer with a stride of 2 and a convolution kernel size of 4×4.

3. The multi-spectral image compression system based on a multi-directional convolutional neural network as claimed in claim 1, characterized in that: The quantization module approximates the quantization function, and the formula is as follows: X q =round[(2 q -1)×X s ] Where X s It is the intermediate feature data obtained after being extracted by the forward encoding network and normalized by the Sigmoid function. q is the quantization level. The approximated quantization function rounds the data during forward propagation, and skips the quantization layer during backward propagation, passing the gradient directly to the previous layer. It also includes a rate-distortion optimization module; the rate-distortion optimization module introduces the entropy value of the intermediate feature data into the loss function calculation of the compression network, and continuously optimizes the distribution of spectral feature and spatial feature data through training to make it more compact. The loss function is expressed as: L=L d +λL r Among them, L d is the distortion loss, expressed by mean square error (MSE); λ is the penalty weight, used to explicitly control the bit rate; L r It is the entropy approximation of the quantized feature map data, reflecting the distribution concentration of the intermediate feature data; L d The calculation formula is: Where I represents the input image, represents the compressed network restored image, N represents the batch data size, H, W, and C represent the height, width, and number of channels of the input image respectively; L r The calculation formula is: L r =-Ε[log2P q ] in, P d (x) is the probability density function of the intermediate feature data.

Images