Image compression method based on deep self-attention transformer network

Abstract

The present invention discloses an image compression method based on a deep self-attention transformation network. The specific steps of the method are as follows: S1, collect and organize standardized high-definition image data sets, construct neural network training sets, test sets and cross-validation sets; S2, establish a multi-layer neural network based on deep self-attention transformation, the neural network includes a coding network and a decoding network, and a deep self-attention transformation module is added to both the coding network and the decoding network for feature information aggregation; train the coding network and the decoding network; S3, input the image into the coding network to obtain the output data, and obtain a compressed code stream after quantization and entropy coding; S4, reconstruct and restore the compressed image through a decoding network symmetrical to the coding network. The image compression method of the present invention can save an average of about 20% of the code rate at the same quality in an equal comparison with the traditional image compression method BPG on a large number of test images.

Description

Technical Field

[0001] The present invention relates to the field of image coding, and in particular to an image compression method based on a deep self-attention transformation network. Background Art

[0002] In recent years, artificial neural networks have developed to the stage of deep learning. Deep learning attempts to use a series of algorithms that contain complex structures or multiple processing layers composed of multiple nonlinear transformations to perform high-level abstraction of data. Its powerful expressive ability enables it to achieve the best results in various machine learning tasks, and its performance in image processing currently exceeds other methods.

[0003] Deep learning uses the idea of ​​hierarchical abstraction, where high-level concepts are learned through low-level concepts. This hierarchical structure is usually constructed using a greedy layer-by-layer training algorithm, from which effective features that are helpful for machine learning are selected. Since many deep learning algorithms appear in the form of unsupervised learning, these algorithms can be applied to unlabeled data that other algorithms cannot reach. This type of data is richer and easier to obtain than labeled data, which is an important advantage of deep learning.

[0004] Image compression is a very important processing method in local storage during network transmission. For those original image data, no wired network can meet the requirements of real-time transmission. Therefore, the standardization of image coding began in the 1980s. Today, the more mature JPEG, JPEG2000 and BPG technologies can reduce the bit rate and improve the transmission efficiency under the same quality.

[0005] Due to the excellent performance of deep learning in image processing, combined with the basic idea of ​​Autoencoder in deep learning for image compression, deep learning methods are used to provide a new image encoding and decoding method. This is a good start for the development of the field of image data compression, and it is also convenient for the future in the direction of image compression. The new neural network solution has better performance and prospects than traditional methods in the process of improving the entire system.

[0006] The current mainstream deep learning-based image compression methods are all based entirely on convolutional neural networks. Due to the inherent characteristics of convolutional neural networks, such as fixed weight sharing and the need for stacked convolutional layers to expand the receptive field, they cannot effectively adapt to the content when building the codec network, and the sliding window mechanism has high computational overhead, which is not friendly to hardware design.

[0007] The deep self-attention transformer network was first used in natural language processing tasks, and has recently achieved good results in visual tasks such as object detection and recognition. With the help of content-adaptive relevance measurement and non-overlapping window design, the deep self-attention transformer network has achieved many performance results that are better than convolutional neural networks. Summary of the invention

[0008] In view of the above prior art, the purpose of the present invention is to combine the neural network method in deep learning and propose an image compression method based on a deep self-attention transformation network.

[0009] The technical solution adopted by the present invention is:

[0010] The image compression method based on the deep self-attention transformation network includes the following steps:

[0011] S1, collect and organize standardized high-definition image datasets, and construct neural network training sets, test sets, and cross-validation sets;

[0012] S2, establish a multi-layer neural network based on deep self-attention transformation, the neural network includes an encoding network and a decoding network, and add a deep self-attention transformation module to both the encoding network and the decoding network for feature information aggregation; train the encoding network and the decoding network;

[0013] S3, input the image into the coding network to obtain the output data, which is then quantized and entropy encoded to obtain a compressed bit stream;

[0014] S4, the compressed image is restored through the decoding network which is symmetrical with the encoding network.

[0015] Furthermore, in step S2, the encoding network first uses a convolutional neural network as a feature extraction layer and performs feature sampling, and then adds a self-attention transformation network module to perform feature downsampling, but the self-attention transformation network module is not added during the last feature downsampling.

[0016] Furthermore, in step S2, the self-attention transformation network module divides the feature map into N×N blocks, calculates the pixel correlation within the block, and aggregates information on features with stronger correlation.

[0017] Furthermore, after information aggregation, the window moving mechanism and convolutional neural network are introduced.

[0018] The present invention proposes a new image compression method using a self-attention transformation network. In a comparison with the existing BPG image coding method on a large number of test images, the present invention can save an average of about 20% of the bit rate at the same image output quality while ensuring the same computational complexity. In addition, since the neural network model has the property of being updateable and iterative, the present invention has better scalability than the traditional BPG image coding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the image compression method based on the deep self-attention transformation network of the present invention.

[0020] Figure 2 It is a system diagram of an image compression method implemented in an embodiment of the present invention.

[0021] Figure 3 2 is a structural diagram of (a) a self-attention transformation module and (b) a self-attention layer in an embodiment of the present invention.

[0022] Figure 4 It is a performance result diagram of each bit rate point of the method of the present invention and the traditional image coding BPG method. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, an image compression method based on a deep self-attention transformation network in this embodiment comprises the following steps:

[0025] (1) First, collect and organize the required high-definition images (such as the Flicker2W image library), organize standardized image datasets, and construct neural network training datasets, test datasets, and cross-validation datasets.

[0026] The organization of the data set has a great impact on the training of the entire neural network. Factors such as data volume and data characteristics affect the degree of fit of the final neural network model to the data. During the training process, a real-time Loss and time T curve is generated, and the model is saved after the curve converges to a flat and stable state.

[0027] (2) Establish a multi-layer neural network based on deep self-attention transformation and train the image encoding network and decoding network.

[0028] First, a multi-layer deep neural network is designed. The codec end adopts a symmetrical structure. The size and number of filters in each layer of the neural network are fully adjusted to maximize performance. In the specific implementation details, taking full connection as an example, the input data is normalized, and the activation function of the middle layer is selected for the characteristics of the quantized output in the present invention. The activation function adapted to quantization (such as sigmoid, tanh function) is selected, and the activation function of the remaining layers is still maintained to improve the convergence speed and prevent overfitting (such as Relu function). The gradient descent optimization algorithm uses an adaptive optimizer (such as Adadelta). After sufficient verification, the above settings can maximize network performance. After sufficient training on the data set established in step (1), after the cost function Loss value reaches convergence, the training model is saved as an intra-frame prediction model and an inter-frame prediction model respectively. The neural network sets different feature map numbers for different bit rate point networks (taking eight bit rate points as an example, the number of feature channels of the first four low bit rate points is 128, and the number of feature channels of the last four high bit rate points is 192), and a neural network model is generated. The neural network has a symmetrical structure. The first N layers serve as the encoding end of the neural network, and the last N layers (including the middle layer) serve as the decoding end of the neural network.

[0029] Based on the encoding network structure, the convolutional neural network is first used as the feature extraction layer, and feature sampling is performed. Then the self-attention transformation network module is added. The encoding network performs feature downsampling four times in total, and the self-attention transformation network module is not added to the last feature downsampling. The self-attention transformation network module divides the feature map into N×N blocks, calculates the self-attention within the block, avoids the calculation of the whole map, saves the amount of calculation, and limits the correlation measurement to a smaller range, which is conducive to information aggregation of features with strong correlation rows. Then the window moving mechanism and convolutional neural network are introduced to ensure that the correlation measurement is not completely limited by the window size.

[0030] (3) The image is input into the coding network to obtain the output data, which is quantized and entropy encoded to obtain a compressed code stream.

[0031] The output of the intermediate layer of the neural network is extracted, and the feature map of the final output is first quantized and lossless entropy coded to form a fixed bit stream, which is saved as the final compressed data of the neural network. The quantization scheme uses scalar quantization to reduce the bit rate of the prediction signal and the residual signal. After scalar quantization, lossless entropy coding (taking arithmetic coding as an example) is used to further compress the quantized data according to the distribution of the intermediate data to obtain the compressed data of the final intermediate layer output.

[0032] (4) The compressed image is reconstructed and restored through a decoding network that is symmetrical to the encoding network.

[0033] The decoding end reconstructs the compressed image by solving the arithmetic code and the last four layers of the neural network in a symmetrical manner to the encoding end.

[0034] The overall system framework of the above steps is shown in Figure 2 Among them, the self-attention transformation module is shown in Figure 3 In (a), the three-dimensional space features are first mapped to two dimensions through feature mapping, followed by the self-attention layer to calculate and aggregate similar features, and finally the two-dimensional features are de-mapped back to three-dimensional space. The skip connection is used to connect the input and the output of the feature de-mapping layer. Figure 3 In (b), a normalization layer, a window self-attention layer, a normalization layer, and a multilayer perceptron are stacked twice in sequence. The second window self-attention layer introduces a moving window self-attention layer to combine the information of each window.

[0035] Compared with the traditional image coding BPG method, the method of the present invention achieves an average bit rate saving of about 20%. The performance results of each bit rate point are shown in Figure 4 In the figure, “Proposed” corresponds to the method of the present invention, and “BPG444” corresponds to the traditional image coding algorithm BPG.

[0036] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image compression method based on a deep self-attention transformer network, characterized in that: The steps include: S1, collect and organize high-definition image datasets, and construct neural network training sets, test sets, and cross-validation sets; S2, establish a multi-layer neural network based on deep self-attention transformation, the neural network includes an encoding network and a decoding network, the neural network is a symmetrical structure, the first N layers are used as the encoding end of the neural network, and the last N layers include an intermediate layer as the decoding end of the neural network, and the neural network sets different feature channel numbers for different bit rate point networks; add a self-attention transformation module to both the encoding network and the decoding network for feature information aggregation; train the encoding network and the decoding network; the encoding network first uses a convolutional neural network as a feature extraction layer and performs feature sampling, and then adds a self-attention transformation module for feature downsampling, but does not add the self-attention transformation module during the last feature downsampling; the self-attention transformation module divides the feature map into N×N blocks, calculates the pixel correlation within the block, aggregates information for features with strong correlation, and then introduces a window moving mechanism and a convolutional neural network; S3, input the image into the encoding network to obtain output data, and then quantize and entropy encode it to obtain a compressed bit stream; S4, the compressed image is restored through the decoding network which is symmetrical with the encoding network.

Images