A Transformer-based lossless compression method for aurora spectral images using spatiotemporal and temporal spectral integration

Abstract

The present invention discloses a lossless compression method for auroral spectral images based on the combined time-space spectrum of Transformer. The Transformer-based neural network can improve modeling accuracy and prediction accuracy; introduce time correlation and use reference pixels in the time, space and spectral directions for prediction at the same time, which can improve prediction accuracy and further eliminate redundancy; introduce exception processing to further reduce the bit rate.

Description

Technical Field

[0001] The present invention belongs to the field of image compression, and in particular relates to a Transformer-based time-space-spectrum combined auroral spectrum image lossless compression method. Background Art

[0002] Auroral spectral images provide detailed information about the causes and characteristics of the aurora and have important research value. However, due to the instantaneous change of the aurora, the aurora spectrometer needs to collect data continuously at very short time intervals (10 to 20 seconds), generating a large amount of data every day, which brings great challenges to storage and transmission. However, the transmission bandwidth between the polar regions and the country is very limited, so the current aurora spectral data is usually stored locally first and then brought back to the country by scientific researchers, which seriously restricts the research and application of the aurora. At present, the lossless compression method of aurora spectral images is mainly based on point prediction methods. For each pixel, its neighboring pixels are used to perform linear or nonlinear predictions. After all pixels are predicted, the original image is subtracted from the predicted image to obtain the residual image.

[0003] The residual image is then entropy-coded to generate a bitstream, and the decompression end uses the residual bitstream to restore the original image. The more accurate the prediction, the smaller the residual, the lower the entropy coding bitrate, and the better the compression performance.

[0004] Among the linear point prediction methods, Online Differential Pulse Code Modulation (Online DPCM) has the best performance. This method uses the leftmost pixel in all rows above each pixel. N Pixel construction of columns N The first-order linear equations are then solved using the least squares method. The predicted coefficients are the same as the ones on the left of the current pixel. N The weighted sum of the pixels in the image is used as the predicted value for the current pixel. This method achieves excellent compression performance by fully utilizing the local context information of each pixel. However, since a linear system of equations must be solved for each pixel, and the larger the row number, the larger the linear system corresponding to the pixel, this method is computationally intensive and takes a long time to compress an image.

[0005] Among nonlinear point prediction algorithms, JPEG-LS is used for auroral spectral image compression. This algorithm uses the three neighboring pixels to the left, above, and above the left of each pixel to predict the current pixel. The algorithm is simple, computationally efficient, and highly real-time. However, due to the small number of reference neighboring pixels, its performance needs to be further improved.

[0006] In summary, the above aurora spectral image compression algorithms all use manually designed prediction operators, and only consider the spatial and spectral correlations of aurora spectral images, without considering the temporal correlation. In addition, the manually designed prediction operators cannot accurately model the relationship between the current pixel to be predicted and the reference pixel. Summary of the Invention

[0007] The present invention provides a Transformer-based lossless compression method for auroral spectral images by combining spatiotemporal and temporal spectral information to solve the problems of low accuracy of the existing technology using manually designed prediction operators and the inability to fully eliminate redundant information without using temporal correlation.

[0008] The purpose of the present invention is to achieve the following technical means: a lossless compression method for auroral spectrum images based on the combined time-space spectrum of Transformer,

[0009] S1. Segmenting the continuous original aurora spectrum image, wherein the segmented images have temporal correlation during compression;

[0010] S2. For the pixel to be predicted, relevant pixels in time, space and spectrum directions are selected as reference pixels, and the reference pixels constitute a context vector;

[0011] S3. Build a neural network, input the context vector into the neural network, and output the predicted value of the current pixel;

[0012] S4, repeat S3 until all pixels are predicted, obtain a predicted image based on the predicted values ​​of all pixels, and subtract the predicted image from the original image to obtain a residual image;

[0013] S5, performing entropy coding on the residual image to obtain a bitstream file;

[0014] S6. Decode the code stream file to obtain a residual image, and calculate a predicted value for each pixel of the residual image through a neural network. The predicted value is added to the residual at the corresponding position in the residual image to obtain the original pixel value, thereby obtaining a lossless original aurora spectrum image.

[0015] The number of temporal reference pixels required for reference is also set for the image to be predicted. If enough temporal reference pixels are available, the image enters S2. If not, Online DPCM is used to compress the entire image.

[0016] Before image prediction, the number of spatial and spectral reference pixels required for the image to be predicted is set.

[0017] If the pixel to be predicted cannot obtain a sufficient number of spatial and spectral reference pixels, it is compressed in the following way:

[0018] If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself;

[0019] If the pixel to be predicted is the pixel in the first row or column of the image, the predicted value is the immediately preceding pixel;

[0020] The remaining pixels are predicted using the JPEG-LS method.

[0021] After S4, the residual image is processed for abnormal values, specifically, a threshold is set for the number of residual occurrences. T , when the frequency of a residual value is less than the threshold T When , the pixel in the original aurora spectrum image with the same position as the residual is called an outlier. The residual is directly set to zero, and the modified residual is entropy coded. Finally, the outlier pixels together with their positions and the residual code stream are transmitted to the decoding end to restore the original image.

[0022] The JPEG-LS method predicts specifically that,

[0023] Each pixel is predicted using the three pixels to its left, above, and above the left according to the following formula; where, is the predicted value of the current pixel, is the pixel to the left of the current pixel, is the pixel above, is the upper left pixel;

[0024] .

[0025] The specific method of obtaining the predicted value through the neural network is:

[0026] Use word embedding technology to p Embed the context vector and map each element in the context vector into a 512-dimensional vector;

[0027] Use position encoding technology to position encode the context vector and generate a 512-dimensional vector for each element according to its position in the context vector;

[0028] Add the word embedding vector to the corresponding position element of the position vector to obtain the final input vector to be input to the neural network;

[0029] The input vector is fed into an encoder consisting of two Transformer encoding units to obtain the hidden layer representation. y；

[0030] Will y Input the decoder consisting of three layers of fully connected layers, and get p The predicted value of ,The sizes of the three fully connected layers are 512, 256, and 1 respectively. ReLU activation function and Dropout layer are added after the first and second layers.

[0031] The context vector is specifically:

[0032] For the image where the pixel to be predicted is located, the number of pixels on the left of the pixel to be predicted and the number of pixels on the top of the pixel in the spectral direction are as follows:

[0033] The image before the pixel to be predicted in the time direction, with the pixel to be predicted as the center, the number of pixels on the left and right of the spatial pixels, and the number of pixels above and below the spectral direction;

[0034] All the above pixels are arranged in a fixed order to form a context vector.

[0035] When selecting pixels in the spatial and spectral directions of the image preceding the pixel to be predicted in the time direction, the number of pixels is smaller than the number of spatial and spectral reference pixels required for the reference previously set.

[0036] The beneficial effects of the present invention are as follows: the Transformer-based neural network can improve modeling accuracy and prediction accuracy; the introduction of time correlation and the use of reference pixels in time, space and spectral directions for prediction can improve prediction accuracy and further eliminate redundancy; the introduction of exception handling can further reduce the bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of the Transformer-based lossless compression method for auroral spectral images using a combined spatiotemporal and temporal approach.

[0038] Figure 2 is a neural network structure diagram;

[0039] Figure 3 Schematic diagram of context vector;

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0041] [Example 1]

[0042] A lossless compression method for aurora spectral images based on the combined temporal, spatial and spectral properties of Transformer.

[0043] This method is a point prediction method that calculates the predicted value of each pixel from left to right and from top to bottom. After all pixel predictions are calculated, the predicted image is subtracted from the original image to obtain a residual image. The residual image is then entropy-encoded to obtain a compressed bitstream, completing the compression. During decompression, the compressed bitstream is first decoded to obtain the residual image. The predicted image is then calculated using the same method as for compression. Finally, the residual image is added to the predicted image to losslessly restore the original image.

[0044] S1. Segmenting the continuous original aurora spectrum image, wherein the segmented images have temporal correlation during compression;

[0045] For example, if the original image segment length is set to 10, these 10 images will have temporal correlation when compressed.

[0046] Compared to single-frame auroral spectral image compression, sequential auroral spectral image compression primarily introduces temporal correlation between adjacent images. This temporal correlation means that to decompress a single image, the previous images upon which it depends must be decompressed first, and these images, in turn, depend on even earlier images. Ultimately, to decompress a single image, all previous images must be decompressed first, resulting in a significant amount of redundant computation and wasted resources.

[0047] In order to alleviate this problem, the continuous aurora spectrum images need to be segmented during compression. The specific length can be set according to user needs. It is worth noting that the network structure is independent of the length of the time segment, so the network only needs to be trained once and does not need to be adjusted for different lengths. Since the compression of each image in the present invention refers to the previous two images, the first two images of each segment after segmentation cannot obtain enough time reference pixels. For these two images, the present invention does not design a special compression algorithm, but adopts the compression algorithm OnlineDPCM without time correlation for compression.

[0048] The number of temporal reference pixels required for reference is also set for the image to be predicted. If enough temporal reference pixels are available, the image enters S2. If not, Online DPCM is used to compress the entire image.

[0049] First, set the number of pixels to be referenced in the time direction. For example, if it is set to 2, it means that when predicting the image, that is, compressing it, two images from the previous time are needed as references.

[0050] Among the 10 images, the S3 method is used for prediction starting from the third image, and the first two images are compressed using OnlineDPCM.

[0051] Before image prediction, the number of spatial and spectral reference pixels required for the image to be predicted is set.

[0052] If the pixel to be predicted cannot obtain a sufficient number of spatial and spectral reference pixels, it is compressed in the following way:

[0053] If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself;

[0054] If the pixel to be predicted is the pixel in the first row or column of the image, the predicted value is the immediately preceding pixel;

[0055] The remaining pixels are predicted using the JPEG-LS method.

[0056] Since in an image, the pixels in the boundary area cannot obtain enough spatial and spectral reference pixels,

[0057] Therefore, for this part of the boundary area, the following method is used for prediction:

[0058] If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself; this is also the first pixel when the image is decompressed, and this step is the key to image decompression;

[0059] The predicted value of the pixel in the first row and first column is the previous adjacent pixel. Therefore, when decompressing, the pixel in the first row and first column can be decompressed based on the known pixel in the upper left corner of the image.

[0060] The previous immediately adjacent pixel specifically refers to, for example, the pixel at the second position from the left to the right in a row, and its predicted value is the previous one, that is, the pixel at the first position on the left.

[0061] The predicted value of the pixel at the third position is the pixel at the second position on the left.

[0062] The same is true for columns, except that the previous immediately adjacent pixel is the pixel above. So when decompressing, based on the known pixel in the upper left corner of the image, the pixels of the first row and the first column can be decompressed;

[0063] For pixels other than the first pixel in the upper left corner, the first row, and the first column where there are not enough spatial and spectral reference pixels, the JPEG-LS method is used for prediction.

[0064] The JPEG-LS method predicts specifically that,

[0065] Each pixel is predicted using the three pixels to its left, above, and above the left according to the following formula; where, is the predicted value of the current pixel, is the pixel to the left of the current pixel, is the pixel above, is the upper left pixel;

[0066] .

[0067] S2. For the pixel to be predicted, relevant pixels in time, space and spectrum directions are selected as reference pixels, and the reference pixels constitute a context vector;

[0068] For pixels that satisfy the correlation in time, space and spectral directions at the same time, they are used as reference pixels of reference pixels and enter s2 to prepare for prediction.

[0069] Corresponding pixels in time, space and spectral directions are selected as reference pixels to form a context vector for predicting the current pixel.

[0070] The number of spatiotemporal reference pixels selected in the present invention is 20 in total, and the specific selection method is as follows: Figure 3 As shown in Figure 2. Context refers to pixels that have a strong correlation with the pixel to be predicted. Using them for prediction can effectively remove redundant information from the image. A single aurora spectral image contains spatial and spectral correlations, and there is temporal correlation between adjacent images. Therefore, the present invention utilizes both spatiotemporal and temporal spectral correlations for prediction.

[0071] The context vector is specifically:

[0072] For the image where the pixel to be predicted is located, the number of pixels on the left of the pixel to be predicted and the number of pixels on the top of the pixel in the spectral direction are as follows:

[0073] The image before the pixel to be predicted in the time direction, with the pixel to be predicted as the center, the number of pixels on the left and right of the spatial pixels, and the number of pixels above and below the spectral direction;

[0074] All the above pixels are arranged in a fixed order to form a context vector.

[0075] by Figure 3 For example, the number of pixels in the temporal direction is 2, and the number of pixels in the spatial and spectral directions is 3.

[0076] Figure 3 In the figure c, the black ones are the pixels to be predicted, and the rest of the gray ones are part of the reference pixels in the context vector.

[0077] The image in the time direction means that when it is used as a reference pixel, the two images before the current image are used as references.

[0078] In this image, Figure 3In Figure c, the spatial direction is left and right, and the spectral direction is up and down. Since Figure c starts to predict from top to bottom and from left to right, the pixels on the right and bottom of the pixel to be predicted are not known yet, so only the pixels on the left 3 and top 3 grids are selected as references.

[0079] In the first two pictures, since they have been processed and all the pixels are known, each picture selects three left and right grids, three top and bottom grids, and the pixel at the position of the pixel to be predicted, a total of 13 pixels as reference pixels, 13 pixels in picture a, 13 pixels in picture b, and 6 pixels in picture c.

[0080] Arranging all pixels in a fixed order forms a context vector. The specific order does not matter and is not required. As long as a fixed order is used, it will be fine.

[0081] When selecting pixels in the spatial and spectral directions of the image preceding the pixel to be predicted in the time direction, the number of pixels is smaller than the number of spatial and spectral reference pixels required for the reference previously set.

[0082] At the same time, since the previous images in the time direction are not accurate, when collecting reference pixels in the previous images, the number of pixels in the spatial and spectral directions can be reduced to 3.

[0083] like Figure 3 In b and a, the number of pixels required in b is 3 minus one, which means taking 2 pixels on each side and the pixel at the position of the pixel to be predicted, a total of 9 pixels as reference pixels.

[0084] Image a is more distant in time and less accurate, so the number of pixels is further reduced, and only the surrounding grids and itself are selected, for a total of 5 grids as reference pixels.

[0085] S3. Build a neural network, input the context vector into the neural network, and output the predicted value of the current pixel;

[0086] Commonly used neural networks include: 1. Fully connected neural network, which inputs the constructed context vector into the fully connected neural network and outputs the predicted value of the current pixel; 2. LSTM neural network. Since LSTM is specifically designed for time series data, the reference pixels in the spatial and spectral directions can also be regarded as time series data. Then, the LSTM network is used to perform predictions in the time, space, and spectral directions respectively to obtain three predicted values ​​(or vectors). Finally, these three values ​​(or vectors) are spliced ​​together and input into the fully connected network to output the predicted value of the current pixel.

[0087] Furthermore, using the modified neural network, the specific architecture of the neural network is as follows Figure 2 As shown in Figure 2, the specific method for obtaining the predicted value through the neural network is:

[0088] Use word embedding technology to p The context vector is embedded and each element in the context vector is mapped into a 512-dimensional vector. Since the length of the context vector in the present invention is 20, after word embedding, the context vector is mapped into a 20×512-dimensional vector.

[0089] The context vector is positionally encoded using positional encoding technology, generating a 512-dimensional vector for each element according to its position in the context vector; therefore, the dimension of the position vector of the entire context vector is 20×512.

[0090] Add the word embedding vector to the corresponding position element of the position vector to obtain the final input vector to be input to the neural network;

[0091] The input vector is fed into an encoder consisting of two Transformer encoding units to obtain the hidden layer representation y；

[0092] Will y Input the decoder consisting of three layers of fully connected layers, and get p The predicted value of The three fully connected layers have sizes of 512, 256, and 1, respectively. ReLU activation functions and Dropout layers are added after the first and second layers. Dropout layers prevent overfitting by dropping some neurons during training. The batch size for training is 32, the loss function is Mean Squared Error (MSE), the optimizer is Adam, and the initial learning rate is 0.01.

[0093] FC represents the fully connected layer, p represents the current pixel, Represents the predicted value of the current pixel, and y is the hidden layer representation.

[0094] S4, repeat S3 until all pixels are predicted, obtain a predicted image based on the predicted values ​​of all pixels, and subtract the predicted image from the original image to obtain a residual image;

[0095] Repeat the steps in S3 to predict all pixels, obtain the predicted values ​​of all pixels, and finally obtain a predicted image. Subtract the predicted image from the original aurora spectrum image to obtain a residual image.

[0096] After S4, the residual image is processed for abnormal values, specifically, a threshold is set for the number of residual occurrences. T , when the frequency of a residual value is less than the threshold TWhen , the pixel in the original aurora spectrum image with the same position as the residual is called an outlier. The residual is directly set to zero, and the modified residual is entropy coded. Finally, the outlier pixels together with their positions and the residual code stream are transmitted to the decoding end to restore the original image.

[0097] The residual image has some values ​​with large absolute values ​​but very few occurrences. The bit rate increase brought by encoding these values ​​together with other residuals is generally greater than the bit rate increase brought by directly transmitting the original pixel values ​​corresponding to these residuals. To solve this problem, the present invention proposes to set a threshold for the number of occurrences of the residual. T , when the frequency of a residual value is less than the threshold T When the error is detected, the pixel in the original aurora spectrum image that is located at the same position as the residual is called an outlier. This residual is then set to zero, entropy encoded, and the outlier pixels, along with their positions and the residual bitstream, are transmitted to the decompression end to restore the original image. Because the outlier pixel values ​​themselves are transmitted to the decompression end, no further calculations are required when decompressing to that position. Therefore, this outlier handling method can reduce both the bit rate and the computational effort during decompression.

[0098] Since an auroral spectrum image has 1024×1024 pixels, 3 bytes are required to store the location of an outlier. Since the maximum pixel value is 16383, 2 bytes are required to store an outlier. Therefore, a total of 5 bytes are required to store an outlier and its location.

[0099] S5, entropy coding the residual image (ie Figure 2 Entropy Coding in

[15] is used to obtain a bitstream file; adaptive arithmetic coding is used for entropy coding to obtain a bitstream file. The entire process from constructing the context vector to completing the residual entropy coding is the image compression process.

[0100] S6. Decode the code stream file to obtain a residual image, and calculate a predicted value for each pixel of the residual image through a neural network. The predicted value is added to the residual at the corresponding position in the residual image to obtain the original pixel value, thereby obtaining a lossless original aurora spectrum image.

[0101] This step is to decompress the code stream file to obtain the original image. Specifically,

[0102] The residual image is predicted according to the same steps as compression, that is, compressed, to obtain the predicted value of each pixel in the residual image. The predicted value is added to the residual value of the corresponding position in the residual image, that is, the pixel of the original image, to obtain a lossless original aurora spectrum image.

[0103] Specifically, the first pixel in the upper left corner of the original image is the first value in the upper left corner of the residual image;

[0104] The pixel at the second position of the first row of the original image is the pixel at the first position plus the residual at the second position, the pixel at the third position is the pixel at the second position plus the residual at the third position, and so on, to get the pixels in the first row. The pixel at the second position of the first column of the original image is the pixel at the first position plus the residual at the second position, the pixel at the third position is the pixel at the second position plus the residual at the third position, and so on, to get the pixels in the first column.

[0105] For the remaining pixels that cannot obtain enough spatial spectrum reference pixels, the JPEG-LS algorithm is used to calculate the predicted value, and then the residual at the corresponding position is added to obtain the original pixel;

[0106] For pixels that can obtain enough spatial spectral reference pixels, a context vector is constructed and input into the network trained during compression to obtain a predicted value. The predicted value is then added to the residual at the corresponding position to obtain the original pixel.

Claims

1. A Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral approach, characterized by: S1. Segmenting the continuous original aurora spectrum image, wherein the segmented images have temporal correlation during compression; S2. For the pixel to be predicted, relevant pixels in time, space and spectrum directions are selected as reference pixels, and the reference pixels constitute a context vector; S3. Build a neural network, input the context vector into the neural network, and output the predicted value of the current pixel; The specific method of obtaining the predicted value through the neural network is: Use word embedding technology to p Embed the context vector and map each element in the context vector into a 512-dimensional vector; Use position encoding technology to position encode the context vector and generate a 512-dimensional vector for each element according to its position in the context vector; Add the word embedding vector to the corresponding position element of the position vector to obtain the final input vector to be input to the neural network; The input vector is fed into an encoder consisting of two Transformer encoding units to obtain the hidden layer representation. y； Will y Input the decoder consisting of three layers of fully connected layers, and get p The predicted value of , the sizes of the three fully connected layers are 512, 256, and 1 respectively. ReLU activation function and Dropout layer are added after the first and second layers; S4, repeat S3 until all pixels are predicted, obtain a predicted image based on the predicted values ​​of all pixels, and subtract the predicted image from the original image to obtain a residual image; S5, performing entropy coding on the residual image to obtain a bitstream file; S6. Decode the code stream file to obtain a residual image, and calculate a predicted value for each pixel of the residual image through a neural network. The predicted value is added to the residual at the corresponding position in the residual image to obtain the original pixel value, thereby obtaining a lossless original aurora spectrum image.

2. The Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral approach according to claim 1, characterized in that: The number of temporal reference pixels required for reference is also set for the image to be predicted. If enough temporal reference pixels are available, the image enters S2. If not, Online DPCM is used to compress the entire image.

3. The Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral analysis according to claim 1 or 2, characterized in that: Before image prediction, the number of spatial and spectral reference pixels required for the image to be predicted is set. If the pixel to be predicted cannot obtain a sufficient number of spatial and spectral reference pixels, it is compressed in the following way: If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself; If the pixel to be predicted is the pixel in the first row or column of the image, the predicted value is the immediately preceding pixel; The remaining pixels are predicted using the JPEG-LS method.

4. The Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral analysis according to claim 1, characterized in that: After S4, the residual image is processed for abnormal values, specifically, a threshold is set for the number of residual occurrences. T , when the frequency of a residual value is less than the threshold T When , the pixel in the original aurora spectrum image with the same position as the residual is called an outlier. The residual is directly set to zero, and the modified residual is entropy coded. Finally, the outlier pixels together with their positions and the residual code stream are transmitted to the decoding end to restore the original image.

5. The Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral approach according to claim 3, characterized in that: The JPEG-LS method predicts specifically that, Each pixel is predicted using the three pixels to its left, above, and above the left according to the following formula; where, is the predicted value of the current pixel, is the pixel to the left of the current pixel, is the pixel above, is the upper left pixel; 。 6. The Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral analysis according to claim 3, characterized in that: The context vector is specifically: For the image where the pixel to be predicted is located, the number of pixels on the left of the pixel to be predicted and the number of pixels on the top of the pixel in the spectral direction are as follows: The image before the pixel to be predicted in the time direction, with the pixel to be predicted as the center, the number of pixels on the left and right of the spatial pixels, and the number of pixels above and below the spectral direction; All the above pixels are arranged in a fixed order to form a context vector.

7. The Transformer-based lossless compression method for auroral spectral images using a combined temporal, spatial, and spectral analysis according to claim 6, characterized in that: When selecting pixels in the spatial and spectral directions of the image preceding the pixel to be predicted in the time direction, the number of pixels is smaller than the number of spatial and spectral reference pixels required for the reference previously set.

Images