Transform-based spatial-temporal spectrum combined aurora spectral image lossless compression method

Abstract

The invention discloses an aurora spectral image lossless compression method based on time-space spectrum combination of Transform. A neural network based on the Transform can improve the modeling precision and the prediction accuracy; time correlation is introduced, and meanwhile, reference pixels in time, space and spectrum directions are used for prediction, so that the prediction accuracy can be improved, and redundancy can be further eliminated; and by introducing exception processing, the code rate can be further reduced.

Description

Technical Field

[0001] The present invention belongs to the field of image compression, and in particular relates to a Transformer-based space-time spectrum joint aurora spectrum image lossless compression method. Background Art

[0002] Aurora spectral images provide detailed information about the causes and characteristics of the aurora and have important research value. However, due to the instantaneous changes in the aurora, the aurora spectrometer needs to collect data continuously at very short time intervals (10 to 20 seconds), generating a large amount of data every day, which poses great challenges to storage and transmission. However, the transmission bandwidth between the poles and the country is very limited, so the current aurora spectral data is usually stored locally first, and then brought back to the country by researchers, which seriously restricts the research and application of the aurora. At present, the lossless compression method of aurora spectral images is mainly based on point prediction methods. For each pixel, its neighboring pixels are used to make linear or nonlinear predictions. After all pixels are predicted, the original image is subtracted from the predicted image to obtain the residual image.

[0003] Then the residual image is entropy encoded to generate a bitstream, and finally the decompression end uses the residual bitstream to restore the original image. The more accurate the prediction, the smaller the residual, the smaller the entropy encoding bitrate, and the better the compression performance.

[0004] Among the linear point prediction methods, Online Differential Pulse Code Modulation (Online DPCM) has superior performance. This method uses the pixels in the left N columns of all rows above each pixel to construct an N-dimensional linear equation group, and then uses the least squares method to solve it. The weighted sum of the prediction coefficient and the N pixels to the left of the current pixel is used as the prediction value of the current pixel. This method has excellent compression performance because it makes full use of the local context information of each pixel. However, since a linear equation group needs to be solved for each pixel, and the larger the row number, the larger the linear equation group corresponding to the pixel, the more computational effort this method has, and it takes a long time to compress an image.

[0005] In the nonlinear point prediction algorithm, JPEG-LS is used for aurora spectrum image compression. The algorithm uses the three neighboring pixels to the left, above, and above the left of each pixel to predict the current pixel. The algorithm is simple, with small calculation amount and high real-time performance. However, due to too few reference neighboring pixels, the algorithm performance needs to be further improved.

[0006] In summary, the above aurora spectral image compression algorithms all use artificially designed prediction operators, and only consider the spatial and spectral correlations of aurora spectral images, without considering the temporal correlation, and the artificially designed prediction operators cannot accurately model the relationship between the current pixel to be predicted and the reference pixel. Summary of the invention

[0007] The present invention provides a Transformer-based time-space spectrum joint auroral spectral image lossless compression method to solve the problems of low accuracy of artificially designed prediction operators in the prior art, and failure to use time correlation and fully eliminate redundant information.

[0008] The purpose of the present invention is to achieve a lossless compression method of aurora spectrum images based on the combined time-space spectrum of Transformer by the following technical means:

[0009] S1, segmenting the continuous original aurora spectrum image, and the divided images have time correlation when compressed;

[0010] S2. For the pixel to be predicted, relevant pixels in time, space and spectrum directions are selected as reference pixels, and the reference pixels constitute a context vector;

[0011] S3, image compression, building a neural network, inputting the context vector into the neural network, and outputting the predicted value of the current pixel;

[0012] S4, repeat S3 until all pixels are predicted, obtain a predicted image according to the predicted values ​​of all pixels, and subtract the predicted image from the original image to obtain a residual image;

[0013] S5, performing entropy coding on the residual image to obtain a bitstream file;

[0014] S6. Decode the bitstream file to obtain a residual image, and calculate a prediction value for each pixel of the residual image through a neural network. The original pixel value can be obtained by adding the prediction value to the residual at the corresponding position in the residual image, thereby obtaining a lossless original aurora spectrum image.

[0015] The number of temporal reference pixels required for reference is also set for the image to be predicted. If enough temporal reference pixels are available, the image enters S2. If not, Online DPCM is used to compress the entire image.

[0016] Before image prediction, the number of spatial and spectral reference pixels to be referenced is set for the image to be predicted.

[0017] If the pixel to be predicted cannot obtain a sufficient number of spatial and spectral reference pixels, it is compressed in the following way:

[0018] If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself;

[0019] If the pixel to be predicted is the pixel in the first row or first column of the image, the predicted value is the immediately preceding pixel;

[0020] The remaining pixels are predicted using the JPEG-LS method.

[0021] After S4, the residual image is also processed for outliers. Specifically, a threshold T is set for the number of occurrences of the residual. When the frequency of occurrence of a residual value is less than the threshold T, the pixel in the original aurora spectrum image that has the same position as the residual is called an outlier. The residual is directly set to zero, and the modified residual is entropy encoded. Finally, the abnormal pixels together with their positions and the residual code stream are transmitted to the decoding end to restore the original image.

[0022] The JPEG-LS method predicts specifically that,

[0023] Each pixel is predicted using the three pixels to its left, above, and above its left according to the following formula. is the predicted value of the current pixel, x 1 is the pixel to the left of the current pixel, x 2 is the pixel on the top, x 3 is the upper left pixel.

[0024]

[0025] The specific method of obtaining the predicted value through the neural network is:

[0026] Use word embedding technology to embed the context vector of the current pixel p, and map each element in the context vector into a 512-dimensional vector;

[0027] Use position encoding technology to position encode the context vector and generate a 512-dimensional vector for each element according to its position in the context vector;

[0028] Add the word embedding vector to the corresponding position element of the position vector to obtain the final input vector to be input to the neural network;

[0029] Input the input vector to the encoder consisting of two Transformer encoding units to obtain the hidden layer representation y;

[0030] Input y into the decoder consisting of three fully connected layers to get the predicted value of p The sizes of the three fully connected layers are 512, 256, and 1, respectively. ReLU activation function and Dropout layer are added after the first and second layers.

[0031] The context vector is specifically:

[0032] For the image where the pixel to be predicted is located, the number of pixels in the spatial direction on the left side and the number of pixels in the spectral direction on the top side are centered on the pixel to be predicted;

[0033] The image before the pixel to be predicted in the time direction, with the pixel to be predicted as the center, the pixels of the number of spatial pixels on the left and right, and the pixels of the number of pixels above and below in the spectral direction;

[0034] All the above pixels are arranged in a fixed order to form a context vector.

[0035] When selecting pixels in the spatial and spectral directions of the image before the pixel to be predicted in the time direction, the number of pixels is smaller than the number of spatial and spectral reference pixels required for the reference previously set.

[0036] The beneficial effects of the present invention are as follows: the Transformer-based neural network can improve modeling accuracy and prediction accuracy; introducing time correlation and using reference pixels in time, space and spectral directions for prediction can improve prediction accuracy and further eliminate redundancy; introducing exception processing can further reduce the bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the Transformer-based lossless compression method for auroral spectral images based on spatiotemporal-spectral combination.

[0038] Figure 2 It is a neural network structure diagram;

[0039] Figure 3 Schematic diagram of context vector;

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0041] [Example 1]

[0042] A Transformer-based lossless compression method for auroral spectral images based on spatiotemporal and temporal spectral integration.

[0043] This method is a point prediction method. The prediction value of each pixel is calculated from left to right and from top to bottom. After all the pixel prediction values ​​are calculated, the original image is subtracted from the predicted image to obtain the residual image, and then the residual image is entropy encoded to obtain the compressed bit stream to complete the compression. When decompressing, the compressed bit stream is first decoded to obtain the residual image, and then the predicted image is calculated in the same way as the compression method. Finally, the residual image is added to the predicted image to restore the original image losslessly.

[0044] S1, segmenting the continuous original aurora spectrum image, and the divided images have time correlation when compressed;

[0045] For example, if the original image segment length is set to 10, there will be temporal correlation between these 10 images when they are compressed.

[0046] Compared with the compression of single-frame aurora spectral images, the main difference of time-series aurora spectral image compression is the introduction of temporal correlation between adjacent images. Due to the introduction of temporal correlation, to decompress an image, it is necessary to first decompress the previous images on which it depends, and these images in turn depend on earlier images. Ultimately, to decompress an image, all previous images must be decompressed first, resulting in a lot of redundant calculations and waste of resources.

[0047] In order to alleviate this problem, it is necessary to segment the continuous aurora spectrum images during compression. The specific length can be set according to user needs. It is worth noting that the network structure is independent of the length of the time segment, so the network only needs to be trained once and does not need to be adjusted for different lengths. Since the compression of each image in the present invention refers to the previous two images, the first two images of each segment after segmentation cannot obtain enough time reference pixels. For these two images, the present invention does not design a special compression algorithm, but adopts the compression algorithm OnlineDPCM without time correlation for compression.

[0048] The number of temporal reference pixels required for reference is also set for the image to be predicted. If enough temporal reference pixels are available, the image enters S2. If not, Online DPCM is used to compress the entire image.

[0049] First, the number of pixels to be referenced in the time direction is set. For example, if it is set to 2, it means that when predicting the image, that is, compressing it, two images from the previous time are needed as references.

[0050] Among the 10 images, the S3 method is used for prediction starting from the third image, and the first two images are compressed using OnlineDPCM.

[0051] Before image prediction, the number of spatial and spectral reference pixels to be referenced is set for the image to be predicted.

[0052] If the pixel to be predicted cannot obtain a sufficient number of spatial and spectral reference pixels, it is compressed in the following way:

[0053] If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself;

[0054] If the pixel to be predicted is the pixel in the first row or first column of the image, the predicted value is the immediately preceding pixel;

[0055] The remaining pixels are predicted using the JPEG-LS method.

[0056] Since in an image, pixels in the boundary area cannot obtain enough spatial and spectral reference pixels,

[0057] Therefore, for this part of the boundary area, the following method is used for prediction:

[0058] If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself; this is also the first pixel when the image is decompressed, and this step is the key to image decompression;

[0059] The predicted value of the pixel in the first row and the first column is the previous adjacent pixel. Therefore, when decompressing, the pixel in the first row and the first column can be decompressed based on the known pixel in the upper left corner of the image.

[0060] The previous immediately adjacent pixel specifically refers to, taking a row as an example, the pixel at the second position from left to right, and the predicted value is the previous one, that is, the pixel at the first position on the left.

[0061] The predicted value of the pixel at the third position is the pixel at the second position on the left.

[0062] The same is true for columns, except that the previous adjacent pixel is the pixel above. So when decompressing, based on the known pixel in the upper left corner of the image, the pixels of the first row and the first column can be decompressed;

[0063] For pixels other than the first pixel in the upper left corner, the first row, and the first column where there are not enough spatial and spectral reference pixels, the JPEG-LS method is used for prediction.

[0064] The JPEG-LS method predicts specifically that,

[0065] Each pixel is predicted using the three pixels to its left, above, and above the left according to the following formula. is the predicted value of the current pixel, x 1 is the pixel to the left of the current pixel, x 2 is the pixel on the top, x 3 is the upper left pixel.

[0066]

[0067] S2. For the pixel to be predicted, relevant pixels in time, space and spectrum directions are selected as reference pixels, and the reference pixels constitute a context vector;

[0068] For pixels that satisfy the correlation in time, space and spectral directions at the same time, they are used as reference pixels of reference pixels and enter s2 to prepare for prediction.

[0069] Corresponding pixels in time, space and spectral directions are selected as reference pixels to form a context vector for predicting the current pixel.

[0070] The number of spatiotemporal reference pixels selected by the present invention is 20 in total, and the specific selection method is as follows: Figure 3 As shown. Context refers to pixels that have a strong correlation with the pixels to be predicted. Using them for prediction can effectively remove redundant information in the image. A single aurora spectrum image contains spatial and spectral correlations, and there is temporal correlation between adjacent images, so the present invention uses both spatiotemporal and spectral correlations for prediction.

[0071] The context vector is specifically:

[0072] For the image where the pixel to be predicted is located, the number of pixels in the spatial direction on the left side and the number of pixels in the spectral direction on the top side are centered on the pixel to be predicted;

[0073] The image before the pixel to be predicted in the time direction, with the pixel to be predicted as the center, the pixels of the number of spatial pixels on the left and right, and the pixels of the number of pixels above and below in the spectral direction;

[0074] All the above pixels are arranged in a fixed order to form a context vector.

[0075] by Figure 3 For example, the number of pixels in the time direction is 2, and the number of pixels in the spatial and spectral directions is 3.

[0076] Figure 3 In c, the black pixels are to be predicted, and the remaining gray pixels are part of the reference pixels in the context vector.

[0077] The image in the time direction means that when it is used as a reference pixel, the two images before this one are used as references.

[0078] In this image, Figure 3 In Figure c, the spatial direction is the left and right sides, and the spectral direction is the top and bottom sides. Since Figure c has just started to predict from top to bottom and from left to right, the pixels on the right and bottom of the pixel to be predicted are not known yet, so only the pixels of the left 3 and top 3 grids are selected as references.

[0079] As for the first two images, since they have been processed and all the pixels are known, each image selects three left and right grids, three upper and lower grids, and the pixel at the position of the pixel to be predicted, a total of 13 pixels as reference pixels, 13 pixels in image a, 13 pixels in image b, and 6 pixels in image c.

[0080] Arranging all pixels in a fixed order forms a context vector. The specific order does not matter and there is no requirement as long as a fixed order is used.

[0081] When selecting pixels in the spatial and spectral directions of the image before the pixel to be predicted in the time direction, the number of pixels is smaller than the number of spatial and spectral reference pixels required for the reference previously set.

[0082] At the same time, since the previous images in the time direction are inaccurate, when collecting reference pixels in the previous images, the number of pixels in the spatial and spectral directions can be reduced to 3.

[0083] like Figure 3 As shown in b and a, the number of pixels required in b is 3 minus one, which means taking the pixels of 2 grids around and the pixel at the position of the pixel to be predicted, a total of 9 grids as reference pixels.

[0084] Image a is more distant in time and less accurate, so the number of pixels is further reduced, and only the surrounding grids and itself are selected, a total of 5 grids as reference pixels.

[0085] S3, image compression, building a neural network, inputting the context vector into the neural network, and outputting the predicted value of the current pixel;

[0086] Commonly used neural networks include: 1. Fully connected neural network. The constructed context vector is input into the fully connected neural network to output the predicted value of the current pixel; 2. LSTM neural network. Since LSTM is specifically for time series data, the reference pixels in the spatial and spectral directions can also be regarded as time series data. Then, the LSTM network is used for prediction in the time, space and spectral directions respectively to obtain three predicted values ​​(or vectors). Finally, these three values ​​(or vectors) are concatenated and input into the fully connected network to output the predicted value of the current pixel.

[0087] Furthermore, using the modified neural network, the specific architecture of the neural network is as follows Figure 2 As shown, the specific method of obtaining the predicted value through the neural network is:

[0088] The context vector of the current pixel p is embedded using word embedding technology, and each element in the context vector is mapped into a 512-dimensional vector; since the length of the context vector in the present invention is 20, after word embedding, the context vector is mapped into a 20×512-dimensional vector.

[0089] The context vector is positionally encoded using position encoding technology, generating a 512-dimensional vector for each element according to its position in the context vector; therefore, the dimension of the position vector of the entire context vector is 20×512.

[0090] Add the word embedding vector to the corresponding position element of the position vector to obtain the final input vector to be input to the neural network;

[0091] The input vector is fed into an encoder consisting of two Transformer encoding units to obtain the hidden representation y.

[0092] Input y into the decoder consisting of three fully connected layers to get the predicted value of p The sizes of the three fully connected layers are 512, 256, and 1 respectively. ReLU activation function and Dropout layer are added after the first and second layers. The Dropout layer can prevent overfitting by discarding some neurons during training. The batch size during training is 32, the loss function is the mean square error loss MSE, the optimizer is Adam, and the initial learning rate is 0.01.

[0093] FC represents the fully connected layer, p represents the current pixel, Represents the predicted value of the current pixel, and y is the hidden layer representation.

[0094] S4, repeat S3 until all pixels are predicted, obtain a predicted image according to the predicted values ​​of all pixels, and subtract the predicted image from the original image to obtain a residual image;

[0095] Repeat the steps in S3 to predict all pixels, obtain the predicted values ​​of all pixels, and finally obtain a predicted image. Subtract the predicted image from the original aurora spectrum image to obtain a residual image.

[0096] After S4, the residual image is also processed for outliers. Specifically, a threshold T is set for the number of occurrences of the residual. When the frequency of occurrence of a residual value is less than the threshold T, the pixel in the original aurora spectrum image that has the same position as the residual is called an outlier. The residual is directly set to zero, and the modified residual is entropy encoded. Finally, the abnormal pixels together with their positions and the residual code stream are transmitted to the decoding end to restore the original image.

[0097] The residual image is subjected to exception processing. There are some values ​​in the residual image that have large absolute values ​​but very few occurrences. The increase in code rate brought about by encoding these values ​​together with other residuals is generally greater than the increase in code rate brought about by directly transmitting the original pixel values ​​corresponding to these residuals. In order to solve this problem, the present invention proposes to set a threshold value T for the number of times the residual occurs. When the frequency of occurrence of a certain residual value is less than the threshold value T, the pixel in the original aurora spectrum image that has the same position as the residual is called an abnormal value. The residual is directly set to zero, and the modified residual is entropy encoded. Finally, the abnormal pixels are transmitted together with their positions and the residual code stream to the decompression end for restoring the original image. Since the abnormal pixel value itself is transmitted to the decompression end, no further calculation is required when decompressing to this position. Therefore, the use of this abnormal value processing method can reduce both the code rate and the amount of calculation during decompression.

[0098] Since an aurora spectrum image has a total of 1024×1024 pixels, 3 bytes are required to store the location of an outlier. Since the maximum value of a pixel is 16383, 2 bytes are required to store an outlier. Therefore, a total of 5 bytes are required to store an outlier and its location.

[0099] S5, entropy encoding the residual image (i.e. Figure 3 Entropy Coding in the code is used to obtain a bitstream file; adaptive arithmetic coding is used to perform entropy coding to obtain a bitstream file. The entire process from constructing the context vector to the end of residual entropy coding is the image compression process.

[0100] S6. Decode the bitstream file to obtain a residual image, and calculate a prediction value for each pixel of the residual image through a neural network. The original pixel value can be obtained by adding the prediction value to the residual at the corresponding position in the residual image, thereby obtaining a lossless original aurora spectrum image.

[0101] This step is to decompress the code stream file to obtain the original image. Specifically,

[0102] The residual image is predicted according to the same steps as compression, i.e. compressed, to obtain the predicted value of each pixel in the residual image. The predicted value is added to the residual value of the corresponding position in the residual image, i.e. the pixel of the original image, to obtain a lossless original aurora spectrum image.

[0103] Specifically, the first pixel in the upper left corner of the original image is the first value in the upper left corner of the residual image;

[0104] The pixel at the second position of the first row of the original image is the pixel at the first position plus the residual at the second position, the pixel at the third position is the pixel at the second position plus the residual at the third position, and so on, to get the first row of pixels. The pixel at the second position of the first column of the original image is the pixel at the first position plus the residual at the second position, the pixel at the third position is the pixel at the second position plus the residual at the third position, and so on, to get the first column of pixels.

[0105] For the remaining pixels that cannot obtain enough spatial spectrum reference pixels, the JPEG-LS algorithm is used to calculate the predicted value, and then the residual at the corresponding position is added to obtain the original pixel;

[0106] For pixels that can obtain enough spatial spectral reference pixels, a context vector is constructed and input into the network trained during compression to obtain a predicted value. Then, the predicted value is added to the residual at the corresponding position to obtain the original pixel.

Claims

1. A Transformer-based spatial-temporal-spectral joint auroral spectrum image lossless compression method, characterized by: S1, segmenting the continuous original aurora spectrum image, and the divided images have time correlation when compressed; S2. For the pixel to be predicted, relevant pixels in time, space and spectrum directions are selected as reference pixels, and the reference pixels constitute a context vector; S3, image compression, building a neural network, inputting the context vector into the neural network, and outputting the predicted value of the current pixel; S4, repeat S3 until all pixels are predicted, obtain a predicted image according to the predicted values ​​of all pixels, and subtract the predicted image from the original image to obtain a residual image; S5, performing entropy coding on the residual image to obtain a bitstream file; S6. Decode the bitstream file to obtain a residual image, and calculate a prediction value for each pixel of the residual image through a neural network. The original pixel value can be obtained by adding the prediction value to the residual at the corresponding position in the residual image, thereby obtaining a lossless original aurora spectrum image.

2. According to claim 1, the method for lossless compression of auroral spectrum images based on Transformer-based spatiotemporal-spectral combination is characterized by: The number of temporal reference pixels required for reference is also set for the image to be predicted. If enough temporal reference pixels are available, the image enters S2. If not, Online DPCM is used to compress the entire image.

3. The method for lossless compression of auroral spectrum images based on Transformer-based spatiotemporal-spectral combination according to claim 1 or 2, characterized in that: Before image prediction, the number of spatial and spectral reference pixels to be referenced is set for the image to be predicted. If the pixel to be predicted cannot obtain a sufficient number of spatial and spectral reference pixels, it is compressed in the following way: If the pixel to be predicted is the first pixel in the image, that is, the pixel in the upper left corner, the predicted value is 0 and the residual is the pixel value itself; If the pixel to be predicted is the pixel in the first row or first column of the image, the predicted value is the immediately preceding pixel; The remaining pixels are predicted using the JPEG-LS method.

4. The method for lossless compression of auroral spectrum images based on the combined temporal, spatial and spectral properties of Transformer according to claim 1, characterized in that: After S4, the residual image is also processed for outliers. Specifically, a threshold T is set for the number of occurrences of the residual. When the frequency of occurrence of a residual value is less than the threshold T, the pixel in the original aurora spectrum image that has the same position as the residual is called an outlier. The residual is directly set to zero, and the modified residual is entropy encoded. Finally, the abnormal pixels together with their positions and the residual code stream are transmitted to the decoding end to restore the original image.

5. The method for lossless compression of auroral spectrum images based on the combined temporal, spatial and spectral properties of Transformer according to claim 3, characterized in that: The JPEG-LS method predicts specifically that, Each pixel is predicted using the three pixels to its left, above, and above its left according to the following formula. is the predicted value of the current pixel, x1 is the pixel to the left of the current pixel, x2 is the pixel above, and x3 is the pixel on the upper left.

6. The method for lossless compression of auroral spectrum images based on Transformer-based spatiotemporal-spectral combination according to claim 1, characterized in that: The specific method of obtaining the predicted value through the neural network is: Use word embedding technology to embed the context vector of the current pixel p, and map each element in the context vector into a 512-dimensional vector; Use position encoding technology to position encode the context vector and generate a 512-dimensional vector for each element according to its position in the context vector; Add the word embedding vector to the corresponding position element of the position vector to obtain the final input vector to be input to the neural network; Input the input vector to the encoder consisting of two Transformer encoding units to obtain the hidden layer representation y; Input y into the decoder consisting of three fully connected layers to get the predicted value of p The sizes of the three fully connected layers are 512, 256, and 1, respectively. ReLU activation function and Dropout layer are added after the first and second layers.

7. The method for lossless compression of auroral spectrum images based on the combined temporal, spatial and spectral properties of Transformer according to claim 3, characterized in that: The context vector is specifically: For the image where the pixel to be predicted is located, the number of pixels in the spatial direction on the left side and the number of pixels in the spectral direction on the top side are centered on the pixel to be predicted; The image before the pixel to be predicted in the time direction, with the pixel to be predicted as the center, the pixels of the number of spatial pixels on the left and right, and the pixels of the number of pixels above and below in the spectral direction; All the above pixels are arranged in a fixed order to form a context vector.

8. The method for lossless compression of auroral spectrum images based on the combined temporal, spatial and spectral properties of Transformer according to claim 7, characterized in that: When selecting pixels in the spatial and spectral directions of the image before the pixel to be predicted in the time direction, the number of pixels is smaller than the number of spatial and spectral reference pixels required for the reference previously set.

Images