An infrared hyperspectral detection data compression method, device and electronic equipment

Abstract

The application discloses an infrared hyperspectral detection data compression method and device and electronic equipment. The method comprises the following steps: carrying out noise normalization preprocessing on infrared hyperspectral detection data; obtaining global principal component scores by carrying out global principal component analysis on the noise normalization preprocessed data, and quantizing the global principal component scores; obtaining global reconstruction data by carrying out global reconstruction, and calculating global reconstruction residuals; carrying out local principal component analysis on the global reconstruction residuals file by file; and carrying out Gzip encoding compression to obtain a compressed data file. Through the infrared hyperspectral detection data compression method and device and electronic equipment provided in the embodiment of the application, a two-stage hybrid compression architecture of global PCA and local PCA can be realized, so that the technical problems of insufficient reconstruction accuracy of a traditional global PCA method on special observation data, lack of individual difference adaptive compensation of a single file and waste of effective information of global reconstruction residuals are solved.

Description

Technical Field

[0001] This application belongs to the field of meteorological satellite remote sensing data processing technology, and in particular relates to an infrared hyperspectral detection data compression method, device and electronic equipment. Background Technology

[0002] The infrared hyperspectral atmospheric sounder acquires atmospheric radiation data from the Earth through thousands of spectral channels. With the exponential increase in the number of channels, the amount of raw data grows exponentially, placing enormous pressure on onboard storage and the data transmission downlink. Currently, mainstream compression schemes primarily employ a global projection mechanism based on principal component analysis (PCA), which uses the global mean and global eigenvector basis obtained from historical statistical training to reduce the dimensionality of the observation data. However, this existing technology has significant drawbacks: First, the global basis can only characterize the high-frequency statistical regularities of large datasets. For atypical spectral characteristics caused by extreme weather events such as typhoons and severe convection, or special surface conditions, the global basis cannot fully represent them, resulting in severely compromised reconstruction accuracy for such special observation data. Second, traditional methods use a uniform global basis to process all data, lacking an adaptive compensation mechanism specific to individual observation files, leading to the loss of individual differences between files. Furthermore, existing schemes typically discard the residuals after global reconstruction, failing to utilize the structured spectral information contained within the residuals, resulting in a significant waste of recoverable data, making it difficult to achieve high-precision reconstruction while maintaining a high compression ratio.

[0003] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] To address the existing technical problems, embodiments of the present invention provide an infrared hyperspectral detection data compression method and apparatus.

[0005] In a first aspect, embodiments of this application provide a method for compressing infrared hyperspectral detection data, comprising: performing noise normalization preprocessing on the infrared hyperspectral detection data; performing global principal component analysis on the noise-normalized preprocessed infrared hyperspectral detection data to obtain global principal component scores, and quantizing the global principal component scores; performing global reconstruction based on the quantized global principal component scores to obtain globally reconstructed data, and calculating the residual between the original radiance data and the globally reconstructed data to obtain global reconstruction residuals; performing file-by-file local principal component analysis on the global reconstruction residuals to obtain local principal component scores, local eigenvectors, and local means; and encapsulating the quantized global principal component scores, local principal component scores, local eigenvectors, local means, and quantization parameters, and then performing Gzip encoding compression to obtain a compressed data file.

[0006] Optionally, the infrared hyperspectral detection data is subjected to noise normalization preprocessing, including: using a pre-trained global noise normalization coefficient vector to perform noise normalization processing on each spectral channel; wherein, the emissivity value R of each channel is divided by its corresponding noise standard deviation NC to obtain the noise-normalized emissivity value.

[0007] Optionally, global principal component analysis is performed on the infrared hyperspectral detection data after noise normalization preprocessing to obtain global principal component scores, and the global principal component scores are quantized, including: subtracting the global mean spectrum from each spectral vector of the noise-normalized observation data to obtain centered data; projecting the centered data onto a pre-trained global feature vector basis to obtain global principal component scores; and linearly quantizing the global principal component scores according to a preset quantization step width and offset to obtain quantized global principal component scores, thus obtaining quantization parameters.

[0008] Optionally, the global feature vector basis is pre-trained in the following way: using a large-scale historical observation dataset, the data of each spectral band is pre-processed by noise normalization and then centered; the feature vector matrix is ​​obtained by singular value decomposition, and the feature vectors are selected and retained in descending order of the singular values ​​to form the global feature vector basis.

[0009] Optionally, file-by-file local principal component analysis is performed on the global reconstruction residuals to obtain local principal component scores, local eigenvectors, and local means, including: constructing a residual data matrix for each observation data file; calculating the local mean of the residual data matrix; performing eigenvalue decomposition on the residual data matrix to obtain local eigenvectors; and extracting a small number of local principal components to obtain the local principal component scores, the local eigenvectors, and the local mean.

[0010] Optionally, the number of local principal components is 5.

[0011] Optionally, the quantized global principal component score, the local principal component score, the local feature vector, the local mean, and the quantization parameter are encapsulated and then compressed using Gzip encoding to obtain a compressed data file. This includes: encapsulating the quantized global principal component score, the local principal component score, the local feature vector, the local mean, the quantization parameter, the data file identifier, and the observation geometric information in a predetermined format; and compressing the encapsulated data using the Gzip lossless encoding method.

[0012] Secondly, embodiments of this application provide an infrared hyperspectral detection data compression device, comprising: a preprocessing module for performing noise normalization preprocessing on the infrared hyperspectral detection data; a global PCA module for performing global principal component analysis on the noise-normalized infrared hyperspectral detection data to obtain global principal component scores, and quantizing the global principal component scores; a residual calculation module for performing global reconstruction based on the quantized global principal component scores to obtain global reconstructed data, and calculating the residual between the original radiance data and the global reconstructed data to obtain global reconstruction residuals; a local PCA module for performing file-by-file local principal component analysis on the global reconstruction residuals to obtain local principal component scores, local eigenvectors, and local means; and a packaging and compression module for packaging the quantized global principal component scores, local principal component scores, local eigenvectors, local means, and quantization parameters and then performing Gzip encoding compression to obtain a compressed data file.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, the processor executes the computer program stored in the memory, and the computer program, when executed by the processor, implements the infrared hyperspectral detection data compression method described in the first aspect.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the infrared hyperspectral detection data compression method described in the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the infrared hyperspectral detection data compression method described in any of the first aspects above.

[0016] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0017] The beneficial effects of this application embodiment compared with the prior art are as follows: The infrared hyperspectral detection data compression method, system, device and medium provided by the embodiments of the present invention effectively solve the problems of the inability of the global basis to represent atypical spectral features and the loss of individual difference information between files in the prior art by adopting a two-level residual compensation mechanism of "global principal component analysis + local principal component analysis". First, by using the pre-trained global feature vector basis to reduce the dimensionality of large-scale datasets, the high correlation between data can be efficiently removed, realizing compression at the basic level; second, in response to the residual error after global reconstruction, the present invention innovatively introduces file-by-file local principal component analysis to specifically extract and compress the residual information unique to each observation file, thereby significantly improving the reconstruction accuracy of abnormal data such as typhoons, strong convection and other extreme weather or special surface conditions, and avoiding the information waste caused by discarding residuals in traditional methods. Furthermore, this method performs noise normalization on the data before compression, eliminating the impact of differences in noise standard deviations across different spectral channels, thus making principal component analysis more robust. In the encapsulation stage, metadata such as quantization parameters, local statistics, and observational geometric information are encoded together with the core compressed data using Gzip lossless encoding, ensuring data integrity while further improving compression efficiency. In summary, this invention significantly reduces the storage and transmission bandwidth requirements of infrared hyperspectral detection data while maintaining extremely high reconstruction accuracy. It is particularly suitable for high-volume remote sensing data transmission in resource-constrained onboard scenarios, greatly expanding the application value of hyperspectral detection data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of an infrared hyperspectral detection data compression method provided by an embodiment of the present invention is shown;

[0020] Figure 2 A detailed flowchart of the infrared hyperspectral detection data compression method provided in an embodiment of the present invention is shown.

[0021] Figure 3 A comparison chart of the global PCA reconstruction residual and the Hybrid PCA reconstruction residual in an embodiment of the present invention is shown;

[0022] Figure 4 This diagram illustrates the structure of an infrared hyperspectral detection data compression device provided in an embodiment of the present invention.

[0023] Figure 5 The diagram shows a schematic of an electronic device for performing an infrared hyperspectral detection data compression method according to an embodiment of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0025] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0026] Figure 1 A flowchart of an infrared hyperspectral detection data compression method provided by an embodiment of the present invention is shown. Figure 1 As shown, the method includes the following steps 101-105.

[0027] Step 101: Perform noise normalization preprocessing on the infrared hyperspectral detection data.

[0028] Specifically, the compression end first loads a pre-trained global noise normalization coefficient (hereinafter referred to as NC coefficient, also known as noise standard deviation) vector. This coefficient vector is obtained through training with large-scale historical observation data, covering multiple seasons and geographical conditions. The compression end processes the raw observation data acquired by the infrared hyperspectral detector channel by channel, dividing the radiance value R of each spectral channel by the corresponding NC coefficient to obtain the noise-normalized radiance value. The corresponding noise normalization formula is as follows: This step ensures that each spectral channel has a uniform noise weight in the subsequent principal component analysis decomposition, preventing high-noise channels from dominating the extraction of principal components. This is especially important when there are significant differences in the noise levels of different channels in an infrared detector.

[0029] Since noise coefficient data exists in the HIRAS data file, in this embodiment of the invention, the NC coefficients can be obtained by collecting large-scale historical observation data covering multiple seasons and geographical conditions, and extracting noise coefficient data from the historical observation data. This noise normalization preprocessing step ensures that each spectral channel has a uniform noise weight in the subsequent principal component analysis, eliminating the influence of differences in noise levels between different channels on the principal component analysis results.

[0030] Step 102: Perform global principal component analysis on the infrared hyperspectral detection data after noise normalization preprocessing to obtain the global principal component score, and quantify the global principal component score.

[0031] Specifically, the compression end subtracts the global mean spectrum from each spectral vector of the noise-normalized observation data. This yields centralized data. Global mean spectrum. Also obtained through statistical training using large-scale historical observation data, it reflects the average spectral characteristics of long-term observations. The compression end projects the centered data onto a pre-trained global feature vector base. The global principal component score is obtained. The formula satisfies: .

[0032] The training method for the global eigenvector basis is as follows: Large-scale historical observation data covering multiple seasons and geographical conditions is collected as the training set. After being divided by spectral bands, noise normalization preprocessing and centering processing are performed. The eigenvector matrix is ​​obtained through singular value decomposition. The first few eigenvectors are selected and retained in descending order of singular values ​​to form the global eigenvector basis. In this embodiment, different spectral bands can retain different numbers of principal components to adapt to the spectral variability characteristics of each band. High-variability bands retain more principal components, while low-variability bands retain fewer principal components.

[0033] Specifically, the compression end performs linear quantization on the global principal component scores according to the preset quantization step width and offset, mapping floating-point numbers to integers and reducing the data bit width. The quantization formula is expressed as: ,in The global principal component score is the quantized result. and The quantization step width and offset (i.e., quantization parameters) are preset for each band, with `round` representing rounding to the nearest integer. The compression end simultaneously records the corresponding quantization parameters `step` and `offset` for each band, used for subsequent dequantization recovery in the decompression end. This quantization step significantly reduces the data volume while preserving principal component score information, creating favorable conditions for subsequent compression.

[0034] Step 103: Perform global reconstruction based on the quantized global principal component scores to obtain global reconstruction data, and calculate the residual between the original radiance data and the global reconstruction data to obtain the global reconstruction residual.

[0035] Specifically, the compression end processes the quantized global principal component score. Perform inverse quantization processing according to the formula. Restore the floating-point score, where, This is the global principal component score after dequantization. The compression process utilizes the global eigenvector basis. and global mean spectrum Perform inverse PCA transformation to project the principal component scores back into the noise normalization space to obtain the globally reconstructed data. The inverse PCA transformation formula is expressed as: ,in For globally reconstructing the data, T stands for transpose. This means swapping the rows and columns of matrix V. It is the transpose of the global eigenvector matrix, where μ is the global mean spectral vector. The compression stage further calculates the original radiance data and the globally reconstructed data. The residuals under noise normalization are calculated using the following formula: ,in The original emissivity data after noise normalization. The global reconstructed data after noise normalization. To reconstruct the residuals globally.

[0036] In this embodiment of the invention, the residual contains spectral variation components that the global PCA base fails to express effectively. It is not completely random noise, but contains certain structured spectral information.

[0037] Step 104: Perform file-by-file local principal component analysis on the global reconstruction residuals to obtain the local principal component scores, local eigenvectors, and local means.

[0038] Specifically, the compression end processes the global reconstruction residual obtained in step 103. Perform per-granule local PCA processing. The rows of the residual data matrix correspond to spectral channels, and the columns correspond to observed sample points. The local mean of the residual data matrix is ​​calculated at the compression stage. The local mean This reflects the average offset of the current file residuals. The compression end performs eigenvalue decomposition on the centered residual data matrix to obtain local eigenvectors and their corresponding eigenvalues. Finally, a small number of local principal components are extracted to obtain local principal component scores, local eigenvectors, and local means.

[0039] This local PCA analysis step can adaptively compensate for the unique spectral characteristics of individual observation files that distinguish them from the statistical average state, thereby enabling the supplementary capture of missing information in the global basis.

[0040] Step 105: Encapsulate the quantized global principal component score, local principal component score, local feature vector, local mean, and quantization parameters, and then compress them using Gzip encoding to obtain a compressed data file.

[0041] Specifically, the compression end encapsulates the quantized global principal component scores, local principal component scores, local eigenvectors, local means, quantization parameters, and other necessary metadata (such as data file identifiers and observation geometric information) according to a predetermined format. The compression end then uses the Gzip lossless encoding method to perform final compression on the encapsulated data, generating a compressed data file. The Gzip algorithm is based on the DEFLATE algorithm and combines LZ77 compression and Huffman coding, effectively removing redundant information from the data. Through this encapsulation and compression process, a compact data file suitable for onboard storage and downlink data transmission can be generated.

[0042] The above method effectively improves the reconstruction accuracy of special observational data that deviates from the statistical average. This invention uses local PCA to perform a file-by-file secondary decomposition of the global reconstruction residuals, extracting structured spectral information not fully expressed by the global basis and incorporating it into the compressed data stream. For observational data containing extreme weather events, special surface conditions, or anomalous atmospheric characteristics, this information is lost in traditional global PCA methods. This invention effectively preserves this information through a local compensation mechanism, thereby reducing the reconstruction error of such data.

[0043] Secondly, it balances the advantages of global comprehensiveness with local individuality. This invention adopts a two-stage hybrid architecture of "global PCA + local PCA," reflecting the organic combination of statistical commonality and individual characteristics. Global PCA utilizes a global basis trained on large-scale historical data to capture the dominant spectral variation patterns commonly found in observational data, ensuring efficient compression and good reconstruction of most conventional observational data, demonstrating the method's comprehensiveness and universality. Local PCA performs personalized processing on the global reconstruction residuals of individual observation files, adaptively compensating for the unique spectral features of those files, demonstrating the ability to refine the processing of local individuals. The two PCA methods are functionally complementary: global PCA is responsible for the efficient expression of commonalities, while local PCA is responsible for the supplementary capture of individual characteristics.

[0044] Furthermore, this invention maintains high compression efficiency. The overall compression ratio of this invention is approximately 2.5% (relative to effective real part emissivity data), which is on par with traditional global PCA methods. The introduction of the local PCA stage does not significantly affect the overall compression ratio because the amount of increased local principal component data is much smaller than the amount of global principal component data.

[0045] In short, the embodiments of the present invention can realize a two-stage hybrid compression architecture of global PCA and local PCA, thereby solving the technical problems of insufficient reconstruction accuracy of special observation data, lack of adaptive compensation for individual differences in single files, and waste of effective information in global reconstruction residuals in traditional global PCA methods.

[0046] Optionally, the infrared hyperspectral detection data is subjected to noise normalization preprocessing, including: using a pre-trained global noise normalization coefficient vector to perform noise normalization processing on each spectral channel; wherein, the emissivity value R of each channel is divided by its corresponding noise standard deviation NC to obtain the noise-normalized emissivity value.

[0047] Specifically, during the preprocessing stage, the compression end first accesses the pre-distributed data storage area and loads the pre-trained global noise normalization coefficient vector. This global noise normalization coefficient vector is a vector with the same dimension as the number of spectral channels, with each element corresponding to the noise normalization coefficient of one spectral channel. The compression end then scans the observation data to be compressed channel by channel, reads the radiance value R of each spectral channel, extracts the NC parameters of the corresponding channel from the global noise normalization coefficient vector, and performs a division operation. The noise-normalized emissivity value is obtained. In this embodiment of the invention, the noise normalization coefficient vector, along with the global eigenvector basis and the global mean spectrum, are pre-distributed to the compression and decompression ends to ensure consistency in processing between the compression and decompression ends. This setting unifies the noise weights of each spectral channel, providing a uniform numerical scale for subsequent principal component analysis.

[0048] Optionally, global principal component analysis is performed on the infrared hyperspectral detection data after noise normalization preprocessing to obtain global principal component scores, and the global principal component scores are quantified, including the following steps A1-A3.

[0049] Step A1: Subtract the global mean spectrum from each spectral vector of the noise-normalized observation data to obtain the centered data.

[0050] Specifically, the compression module loads the global mean spectral vector from the pre-distributed data. This vector, obtained through statistical training using large-scale historical observation data, represents the average spectral characteristics of long-term observations. The compression module then subtracts the global mean spectral vector element-wise from the noise-normalized observation data matrix to obtain a centered data matrix. This centering process is a standardized pre-operation for principal component analysis, ensuring that the subsequent projection process can correctly extract the main directions of data variation. This setting eliminates the interference of the data mean on principal component analysis, allowing the principal component directions to truly reflect the data's variation patterns.

[0051] Step A2: Project the centered data onto a pre-trained global feature vector basis to obtain the global principal component score.

[0052] Specifically, the compression end loads a global eigenvector matrix V from the pre-distributed data. This matrix is ​​obtained through singular value decomposition training on large-scale historical observation data, with each column corresponding to the principal component direction. The compression end performs matrix multiplication on the centered data matrix and the global eigenvector matrix to obtain the global principal component score matrix. This projection operation projects the original high-dimensional spectral data onto a low-dimensional subspace spanned by the global eigenvectors, achieving data dimensionality reduction. In this embodiment, the global eigenvector basis can be stored separately by band, with different eigenvector bases used for different bands to adapt to the spectral variability characteristics of each band. Through this setting, the dominant spectral variability patterns in the observation data can be captured using the global eigenvector basis.

[0053] Step A3: Perform linear quantization on the global principal component score according to the preset quantization step width and offset to obtain the quantized global principal component score and obtain the quantization parameters.

[0054] Specifically, the compression end presets the corresponding quantization step width and offset based on the numerical distribution range of the principal component scores for each band. The principle for setting the quantization step width is to maximize the step width while meeting the reconstruction accuracy requirements to reduce the data volume; the principle for setting the offset is to shift the numerical range of the principal component scores to the non-negative interval for easier integer representation. Through this setting, the principal component scores, represented as floating-point numbers, can be converted to integer representations, significantly reducing the data bit width.

[0055] Optionally, the global eigenvector basis is pre-trained in the following way: using a large-scale historical observation dataset, the data of each spectral band is preprocessed by noise normalization and then centered; the eigenvector matrix is ​​obtained by singular value decomposition, and some eigenvectors are selected and retained in descending order of singular values ​​to form the global eigenvector basis.

[0056] In this embodiment of the invention, large-scale historical observation data covering multiple seasons and geographical conditions are collected as a training set. After being divided according to spectral bands, noise normalization preprocessing and centering processing are performed. The eigenvector matrix is ​​obtained through Singular Value Decomposition (SVD). The first few eigenvectors are selected and retained in descending order of singular values ​​to form a global eigenvector basis.

[0057] Optionally, file-by-file local principal component analysis is performed on the global reconstruction residuals to obtain local principal component scores, local eigenvectors, and local means, including the following steps B1-B4.

[0058] Step B1: Construct a residual data matrix for each observation data file.

[0059] Specifically, the compression end organizes the global reconstruction residuals calculated in step 103 into a residual data matrix. The row dimension of this residual data matrix corresponds to the number of spectral channels, and the column dimension corresponds to the number of samples in the current observation file. Each observation file typically contains several observation scan points or time-series samples, which together constitute the column dimension of the residual data matrix. In this embodiment of the invention, for HIRAS instruments, a single observation file may contain hundreds to thousands of spectral samples. The compression end performs a validity check on the residual data matrix to ensure that the matrix dimensions are correct and that there are no missing values. This setting provides a standardized data organization format for subsequent local PCA analysis.

[0060] Step B2: Calculate the local mean of the residual data matrix.

[0061] Specifically, a local mean is first obtained, which reflects the average offset of the current file's residuals and represents the unique residual characteristics of that file that distinguish it from the global mean. The compression end is based on a formula... Centering the residual matrix yields the centered residual data, where... For the global reconstruction residual matrix of the g-th observation granule, It is a local mean. The centralized residual is the result of subtracting the local mean from the g-th block. In this embodiment of the invention, the compression end stores the local mean and the global reconstruction residual together for mean compensation during subsequent residual reconstruction. This setting allows the average offset characteristics of the residuals of a single file to be captured.

[0062] Step B3: Perform eigenvalue decomposition on the residual data matrix to obtain local eigenvectors.

[0063] Specifically, the compression end performs eigenvalue decomposition on the centered residual matrix, which can be achieved using the singular value decomposition method. Local eigenvectors are obtained through eigenvalue decomposition. .

[0064] Step B4: Extract a small number of local principal components to obtain local principal component scores, local feature vectors, and local means.

[0065] Specifically, a small number of local principal components are extracted to obtain local principal component scores. Local feature vectors and local mean (Where K is the number of local principal components retained). Optionally, the number of local principal components is 5. In this embodiment of the invention, only a small number of local principal components (usually 5) need to be retained in the local PCA stage, and the amount of additional data generated is very small compared to the amount of global principal component data. Experiments show that after adding 5 local principal components on the basis of global PCA, the magnitude of the reconstructed residual is reduced by about one order of magnitude. This feature of exchanging a large accuracy improvement with a small storage cost makes this invention have good engineering applicability, and can exchange a significant improvement in reconstruction accuracy for a small amount of additional data. The dimension of the local feature vector matrix is ​​the number of spectral channels multiplied by K. With this setting, the main variation direction in the residual data can be extracted. Its inverse transformation formula is: first calculate the approximate value of the centered residual. Adding the mean yields an approximate residual. .in, , These are the local eigenvectors and eigenvalues ​​obtained from the g-th block residual decomposition. The score of the local principal component corresponding to the g-th block; The approximate residual after centering; This is an approximate residual.

[0066] The key role of this step is that the global PCA basis is obtained through statistical training on large-scale historical data, preserving the most statistically significant variation directions in the training dataset. For spectral features in individual observation files that deviate from the statistical mean (such as anomalous radiation signals caused by extreme weather events), their representation in the global basis may be insufficient, resulting in structured biases in the global reconstruction residuals. Local PCA directly extracts the main variation patterns from the residuals, essentially performing a supplementary secondary capture of information missed by the global PCA, allowing this information to be preserved in the compressed data and recovered during decompression. Since the signal-to-noise ratio of the residuals is much lower than that of the original data, only a very small number of local principal components are needed to effectively represent the main structured information in the residuals.

[0067] Optionally, the quantized global principal component score, local principal component score, local feature vector, local mean, and quantization parameters are encapsulated and then compressed using Gzip encoding to obtain a compressed data file, including the following steps C1-C2.

[0068] Step C1: Encapsulate the quantized global principal component scores, local principal component scores, local eigenvectors, local means, quantization parameters, data file identifiers, and observation geometric information in a predetermined format.

[0069] Specifically, the compression end organizes each data component into a continuous byte stream according to a predetermined binary encapsulation format, which is h5 format. In this embodiment of the invention, little-endian is used to store multi-byte integers during encapsulation, and floating-point numbers are represented using the IEEE 754 standard single-precision or double-precision representation. This configuration allows for the generation of structured compressed input data.

[0070] Step C2: Compress the encapsulated data using the Gzip lossless encoding method.

[0071] Specifically, the compression end calls the Gzip compression algorithm to perform final compression on the encapsulated byte stream. The Gzip algorithm is based on the DEFLATE algorithm and internally includes two stages: LZ77 sliding window matching compression and Huffman coding. The LZ77 stage eliminates data redundancy by finding duplicate strings and replacing them with historical offset references; the Huffman coding stage dynamically constructs an encoding table based on character frequency to perform variable-length encoding on the data. The compression end sets an appropriate compression level to achieve a balance between compression ratio and computational complexity. The output of Gzip compression is a standard .gz format byte stream, which can be directly used for on-board storage or downlink data transmission. This setting further reduces the final data size while ensuring data integrity.

[0072] This invention, through the aforementioned technical solutions, achieves efficient compression and high-precision reconstruction of infrared hyperspectral detection data, thereby solving problems in existing technologies such as insufficient reconstruction accuracy for special observation data, lack of adaptive compensation for individual differences in single files, waste of effective information in global reconstruction residuals, and difficulty in balancing compression ratio and reconstruction accuracy. This invention uses noise normalization preprocessing to ensure uniform noise weights for each spectral channel in principal component analysis, improving the rationality of the analysis. This invention uses global PCA to capture prevalent dominant spectral variation patterns, achieving efficient dimensionality reduction; and uses local PCA to compensate for the unique spectral features of individual files, achieving an organic combination of statistical commonality and individual characteristics. This invention uses local PCA processing on the global reconstruction residuals, which are directly discarded by traditional methods, to extract and retain the structured information, significantly improving the reconstruction accuracy of special observation data such as extreme weather events and special surface conditions. This invention uses a lightweight local compensation strategy, requiring only the retention of a small number of local principal components to reduce the reconstruction residual amplitude by approximately one order of magnitude, achieving a significant improvement in reconstruction accuracy with a small incremental data cost. Through the above technical solution, the present invention achieves a balance between high compression ratio and high reconstruction accuracy, maintaining an overall compression ratio of approximately 2.5%, and has good engineering application value.

[0073] Furthermore, embodiments of the present invention can also perform a corresponding decompression and reconstruction process, specifically: performing Gzip decoding on the compressed data file to restore each data component; performing inverse quantization on the global principal component score to restore the floating-point score; performing inverse PCA transformation using the global eigenvector basis and the global mean, and reconstructing the residual data using the local principal component score, local eigenvector, and local mean; and adding the globally reconstructed data with the locally reconstructed residual.

[0074] The final reconstructed noise-normalized emissivity data are obtained. .

[0075] Then, inverse noise normalization is performed to obtain the original data for reconstruction. .

[0076] Based on the above methods, refer to Figure 2 The following is a detailed flowchart of the infrared hyperspectral detection data compression method provided in the embodiments of the present invention (RMSE stands for Root mean square error). The process will be further detailed with a specific example.

[0077] This embodiment uses Level 1 radiance data from the FY-3F satellite HIRAS instrument as the processing object to illustrate the specific implementation of the present invention. HIRAS data includes three spectral bands: longwave (LW: 650–1160 cm⁻¹), midwave 1 (MW1: 1160–2000 cm⁻¹), and midwave 2 (MW2: 2000–2550 cm⁻¹). The radiance data in each observation data file (granule) is organized in the form of a four-dimensional array, with dimensions of [number of scan lines × number of Earth-based observation steps × number of detectors × number of spectral channels], typically [38 × 28 × 9 × N_channels]. Only the real part of the radiance data is valid; the imaginary part is a noise estimate. Compression processing is performed only on the real part. The typical effective real part data size of a single data file is approximately 100 MB.

[0078] Step 1: Pre-training of noise normalization coefficients and global PCA base.

[0079] A total of 15 days of complete HIRAS Level 1 data from August to December 2024 was selected as the training set (approximately 400 GB, about 40 million spectra), covering multiple seasons and geographical regions. The data was processed separately into three bands: for each band, the first three spatial dimensions (number of scan lines, number of ground-based probe steps, and number of probes) were stacked and flattened into a unified sample dimension, retaining the spectral channel dimension, forming a two-dimensional data matrix of N_samples x N_channels. First, noise normalization coefficients (NC coefficients) were trained: NC coefficients were extracted from the observation data. Then, global PCA basis training was performed: noise normalization was performed on the data of each band using the NC coefficients, and then the normalized data was centered (subtracting the mean spectrum). Eigenvector matrices and singular values ​​were obtained through singular value decomposition (SVD). After sorting the singular values ​​from largest to smallest, the first few eigenvectors were retained to form a truncated global eigenvector basis. In this embodiment, the LW band retains 40 principal components, the MW1 band retains 45 principal components, and the MW2 band retains 35 principal components. The truncated global eigenvector basis is then used. Mean spectrum The noise normalization coefficient NC is saved as a feature vector file (EV file) and pre-distributed to the compression and decompression ends.

[0080] Step 2: Noise normalization and global PCA projection quantization.

[0081] For a single observation data file to be compressed, data preprocessing is first performed, flattening the four-dimensional data into two-dimensional spectral matrices by band. For each band, noise normalization is performed on each channel using a pre-trained global noise normalization coefficient (NC), i.e., the radiance value of each channel is divided by the corresponding NC coefficient. The noise-normalized spectral vector is then subtracted from the global mean spectrum. Then, project onto the global feature vector basis. The global principal component score is obtained. Linear quantization is applied to the global principal component scores, mapping floating-point numbers to integers and reducing the data bit width. The quantization formula is as follows: ,in and The preset quantization step width and offset (quantization parameters) for each band.

[0082] Step 3: Global Reconstruction and Residual Calculation.

[0083] Quantized global principal component scores

[0084] First, perform dequantization to recover the floating-point score. Then, an inverse PCA transformation is performed to obtain the reconstructed data in the noise-normalized space. Calculate the residuals between the original radiance data and the globally reconstructed data under noise normalization. .

[0085] Step 4: Local Principal Component Analysis – The core step of this invention.

[0086] The global reconstruction residual obtained in step three Perform per-granule local PCA processing. For each observation data file, independently construct the residual data matrix and calculate the local mean of the residuals. Local eigenvectors are obtained through eigenvalue decomposition. Extract a small number of local principal components (usually 5) to obtain the local principal component scores. Local feature vectors and local mean (Where K is the number of retained local principal components). The key role of this step is that the global PCA basis is obtained through statistical training on large-scale historical data, retaining only the statistically significant variation directions in the training dataset. For spectral features that deviate from the statistical mean in a single observation file (such as anomalous radiation signals caused by extreme weather events), their representation in the global basis may be insufficient, thus creating structured biases in the global reconstruction residuals. Local PCA directly extracts the main variation patterns from the residuals, essentially performing a supplementary secondary capture of information missed by the global PCA, allowing this information to be retained in the compressed data and recovered during decompression. Since the signal-to-noise ratio of the residuals is much lower than that of the original data, only a very small number of local principal components are needed to effectively express the main structured information in the residuals.

[0087] This combination Figure 3 Explain the effects of local PCA. Figure 3The horizontal axis represents wavenumber, and the vertical axis represents radiance. The title ES_RealLW_diff indicates that this is the reconstruction residual value for the LW longwave band. (Scan, For, and Fov represent the values ​​of the first three dimensions in the four-dimensional array, respectively). The blue curve shows the distribution of the reconstruction residual with channels when only global PCA is used (k=0, i.e., no local PCA is performed), indicating that 0 local principal components are retained. The red curve shows the reconstruction residual when using Hybrid PCA (k=5, i.e., 5 local principal components are added), indicating that 5 local principal components are retained. It can be observed that after adding 5 local principal components, the magnitude of the residual is significantly reduced across the entire LW band. This is because the global PCA basis retains the 40 statistically most significant variation directions in the training set, but for spectral features that deviate from the statistical average in the current specific observation file—such as atmospheric temperature profile anomalies caused by special weather systems in the observation area—the projection of these features onto the 40 principal component directions of the global basis cannot fully express the feature, and the remaining unexpressed part enters the reconstruction residual. Local PCA extracts the five most important mutation directions from the residuals, captures and encodes the structured information that was "missed by global PCA", so that this information can be recovered during decompression and reconstruction, reducing the final reconstruction error.

[0088] Step 5: Data encapsulation and Gzip encoding.

[0089] The quantized global principal component score Local principal component scores Local feature vectors Local mean Quantization parameters ( and The data, along with other necessary metadata (such as data file identifiers, observation geometry information, etc.), is encapsulated in a predetermined format and compressed using the Gzip lossless coding algorithm to generate the final PCS (Principal Component Score) compressed data file. The quantized global principal component score constitutes the main payload of the compressed data, while the local principal component scores and related parameters are appended as residual compensation information. Both are compressed together by Gzip to form the final compressed data stream. Gzip is based on the DEFLATE algorithm (a combination of LZ77 and Huffman coding), which can further remove statistical redundancy in the data stream. In this embodiment, the final PCS file size is approximately 2.5 MB, representing a compression ratio of approximately 2.5% compared to the original real effective emissivity data (approximately 100 MB).

[0090] Step Six: Ground Decompression and Reconstruction Process.

[0091] Ground decompression and reconstruction are performed sequentially according to the following sub-steps:

[0092] (6a) Decode the PCS compressed file using Gzip and restore each data component.

[0093] (6b) Utilization and The parameters are dequantized to restore the global principal component scores to floating-point form. .

[0094] (6c) Perform global PCA reconstruction: Use the dequantized global principal component scores to perform inverse PCA transformation to obtain the noise-normalized global reconstructed radiance data. .

[0095] (6d) Perform local PCA reconstruction: , This yields the reconstructed residual data.

[0096] (6e) Synthesizing the final reconstructed data: First, obtain the reconstructed data under noise normalization. Then, inverse noise normalization is performed to obtain the original data for reconstruction. .

[0097] Step 7: Verify the accuracy of the reconstruction.

[0098] Validation was performed using independent HIRAS observation data (data from May 12, 2026) that was not used in training. Spectra at fixed spatial locations (iscan=5, ifor=5, ifov=0) in the file were selected for comparison between the original and reconstructed data. The reconstructed spectral curves showed high consistency with the original spectral curves across all three bands. Here, isca, ifor, and ifov represent the scan line, the number of ground-based steps, and the number of probes, respectively.

[0099] To quantitatively evaluate the improvement in reconstruction accuracy of Hybrid PCA compared to traditional global PCA, statistics of the reconstruction residuals under both modes were calculated. When using only global PCA (0 local principal components), the reconstruction residual (original radiance minus reconstructed radiance) in the LW band is relatively large, especially near the CO2 absorption band (approximately 667 cm⁻¹), which is consistent with the large dynamic range and strong spectral variability of radiance in this region. Adding 5 local principal components significantly reduces the residual amplitude. In the MW1 and MW2 bands, the overall reconstruction residuals are smaller than those in the LW band, and the addition of local principal components further improves the results.

[0100] In terms of cross-instrument validation, the same Hybrid PCA method (also retaining 5 local principal components) was used to process the LW band data of the IASIMETOP-B instrument. The reconstruction residuals showed a consistent decreasing trend with the HIRAS data, indicating that the effectiveness of the method does not depend on a specific instrument.

[0101] Step 8: Selection of the number of local principal components.

[0102] The selection of the number of local principal components, k, requires a trade-off between reconstruction accuracy and additional data volume. If k is too small (e.g., 1-2), the structured information in the residuals cannot be fully captured; if k is too large (e.g., more than 10), the additional data volume increases, but the accuracy improvement tends to saturate. Experiments show that k between 3 and 10 can achieve significant residual reduction effects, with k=5 achieving a good balance between accuracy improvement and data increment.

[0103] Corresponding to the infrared hyperspectral detection data compression method described in the above embodiments, Figure 4 A structural block diagram of the infrared hyperspectral detection data compression device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0104] Reference Figure 4 As shown, the infrared hyperspectral detection data compression device includes:

[0105] Preprocessing module 41 is used to perform noise normalization preprocessing on infrared hyperspectral detection data;

[0106] The global PCA module 42 is used to perform global principal component analysis on the infrared hyperspectral detection data after noise normalization preprocessing, obtain global principal component scores, and quantify the global principal component scores.

[0107] The residual calculation module 43 is used to perform global reconstruction based on the quantized global principal component scores to obtain global reconstruction data, and to calculate the residual between the original radiance data and the global reconstruction data to obtain the global reconstruction residual.

[0108] Local PCA module 44 is used to perform file-by-file local principal component analysis on the global reconstruction residuals to obtain local principal component scores, local eigenvectors and local means.

[0109] The encapsulation and compression module 45 is used to encapsulate the quantized global principal component score, the local principal component score, the local feature vector, the local mean, and the quantization parameters, and then perform Gzip encoding compression to obtain a compressed data file.

[0110] Optionally, the preprocessing module 41 includes a normalization processing submodule.

[0111] The normalization processing submodule is used to perform noise normalization processing on each spectral channel using a pre-trained global noise normalization coefficient vector; wherein, the radiance value R of each channel is divided by its corresponding noise standard deviation NC to obtain the noise-normalized radiance value.

[0112] Optionally, the global PCA module 42 includes: a centralization submodule, a projection submodule, and a quantization submodule.

[0113] The centralization submodule is used to subtract the global mean spectrum from each spectral vector of the noise-normalized observation data to obtain the centralized data.

[0114] The projection submodule is used to project the centralized data onto a pre-trained global feature vector basis to obtain the global principal component score.

[0115] The quantization submodule is used to perform linear quantization on the global principal component score according to the preset quantization step width and offset, so as to obtain the quantized global principal component score and the quantization parameters.

[0116] Optionally, the global feature vector basis is obtained by pre-training in the following manner:

[0117] Using a large-scale historical observation dataset, the data for each spectral band were preprocessed by noise normalization and then centered.

[0118] The eigenvector matrix is ​​obtained by singular value decomposition, and a portion of the eigenvectors are selected and retained according to the singular values ​​from largest to smallest to form the global eigenvector basis.

[0119] Optionally, the local PCA module 44 includes: a matrix construction submodule, a local mean calculation submodule, and an extraction submodule.

[0120] The matrix construction submodule is used to construct the residual data matrix for each observation data file.

[0121] The local mean calculation submodule is used to calculate the local mean of the residual data matrix.

[0122] The residual data matrix is ​​subjected to eigenvalue decomposition to obtain local eigenvectors.

[0123] The extraction submodule is used to extract a small number of local principal components to obtain the local principal component scores, the local feature vectors, and the local means.

[0124] Optionally, the number of local principal components is 5.

[0125] Optionally, the encapsulation and compression module 45 includes: an encapsulation submodule and a compression submodule.

[0126] The encapsulation submodule is used to encapsulate the quantized global principal component score, the local principal component score, the local feature vector, the local mean, the quantization parameters, the data file identifier, and the observation geometric information in a predetermined format.

[0127] The compression submodule is used to compress the encapsulated data using the Gzip lossless encoding method.

[0128] The apparatus provided in this invention can, firstly, effectively improve the reconstruction accuracy of special observation data that deviates from the statistical average. This invention performs a file-by-file secondary decomposition of the global reconstruction residual using local PCA, extracting structured spectral information not fully expressed by the global basis and incorporating it into the compressed data stream. For observation data containing extreme weather events, special surface conditions, or anomalous atmospheric state characteristics, this information is lost in traditional global PCA devices, but this invention effectively preserves it through a local compensation mechanism, thereby reducing the reconstruction error of such data.

[0129] Secondly, it balances the advantages of global comprehensiveness with local individuality. This invention adopts a two-stage hybrid architecture of "global PCA + local PCA," reflecting the organic combination of statistical commonality and individual characteristics. Global PCA utilizes a global basis trained on large-scale historical data to capture the dominant spectral variation patterns commonly found in observational data, ensuring efficient compression and good reconstruction of most routine observational data, demonstrating the integrity and universality of the device. Local PCA performs personalized processing on the global reconstruction residuals of individual observation files, adaptively compensating for the unique spectral characteristics of those files, demonstrating the ability to finely process local individuals. The two PCA methods are functionally complementary: global PCA is responsible for the efficient expression of commonalities, while local PCA is responsible for the supplementary capture of individual characteristics.

[0130] Furthermore, this invention maintains high compression efficiency. The overall compression ratio of this invention is approximately 2.5% (relative to effective real emissivity data), which is on par with conventional global PCA devices. The introduction of the local PCA stage does not significantly affect the overall compression ratio because the amount of increased local principal component data is much smaller than the amount of global principal component data.

[0131] In short, the embodiments of the present invention can realize a two-stage hybrid compression architecture of global PCA and local PCA, thereby solving the technical problems of insufficient reconstruction accuracy of special observation data, lack of adaptive compensation for individual differences in single files, and waste of effective information in global reconstruction residuals of traditional global PCA devices.

[0132] It should be noted that the information interaction and execution process between the above-mentioned devices / units / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0134] This application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0136] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0137] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 5 As shown, the computer device of this embodiment includes: at least one processor 20 ( Figure 5 (Only one is shown in the diagram), memory 21, and computer program 22 stored in said memory 21 and executable on said at least one processor 20, wherein said processor 20 executes said computer program 22 to implement the steps in any of the above-described visual programming method embodiments.

[0138] The computer device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that... Figure 5The examples of computer devices are merely examples and do not constitute a limitation on computer devices. They may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, they may also include input / output devices, network access devices, etc.

[0139] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0140] In some embodiments, the memory 21 may be an internal storage unit of the computer device, such as a hard drive or memory. In other embodiments, the memory 21 may be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 21 may include both internal and external storage units of the computer device. The memory 21 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / computer equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for compressing infrared hyperspectral detection data, characterized in that, include: Noise normalization preprocessing is performed on the infrared hyperspectral detection data; Global principal component analysis is performed on the infrared hyperspectral detection data after noise normalization preprocessing to obtain global principal component scores, and the global principal component scores are quantified. Global reconstruction is performed based on the quantized global principal component scores to obtain global reconstruction data, and the residual between the original radiance data and the global reconstruction data is calculated to obtain the global reconstruction residual. File-by-file local principal component analysis is performed on the global reconstruction residuals to obtain local principal component scores, local eigenvectors, and local means. The quantized global principal component score, the local principal component score, the local feature vector, the local mean, and the quantization parameters are encapsulated and then compressed using Gzip encoding to obtain a compressed data file.

2. The method according to claim 1, characterized in that, The noise normalization preprocessing of the infrared hyperspectral detection data includes: The pre-trained global noise normalization coefficient vector is used to normalize the noise of each spectral channel. The emissivity value R of each channel is divided by its corresponding noise standard deviation NC to obtain the noise-normalized emissivity value.

3. The method according to claim 1, characterized in that, The process of performing global principal component analysis on the infrared hyperspectral detection data after noise normalization preprocessing to obtain global principal component scores, and quantifying the global principal component scores, includes: The centered data is obtained by subtracting the global mean spectrum from each spectral vector of the noise-normalized observation data. The centralized data is projected onto a pre-trained global feature vector basis to obtain the global principal component score; The global principal component score is linearly quantized according to the preset quantization step width and offset to obtain the quantized global principal component score, and thus the quantization parameters are obtained.

4. The method according to claim 3, characterized in that, The global feature vector basis is obtained through pre-training in the following manner: Using a large-scale historical observation dataset, the data for each spectral band were preprocessed by noise normalization and then centered. The eigenvector matrix is ​​obtained by singular value decomposition, and a portion of the eigenvectors are selected and retained according to the singular values ​​from largest to smallest to form the global eigenvector basis.

5. The method according to claim 1, characterized in that, The step of performing file-by-file local principal component analysis on the global reconstruction residual to obtain local principal component scores, local eigenvectors, and local means includes: For each observation data file, construct a residual data matrix; Calculate the local mean of the residual data matrix; The residual data matrix is ​​subjected to eigenvalue decomposition to obtain local eigenvectors; Extract a small number of local principal components to obtain the local principal component scores, the local feature vectors, and the local means.

6. The method according to claim 5, characterized in that, The number of local principal components is 5.

7. The method according to claim 1, characterized in that, The quantized global principal component score, the local principal component score, the local feature vector, the local mean, and the quantization parameters are encapsulated and then compressed using Gzip encoding to obtain a compressed data file, including: The quantized global principal component score, the local principal component score, the local feature vector, the local mean, the quantization parameter, the data file identifier, and the observation geometric information are encapsulated in a predetermined format. The Gzip lossless encoding method is used to compress the encapsulated data.

8. An infrared hyperspectral detection data compression system, characterized in that, include: The preprocessing module is used to perform noise normalization preprocessing on the infrared hyperspectral detection data; The global PCA module is used to perform global principal component analysis on the infrared hyperspectral detection data after noise normalization preprocessing, obtain global principal component scores, and quantify the global principal component scores. The residual calculation module is used to perform global reconstruction based on the quantized global principal component scores to obtain global reconstruction data, and to calculate the residual between the original radiance data and the global reconstruction data to obtain the global reconstruction residual. The local PCA module is used to perform file-by-file local principal component analysis on the global reconstruction residuals to obtain local principal component scores, local eigenvectors, and local means. The encapsulation and compression module is used to encapsulate the quantized global principal component score, the local principal component score, the local feature vector, the local mean, and the quantization parameters, and then perform Gzip encoding compression to obtain a compressed data file.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 7.

Images