A method, device and equipment for compressing and reconstructing hyperspectral images

Abstract

The present invention discloses a method, device and equipment for compressing and reconstructing a hyperspectral image, which includes: obtaining Matiwan data and performing preprocessing including random cropping to generate image blocks of preset size as original hyperspectral images; inputting channel indexes and spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image; performing compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file; performing data decompression of the compressed file by Huffman inverse coding and data reconstruction by forward propagation to obtain a reconstructed spectral image. It is possible to achieve a high compression ratio while retaining the spectral characteristics of the hyperspectral image, so that the reconstructed hyperspectral image has more practical physical significance, improves the quality of compression reconstruction, and provides data support for subsequent applications.

Description

Technical Field

[0001] The present invention relates to the technical field of image compression, and in particular to a method, device and equipment for compressing and reconstructing a hyperspectral image. Background Art

[0002] Hyperspectral images have been widely used in many key fields (such as disaster monitoring, emergency response, environmental monitoring and traffic management), but their huge amount of data poses a severe challenge to storage and transmission. In the existing technology, hyperspectral image compression technology is mainly divided into traditional methods and deep learning-based methods. Although traditional methods such as 3D-SPECK and 3D-SPIHT can achieve a certain degree of compression, they have the limitations of independent step design, lack of joint optimization, and difficulty in mining high-level semantic information and complex nonlinear relationships. Deep learning-based methods, such as 1D and 3D convolutional autoencoders, improve compression performance through feature extraction and joint optimization, but still face problems such as reliance on a large number of training samples, inability to optimize each compressed reconstructed image individually, high computational complexity, and neglect of mixed pixel spectral information protection. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a method, device and equipment for compressing and reconstructing hyperspectral images, aiming to achieve a high compression ratio while retaining the spectral characteristics of the hyperspectral image, improve the quality of compression and reconstruction, and provide reliable data support for subsequent applications.

[0004] To achieve the above object, the present invention provides a method for compressing and reconstructing a hyperspectral image, the method comprising:

[0005] Obtain Matiwan data and perform preprocessing including random cropping to generate image blocks of preset size as original hyperspectral images;

[0006] Inputting the channel index and the spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image;

[0007] Performing compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file;

[0008] The compressed file is decompressed by inverse Huffman coding and reconstructed by forward propagation to obtain a reconstructed spectral image.

[0009] Preferably, the neural network includes an implicit neural representation module and a spectral unmixing and reconstruction module; the channel index and spatial grid coordinates obtained based on the original hyperspectral image are input into a pre-constructed neural network and overfitting training is performed under the constraint of a preset loss function, including:

[0010] According to the actual spectral sampling of the original hyperspectral image, the channel index is obtained;

[0011] Normalizing the channel index to obtain a normalized channel index;

[0012] Based on the spatial dimension of the original hyperspectral image, generating spatial grid coordinates through the meshgrid function;

[0013] Mapping the spatial grid coordinates and the normalized channel index to a high-dimensional embedding space using regular frequency position coding to obtain first spatial coordinates and first channel indexes, respectively;

[0014] Inputting the first spatial coordinate and the first channel index into the implicit neural representation module to extract image features to obtain a feature map;

[0015] The feature map is input into the spectral unmixing and reconstruction module to perform linear unmixing and reconstruction of the image to obtain a reconstructed spectral image.

[0016] Preferably, the implicit neural representation module includes an MLP_F module, an MLP_G module, a single-head self-attention network, a lightweight multi-head self-attention network and a plurality of CBlock modules; the first spatial coordinate and the first channel index are input into the implicit neural representation module for image feature extraction to obtain a feature map, including:

[0017] Perform channel feature extraction on the first channel index through the MLP_F module to obtain a first feature vector;

[0018] Inputting the first spatial coordinate into the single-head self-attention network to extract spatial context features to obtain spatial features;

[0019] Inputting the first feature vector and the spatial feature into the multi-head self-attention network for fusion to obtain a fused feature map;

[0020] Perform channel feature extraction on the first channel index through the MLP_G module to obtain a second feature vector;

[0021] The second feature vector and the fused feature map are input into the CBlock module for upsampling operation to obtain the feature map.

[0022] Preferably, the CBlock module includes , convolutional layers, and - Upsampling block.

[0023] Preferably, the spectral unmixing and reconstruction module includes a first convolutional subnetwork and a second convolutional subnetwork; the step of inputting the feature map into the spectral unmixing and reconstruction module to perform linear unmixing and reconstruction of the image to obtain a reconstructed spectral image includes:

[0024] Inputting the feature map into the first convolutional subnetwork to obtain an abundance map;

[0025] The abundance map is input into the second convolutional subnetwork for image reconstruction to obtain the reconstructed spectral image; wherein the parameters of the convolutional layer in the second convolutional subnetwork are used as end members; the calculation of the end members includes obtaining the end members constituting the mixed pixels of the hyperspectral image by using the vertex component analysis method, and using the end members to initialize the parameters of the convolutional layer in the second convolutional subnetwork, and updating them by back propagation of the gradient.

[0026] Preferably, the preset loss function includes:

[0027] ;in,

[0028] ,

[0029] ,

[0030] In the formula, represents the reconstruction optimization loss, represents the optimization loss when the abundance sum is 1, B represents the total number of channels of the original hyperspectral image output by the neural network, represents the reconstructed spectral image of the b-th channel index of the decompressed reconstruction, represents the original hyperspectral image corresponding to the b-th channel index, SSIM represents the structural similarity loss, represents the weight of the loss term, represents a column vector whose elements are all 1, with HW elements, and E represents the abundance map output by the first convolutional subnetwork. represents the number of channels of the abundance map, represents matrix multiplication, Represents a column vector whose elements are all 1 and the number of elements is k.

[0031] Preferably, the compression processing including weight quantization and Huffman coding of the network weights in the first model to obtain a compressed file includes:

[0032] The first model is converted into a format supported by ONNX through the export function provided by the ONNX framework, and the network weights of the first model are quantized through the convert_float_to_float16 function to obtain a quantized second model;

[0033] The network weights of the second model are losslessly compressed by Huffman coding, and the encoded network weights and network structure are stored in a physical file by a serialization method to obtain the compressed file.

[0034] To achieve the above object, the present invention also provides a hyperspectral image compression and reconstruction device, the device comprising:

[0035] A preprocessing unit, used to obtain Matiwan data and perform preprocessing including random cropping to generate image blocks of a preset size as original hyperspectral images;

[0036] An overfitting training unit, used for inputting the channel index and the spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image;

[0037] A compression processing unit, used for performing compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file;

[0038] The data reconstruction unit is used to decompress the compressed file by performing Huffman inverse coding and to reconstruct the data by forward propagation to obtain a reconstructed spectral image.

[0039] In order to achieve the above objectives, the present invention also proposes a hyperspectral image compression and reconstruction device, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a hyperspectral image compression and reconstruction method as described in the above embodiment.

[0040] In order to achieve the above object, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of a hyperspectral image compression and reconstruction method as described in the above embodiment.

[0041] In order to achieve the above object, the present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a hyperspectral image compression and reconstruction method as described in the above embodiment.

[0042] Beneficial effects:

[0043] The above scheme can achieve a high compression ratio while retaining the spectral characteristics of the hyperspectral image, so that the reconstructed hyperspectral image has more practical physical meaning, improves the quality of compression reconstruction, and provides reliable data support for subsequent applications. By using neural networks to overfit the hyperspectral image, it can more accurately capture the spectral and spatial characteristics of HSI data, providing a basis for subsequent data reconstruction; by compressing the model through weight quantization and Huffman coding, the storage space of the model is significantly reduced and the compression ratio is improved; and in the data reconstruction process, Huffman inverse coding and forward propagation can achieve higher reconstruction efficiency, and can retain the important features of the original HSI data, improving the quality of the reconstructed spectral image.

[0044] The above scheme provides a basis for subsequent frequency position encoding and high-dimensional embedding by normalizing channel indexes and generating spatial grid coordinates, thereby enhancing the model's ability to capture spectral and spatial information. A continuous mapping relationship from channel index to corresponding spectrum map is established through a neural network to maintain the spectral continuity of HSI. Considering the widespread existence of mixed pixels in hyperspectral images, a spectral unmixing and reconstruction module is introduced on the basis of the implicit neural representation module to effectively identify and retain the spectral mixing characteristics of mixed pixels. In addition, a non-uniform spectral sampling position encoding is designed to reflect the true band distribution of hyperspectral images. This case simplifies the compression task of hyperspectral images and retains the original physical properties of the image while compressing it, which is of great significance for improving the decompression and reconstruction quality of the image and promoting the widespread application of HSI.

[0045] In the above scheme, the network structure of the implicit neural representation module includes MLP_F, MLP_G, single-head self-attention network, multi-head self-attention network, and CBlock module. These structures work together to extract image features, improving the accuracy and efficiency of feature extraction. Among them, the spatial context feature extraction through the single-head self-attention network enhances the model's ability to capture spatial information; the use of multi-head self-attention network and CBlock module further improves the effect of feature fusion and upsampling, thereby improving the quality of reconstructed spectral images. The CBlock module contains convolutional layers and upsampling blocks. These structures enable the feature map to retain more detailed information during the upsampling process, thereby improving the resolution and clarity of the reconstructed spectral image.

[0046] In the above scheme, the spectral unmixing and reconstruction module includes the first convolutional subnetwork and the second convolutional subnetwork, which work together in the linear unmixing and reconstruction process of the image, improving the accuracy and efficiency of the reconstructed spectral image. The end members are initialized by the vertex component analysis method and updated by the back propagation of the gradient, making the calculation of the end members more accurate, thereby improving the quality of the reconstructed spectral image.

[0047] In the above scheme, the preset loss function includes the reconstruction optimization term loss and the abundance sum optimization term loss. These two loss terms work together in the training process of the neural network, making the reconstructed spectral image more accurate while ensuring the physical properties of the reconstructed spectral image.

[0048] The above scheme, the compression process of this case makes the model more convenient and efficient in practical application, while ensuring the high compression ratio and reconstruction quality of the data. The model is converted into a supported format through the ONNX framework, and the storage and computing costs of the model are reduced through quantization processing; the lossless compression of Huffman coding further reduces the storage space of the model and improves the compression ratio. The serialization method stores the encoded network weights in a physical file, which is convenient for the storage and transmission of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic flow chart of a method for compressing and reconstructing a hyperspectral image provided by an embodiment of the present invention.

[0051] Figure 2 An overall schematic diagram of the compression and reconstruction of a hyperspectral image provided by an embodiment of the present invention.

[0052] Figure 3 A schematic diagram of a neural network overfitting a hyperspectral image according to an embodiment of the present invention.

[0053] Figure 4 A network diagram of an implicit neural representation module provided by an embodiment of the present invention.

[0054] Figure 5 A network diagram of a spectral unmixing and reconstruction module provided in one embodiment of the present invention.

[0055] Figure 6 A schematic structural diagram of a hyperspectral image compression and reconstruction device provided by an embodiment of the present invention.

[0056] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0059] The content of the present invention is described in detail below with reference to embodiments.

[0060] Reference Figure 1 FIG. 1 is a flow chart of a method for compressing and reconstructing a hyperspectral image provided by an embodiment of the present invention.

[0061] In this embodiment, the method includes:

[0062] S11, acquiring Matiwan data and performing preprocessing including random cropping to generate image blocks of a preset size as original hyperspectral images.

[0063] S12, inputting the channel index and the spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image.

[0064] S13, performing compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file.

[0065] S14, decompressing the compressed file by performing Huffman inverse coding and reconstructing the data by forward propagation to obtain a reconstructed spectral image.

[0066] See also Figure 2The compression and decompression process diagram of the hyperspectral image compression method shown in the figure includes overfitting training and neural network compression; neural network compression includes weight quantization and Huffman coding; the decompression and reconstruction process includes inverse coding and forward propagation. Before compression, the Matiwan data needs to be preprocessed. Represents the original hyperspectral image. Specifically, before training, the acquired Matiwan data needs to be randomly cropped to generate image blocks of size 512x512 , and as the original hyperspectral image to be compressed (original HSI), after being stored in uint16, the physical storage space occupied by the image block is 128MB. Among them, 256 represents the number of channels, and 512 represents the length H and width W.

[0067] The neural network includes an implicit neural representation module and a spectral unmixing and reconstruction module. Further, in step S12, the channel index and spatial grid coordinates obtained based on the original hyperspectral image are input into a pre-constructed neural network and overfitting training is performed under the constraint of a preset loss function, including:

[0068] S12-0, converting according to the actual spectral sampling of the original hyperspectral image to obtain a channel index;

[0069] S12-1, normalizing the channel index to obtain a normalized channel index;

[0070] S12-2, generating spatial grid coordinates through a meshgrid function based on the spatial dimension of the original hyperspectral image;

[0071] S12-3, mapping the spatial grid coordinates and the normalized channel index to a high-dimensional embedding space using regular frequency position coding to obtain first spatial coordinates and first channel index respectively;

[0072] S12-4, inputting the first spatial coordinate and the first channel index into the implicit neural representation module to extract image features to obtain a feature map;

[0073] S12-5, inputting the feature map into the spectral unmixing and reconstruction module to perform linear unmixing and reconstruction of the image to obtain a reconstructed spectral image.

[0074] The implicit neural representation module includes an MLP_F module, an MLP_G module, a single-head self-attention network, a lightweight multi-head self-attention network, and multiple CBlock modules. The CBlock module includes , convolutional layers, and - Upsampling block.

[0075] Furthermore, in step S12-4, the first spatial coordinate and the first channel index are input into the implicit neural representation module to extract image features to obtain a feature map, including:

[0076] S12-4-1, extracting channel features from the first channel index through the MLP_F module to obtain a first feature vector;

[0077] S12-4-2, inputting the first spatial coordinate into the single-head self-attention network to extract spatial context features to obtain spatial features;

[0078] S12-4-3, inputting the first feature vector and the spatial feature into the multi-head self-attention network for fusion to obtain a fused feature map;

[0079] S12-4-4, extracting channel features from the first channel index through the MLP_G module to obtain a second feature vector;

[0080] S12-4-5, input the second feature vector and the fused feature map into the CBlock module for upsampling operation to obtain the feature map.

[0081] The spectral unmixing and reconstruction module includes a first convolutional subnetwork and a second convolutional subnetwork. Further, in step S12-5, the feature map is input into the spectral unmixing and reconstruction module for linear unmixing and reconstruction of the image to obtain a reconstructed spectral image, including:

[0082] S12-5-1, inputting the feature map into the first convolutional subnetwork to obtain an abundance map;

[0083] S12-5-2, input the abundance map into the second convolutional subnetwork for image reconstruction to obtain the reconstructed spectral image; wherein, the parameters of the convolutional layer in the second convolutional subnetwork are used as end members; the calculation of the end members includes obtaining the end members that constitute the mixed pixels of the hyperspectral image by using the vertex component analysis method, and using the end members to initialize the parameters of the convolutional layer in the second convolutional subnetwork, and updating them through gradient back propagation.

[0084] See also Figure 3 As shown, it uses a neural network Overfitting process, in which represents the input original hyperspectral image, represents the reconstructed spectral image, express The number of channels, H and W represent The length and width of Represents a specific channel index of a hyperspectral image. Indicates the channel index The corresponding hyperspectral image, is the corresponding reconstructed spectral image. Specifically including:

[0085] (1) Non-uniform spectral sampling position encoding: Since the sampling of hyperspectral images in the spectral dimension is usually non-uniform, the channel index It needs to be converted according to the actual spectrum sampling. Set the channel index The corresponding wavelength center is , the actual wavelength range covered by the original hyperspectral image is , then the normalized channel index for: Through neural network A continuous mapping relationship from channel index to corresponding spectral image is established to fit the hyperspectral image to maintain the spectral continuity of the hyperspectral image. On this basis, in order to adapt to the real band distribution of the hyperspectral image, a non-uniform spectral sampling position encoding is further designed.

[0086] (2) Establishing spatial grid coordinates: Based on the spatial dimensions of the hyperspectral image (i.e., the length and width of the image), The framework provides Function generates spatial grid coordinates ,in 8.

[0087] (3) In order to better fit the high-frequency information, regular frequency position encoding is adopted The spatial grid coordinates and the normalized channel index Mapped to high-dimensional embedding space, we get (first space coordinate) and (First channel index). That is,

[0088] ;

[0089] ;

[0090] ;

[0091] in, The data that need to be embedded are and , and are hyperparameters, representing mapping frequency and mapping length respectively. In this embodiment, Setting 1.25, Set to 80.

[0092] (4) The encoded spatial grid coordinates and channel index Input to the implicit neural representation module to obtain the feature map. The implicit neural representation module uses E-NeRV as the backbone network and is responsible for outputting the feature map , and generate the reconstructed spectral image through the subsequent spectral unmixing and reconstruction module See also Figure 4 As shown, the implicit neural representation module specifically includes the following contents:

[0093] Extraction and fusion of spatial features and channel features: Index the first channel go through The module performs feature extraction and obtains The feature vector of ; The first space coordinate After single-head self-attention network Perform spatial context feature extraction and output features ;Will and pass Fusion is performed to obtain a fused feature map containing both spatial and channel information ,in .Right now, , Represents C and Perform element-by-element multiplication at corresponding positions, Represents a lightweight multi-head self-attention network. Index the first channel The second feature vector is obtained by feature extraction through the MLP_G module. ; The second eigenvector And the fusion feature map Input CBlock module for upsampling operation to obtain feature map .

[0094] like Figure 4 As shown, the implicit neural representation module consists of 5 Module, each Include , convolution, and - The upsampling blocks have upsampling coefficients of 4, 2, 2, 2, and 2. , Finally, it is upsampled to . As shown in the following formula: .

[0095] (5) Feature map Input to the spectrum unmixing and reconstruction module to obtain the reconstructed spectrum image. Figure 5 As shown, the spectral unmixing and reconstruction module includes G1 and G2, where: and Each consists of a lightweight convolutional sub-network. Output abundance , The parameters of the convolutional layer in Input to Afterwards, by Reconstruct and obtain the reconstructed spectral image To calculate the end members, The network weights of the convolutional layers need to be initialized using the endmembers obtained from the vertex component analysis method (VCA) and updated through the back propagation of gradients throughout the learning process. Provided Functions and Optimizations To enhance abundance Two physical property constraints: the abundance is non-negative and the sum of the abundances is unity.

[0096] Taking into account the common problem of mixed pixels in hyperspectral images, this embodiment adds a spectral unmixing and reconstruction module and physical constraints of spectral unmixing on the basis of the implicit neural representation module to protect the spectral mixing characteristics of the hyperspectral image, thereby making the reconstructed hyperspectral image more practical physically meaningful and improving the decompression and reconstruction quality.

[0097] In this embodiment, the optimization term loss is reconstructed And the abundance sum is 1 optimization term loss The combination of is used as the optimization loss function and is used to calculate the reconstruction error. That is,

[0098] ,in,

[0099] ,

[0100] ,

[0101] in, represents the reconstruction optimization loss, represents the optimization loss when the abundance sum is 1, is the total number of channels of the hyperspectral image output by the neural network, Represents the decompression and reconstruction of The reconstructed spectral image of bands (index of b-th channel), Indicates the channel index The corresponding original hyperspectral image, SSIM represents the structural similarity loss, represents the weight of the loss term, represents a column vector whose elements are all 1, with HW elements, and E represents the abundance map output by the first convolutional subnetwork. represents the number of channels of the abundance map, represents matrix multiplication, Represents a column vector whose elements are all 1 and the number of elements is k.

[0102] Further, in step S13, the network weights of the first model are compressed including weight quantization and Huffman coding to obtain a compressed file, including:

[0103] S13-1, converting the first model into a format supported by ONNX through the export function provided by the ONNX framework, and quantizing the network weights of the first model through the convert_float_to_float16 function to obtain a quantized second model;

[0104] S13-2, losslessly compress the network weights of the second model through Huffman coding, store the encoded network weights and network structure in a physical file through a serialization method, and obtain the compressed file.

[0105] In this embodiment, since network weights are used to represent compressed data, limiting the storage size of network weights is conducive to improving the compression ratio. Neural network compression includes two parts, weight quantization and Huffman coding. After the neural network is trained, weight quantization is applied to all network weights. The specific process is as follows: First, the neural network is exported through the export function provided by the ONNX framework. Convert to a format supported by ONNX; then, convert_float_to_float16 function to convert the network weights Perform quantization and export the quantized neural network ; Then, Huffman coding is used to encode the quantized Lossless compression is performed to further improve the compression ratio; finally, the encoded network weights are stored in a physical file through a serialization method to form the final compressed file The present invention aims to use implicit neural network To overfit the hyperspectral image, use the network weights Represents compressed data. After the network training is completed, the network weights are quantized and Huffman coded to further improve the compression ratio. A model corresponds to a specific hyperspectral image; when decompressing, only the channel subscript and spatial grid need to be input into the model, and the model outputs a channel image corresponding to the hyperspectral image.

[0106] Finally, the compressed file is decompressed and reconstructed to obtain the reconstructed spectral image. Specifically: first load the compressed file M into the memory, then decompress the compressed file through Huffman inverse coding to obtain the quantized neural network ; Input the spatial grid coordinates S and all channel indices b to In the process, reconstruction is performed through forward propagation to obtain the reconstructed spectral image corresponding to each channel index. , and finally stitched together in sequence to form a complete hyperspectral image ,Right now .

[0107] The hyperspectral image (HSI) compression of the present invention is mainly realized by neural overfitting. The process of network overfitting HSI is the compression process, and the network weight represents the compressed data. express, Represents the network weight; It includes an implicit neural representation module and a spectral unmixing and reconstruction module. The implicit neural representation module uses E-NeRV as the backbone network, which is responsible for extracting image features and inputting them into the spectral unmixing and reconstruction module; the spectral unmixing and reconstruction module is based on the linear unmixing framework of the autoencoder and is responsible for linear unmixing and reconstruction of the image. Specifically, through A continuous mapping relationship from channel index to corresponding spectrum is established to maintain the spectral continuity of HSI. Considering the common phenomenon of mixed pixels in hyperspectral images, the present invention introduces a spectral unmixing and reconstruction module based on the implicit neural representation module to effectively identify and retain the spectral mixing characteristics of mixed pixels. In addition, in order to reflect the real band distribution of hyperspectral images, a non-uniform spectral sampling position encoding is designed. Finally, since this method uses weights As compressed data, after the network training is completed, the method continues to Quantization and Huffman coding are performed to further improve the compression ratio. The present invention simplifies the compression task of hyperspectral images and retains the original physical properties of the images while compressing them, which is of great significance for improving the decompression and reconstruction quality of images and promoting the widespread application of HSI.

[0108] Reference Figure 6 FIG. 4 is a schematic diagram of the structure of a hyperspectral image compression and reconstruction device provided by an embodiment of the present invention.

[0109] In this embodiment, the device 20 includes:

[0110] A preprocessing unit 21 is used to obtain Matiwan data and perform preprocessing including random cropping to generate image blocks of a preset size as original hyperspectral images;

[0111] An overfitting training unit 22, used for inputting the channel index and the spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image;

[0112] A compression processing unit 23, configured to perform compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file;

[0113] The data reconstruction unit 24 is used to decompress the compressed file by performing Huffman inverse coding and reconstruct the data by forward propagation to obtain a reconstructed spectral image.

[0114] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0115] The embodiment of the present invention further provides a hyperspectral image compression and reconstruction device, the device comprising the hyperspectral image compression and reconstruction device as described above, wherein the hyperspectral image compression and reconstruction device can be used Figure 6 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0116] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0117] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0118] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0119] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0120] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The compression and reconstruction method of the hyperspectral image shown in the figure. The computer readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0121] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A compression and reconstruction method for a hyperspectral image is shown.

[0122] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0123] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0124] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for compressing and reconstructing a hyperspectral image, characterized in that: The method comprises: Obtain Matiwan data and perform preprocessing including random cropping to generate image blocks of preset size as original hyperspectral images; Inputting the channel index and the spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image; Performing compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file; Decompressing the compressed file by performing Huffman inverse coding and reconstructing the data by forward propagation to obtain a reconstructed spectral image; Furthermore, the neural network includes an implicit neural representation module and a spectral unmixing and reconstruction module; the channel index and spatial grid coordinates obtained based on the original hyperspectral image are input into a pre-constructed neural network and overfitting training is performed under the constraint of a preset loss function, including: According to the actual spectral sampling of the original hyperspectral image, the channel index is obtained; Normalizing the channel index to obtain a normalized channel index; Based on the spatial dimension of the original hyperspectral image, generating spatial grid coordinates through the meshgrid function; Mapping the spatial grid coordinates and the normalized channel index to a high-dimensional embedding space using regular frequency position coding to obtain first spatial coordinates and first channel indexes, respectively; Inputting the first spatial coordinate and the first channel index into the implicit neural representation module to extract image features to obtain a feature map; Inputting the feature map into the spectral unmixing and reconstruction module to perform linear unmixing and reconstruction of the image to obtain a reconstructed spectral image; The spectral unmixing and reconstruction module includes a first convolution subnetwork and a second convolution subnetwork; the feature map is input into the spectral unmixing and reconstruction module for linear unmixing and reconstruction of the image to obtain a reconstructed spectral image, including: Inputting the feature map into the first convolutional subnetwork to obtain an abundance map; The abundance map is input into the second convolutional subnetwork for image reconstruction to obtain the reconstructed spectral image; wherein the parameters of the convolutional layer in the second convolutional subnetwork are used as end members; the calculation of the end members includes obtaining the end members constituting the mixed pixels of the hyperspectral image by using the vertex component analysis method, and using the end members to initialize the parameters of the convolutional layer in the second convolutional subnetwork, and updating them by back propagation of the gradient.

2. The method for compressing and reconstructing a hyperspectral image according to claim 1, characterized in that: The implicit neural representation module includes an MLP_F module, an MLP_G module, a single-head self-attention network, a lightweight multi-head self-attention network and a plurality of CBlock modules; the first spatial coordinate and the first channel index are input into the implicit neural representation module to extract image features to obtain a feature map, including: Perform channel feature extraction on the first channel index through the MLP_F module to obtain a first feature vector; Inputting the first spatial coordinate into the single-head self-attention network to extract spatial context features to obtain spatial features; Inputting the first feature vector and the spatial feature into the multi-head self-attention network for fusion to obtain a fused feature map; Perform channel feature extraction on the first channel index through the MLP_G module to obtain a second feature vector; The second feature vector and the fused feature map are input into the CBlock module for upsampling operation to obtain the feature map.

3. The method for compressing and reconstructing a hyperspectral image according to claim 2, characterized in that: The CBlock module includes , convolutional layers, and - Upsampling block.

4. The method for compressing and reconstructing a hyperspectral image according to claim 1, characterized in that: The preset loss function includes: ;in, ， ， In the formula, represents the reconstruction optimization term loss, represents the optimization loss when the abundance sum is 1, B represents the total number of channels of the original hyperspectral image output by the neural network, represents the reconstructed spectral image of the b-th channel index of the decompressed reconstruction, represents the original hyperspectral image corresponding to the b-th channel index, SSIM represents the structural similarity loss, represents the weight of the loss term, represents a column vector whose elements are all 1, with HW elements, and E represents the abundance map output by the first convolutional subnetwork. represents the number of channels of the abundance map, represents matrix multiplication, Represents a column vector whose elements are all 1 and the number of elements is k.

5. The method for compressing and reconstructing a hyperspectral image according to claim 1, characterized in that: The compression process including weight quantization and Huffman coding is performed on the network weights of the first model to obtain a compressed file, including: The first model is converted into a format supported by ONNX through the export function provided by the ONNX framework, and the network weights of the first model are quantized through the convert_float_to_float16 function to obtain a quantized second model; The network weights of the second model are losslessly compressed by Huffman coding, and the encoded network weights and network structure are stored in a physical file by a serialization method to obtain the compressed file.

6. A hyperspectral image compression and reconstruction device, characterized in that: The device comprises: A preprocessing unit, used to obtain Matiwan data and perform preprocessing including random cropping to generate image blocks of a preset size as original hyperspectral images; An overfitting training unit, used for inputting the channel index and the spatial grid coordinates obtained based on the original hyperspectral image into a pre-constructed neural network and performing overfitting training under the constraint of a preset loss function to obtain a first model for representing the original hyperspectral image; A compression processing unit, used for performing compression processing including weight quantization and Huffman coding on the network weights of the first model to obtain a compressed file; A data reconstruction unit, used for decompressing the compressed file by performing Huffman inverse coding and reconstructing the data by forward propagation to obtain a reconstructed spectral image; Furthermore, the neural network includes an implicit neural representation module and a spectral unmixing and reconstruction module; the channel index and spatial grid coordinates obtained based on the original hyperspectral image are input into a pre-constructed neural network and overfitting training is performed under the constraint of a preset loss function, including: According to the actual spectral sampling of the original hyperspectral image, the channel index is obtained; Normalizing the channel index to obtain a normalized channel index; Based on the spatial dimension of the original hyperspectral image, generating spatial grid coordinates through a meshgrid function; Mapping the spatial grid coordinates and the normalized channel index to a high-dimensional embedding space using regular frequency position coding to obtain first spatial coordinates and first channel indexes, respectively; Inputting the first spatial coordinate and the first channel index into the implicit neural representation module to extract image features to obtain a feature map; Inputting the feature map into the spectral unmixing and reconstruction module to perform linear unmixing and reconstruction of the image to obtain a reconstructed spectral image; The spectral unmixing and reconstruction module includes a first convolution subnetwork and a second convolution subnetwork; the feature map is input into the spectral unmixing and reconstruction module for linear unmixing and reconstruction of the image to obtain a reconstructed spectral image, including: Inputting the feature map into the first convolutional subnetwork to obtain an abundance map; The abundance map is input into the second convolutional subnetwork for image reconstruction to obtain the reconstructed spectral image; wherein the parameters of the convolutional layer in the second convolutional subnetwork are used as end members; the calculation of the end members includes obtaining the end members constituting the mixed pixels of the hyperspectral image by using the vertex component analysis method, and using the end members to initialize the parameters of the convolutional layer in the second convolutional subnetwork, and updating them by back propagation of the gradient.

7. A hyperspectral image compression and reconstruction device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the hyperspectral image compression and reconstruction method according to any one of claims 1 to 5.

8. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of the hyperspectral image compression and reconstruction method according to any one of claims 1 to 5.

Images