Method and device for super-resolution reconstruction of hyperspectral image of object

Abstract

The embodiment of the invention relates to the field of image data processing, in particular to a method and device for super-resolution reconstruction of a hyperspectral image of an object. According to the method and the device provided by the embodiment of the invention, the sampling results of different resolutions of the high-resolution hyperspectral image and the high-resolution multispectral image are acquired, and processing is carried out according to the corresponding channel dimensions so as to acquire the difference image and the plurality of fusion images; correlation and multi-scale characteristics between a multispectral image and a hyperspectral image can be fully considered; and feature extraction and enhancement processing are performed on the fused image, so that local and global feature extraction precision can be improved, and high-frequency features such as image details and edges can be enhanced. Compared with a traditional fusion reconstruction method and an existing super-resolution reconstruction method introducing a deep neural network, the cross-modal information interaction capability is improved, the spectral fidelity and the spatial detail retention degree are good, and a high-quality high-resolution hyperspectral image is obtained.

Description

Technical Field

[0001] The embodiments of this application relate to the field of image data processing, and in particular to a method and apparatus for super-resolution reconstruction of hyperspectral images of an object. Background Technology

[0002] The statements herein are provided merely as background information in connection with this application and do not necessarily constitute prior art.

[0003] Hyperspectral images contain rich spectral information, enabling them to characterize the properties of matter. However, due to limitations in imaging mechanisms and optical devices, hyperspectral sensors often have to sacrifice some spatial resolution in exchange for higher spectral resolution. This results in hyperspectral images failing to capture fine spatial details, limiting their application in scenarios requiring high-precision spatial information. Summary of the Invention

[0004] A brief overview of this application is provided below to offer a basic understanding of certain aspects thereof. It should be understood that this overview is not an exhaustive summary of the application. It is not intended to identify key or essential parts of the application, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0005] This application provides a method for super-resolution reconstruction of a hyperspectral image of an object, comprising the following steps: S10: acquiring a low-resolution hyperspectral image and a high-resolution multispectral image of the object; S20: upsampling the low-resolution hyperspectral image by a factor of 2 to obtain an upsampled hyperspectral image. The low-resolution hyperspectral image is downsampled by a factor of 2 to obtain the downsampled hyperspectral image. The high-resolution multispectral image is downsampled by a factor of 2 to obtain the downsampled multispectral image. The high-resolution multispectral image was downsampled by 4 times to obtain the downsampled multispectral image. S30: Downsample the hyperspectral image With downsampled multispectral images Connect the channels to obtain the fused image F1; combine the low-resolution hyperspectral image with the downsampled multispectral image. Connect the channels to obtain the fused image F2; then upsample the hyperspectral image. With high-resolution multispectral images Connect the channels to obtain the fused image F3; then upsample the hyperspectral image. With high-resolution multispectral images Perform pixel-level subtraction to obtain the difference image F4; S40: Extract features from the fused images F1, F2, F3, and F4 respectively to obtain multiple shallow features; S50: Enhance the multiple shallow features to obtain multiple enhanced features; S60: Extract features from the multiple enhanced features to obtain multiple deep features; S70: Fuse the multiple shallow features and multiple deep features respectively to obtain the fused features; S80: Based on the upsampled hyperspectral image... By combining features, high-resolution hyperspectral images are obtained.

[0006] Another aspect of this application provides an apparatus for super-resolution reconstruction of a hyperspectral image of an object, comprising: an image acquisition module configured to acquire a low-resolution hyperspectral image and a high-resolution multispectral image of the object; and an image preprocessing module configured to upsample the low-resolution hyperspectral image by a factor of 2 to obtain an upsampled hyperspectral image. The low-resolution hyperspectral image is downsampled by a factor of 2 to obtain the downsampled hyperspectral image. The high-resolution multispectral image is downsampled by a factor of 2 to obtain the downsampled multispectral image. The high-resolution multispectral image was downsampled by 4 times to obtain the downsampled multispectral image. The image fusion module is configured to: convert downsampled hyperspectral images... With downsampled multispectral images Connect the channels to obtain the fused image F1; combine the low-resolution hyperspectral image with the downsampled multispectral image. Connect the channels to obtain the fused image F2; then upsample the hyperspectral image. The high-resolution multispectral image is concatenated along the channel dimension to obtain the fused image F3; the upsampled hyperspectral image is then... The system performs pixel-level subtraction with the high-resolution multispectral image to obtain the difference image F4; a shallow feature extraction module is configured to extract features from the fused images F1, F2, F3, and the difference image F4 respectively to obtain multiple shallow features; a feature enhancement module is configured to enhance the multiple shallow features separately to obtain multiple enhanced features; a deep feature extraction module is configured to extract features from the multiple enhanced features separately to obtain multiple deep features; a feature fusion module is configured to fuse the multiple shallow features and multiple deep features separately to obtain a fused feature; and an image reconstruction module is configured to reconstruct the image based on the upsampled hyperspectral image. By combining features, high-resolution hyperspectral images are obtained.

[0007] The method for super-resolution reconstruction of hyperspectral images of an object provided in this application acquires sampling results of different resolutions from high-resolution hyperspectral images and high-resolution multispectral images, and processes them according to the corresponding channel dimensions to obtain difference images and multiple fused images. This method can fully consider the correlation and multi-scale characteristics between multispectral and hyperspectral images. Feature extraction and enhancement processing of the fused images can improve the accuracy of local and global feature extraction and enhance high-frequency features such as image details and edges. Compared with traditional fusion reconstruction methods and existing super-resolution reconstruction methods that incorporate deep neural networks, this method improves cross-modal information interaction capabilities, possesses better spectral fidelity and spatial detail preservation, and obtains high-quality high-resolution hyperspectral images.

[0008] The apparatus for super-resolution reconstruction of hyperspectral images of an object provided in this application obtains sampling results of different resolutions from high-resolution hyperspectral images and high-resolution multispectral images through an image preprocessing module, and processes the images according to the corresponding channel dimensions to obtain difference images and multiple fused images, which can fully consider the correlation and multi-scale characteristics between multispectral and hyperspectral images. Through a feature enhancement module and a deep feature extraction module, features are extracted and enhanced from the fused images, improving the accuracy of local and global feature extraction and enhancing high-frequency features such as image details and edges. Compared with traditional fusion reconstruction methods and existing super-resolution reconstruction methods that incorporate deep neural networks, this apparatus improves cross-modal information interaction capabilities, has better spectral fidelity and spatial detail preservation, and obtains high-quality high-resolution hyperspectral images. Attached Figure Description

[0009] To further illustrate the above and other advantages and features of this application, the specific embodiments of this application will be described in more detail below with reference to the accompanying drawings. The drawings, together with the following detailed description, are included in and form a part of this specification. Elements having the same function and structure are indicated by the same reference numerals. It should be understood that these drawings only depict typical examples of this application and should not be considered as limiting the scope of this application.

[0010] Figure 1 It is a ground truth diagram of a test image; Figure 2 The method provided in the embodiments of this application is used to... Figure 1 A schematic diagram of the reconstructed structure; Figure 3 yes Figure 2 A schematic diagram of the error in the reconstruction results. Detailed Implementation

[0011] Exemplary embodiments of this application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual implementations are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer's specific goals, such as complying with constraints related to the system and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from the content of this application.

[0012] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0013] The following disclosure provides several different implementations or examples for carrying out this application. To simplify the disclosure of this application, specific examples of components and methods are described below. Of course, these are merely examples and are not intended to limit this application. In the description of the embodiments of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0014] Currently, existing methods often neglect the correlation and multi-scale characteristics between input images when processing hyperspectral and multispectral images, resulting in problems such as weak local feature extraction capabilities, insufficient utilization of high-frequency image features, and insufficient cross-modal information interaction capabilities.

[0015] To address the aforementioned problems, embodiments of this application provide a method for super-resolution reconstruction of a hyperspectral image of an object, comprising the following steps: S10: acquiring a low-resolution hyperspectral image and a high-resolution multispectral image of the object; S20: upsampling the low-resolution hyperspectral image by a factor of 2 to obtain an upsampled hyperspectral image. The low-resolution hyperspectral image is downsampled by a factor of 2 to obtain the downsampled hyperspectral image. The high-resolution multispectral image is downsampled by a factor of 2 to obtain the downsampled multispectral image. The high-resolution multispectral image was downsampled by 4 times to obtain the downsampled multispectral image. S30: Downsample the hyperspectral image With downsampled multispectral images Connect the channels to obtain the fused image F1; combine the low-resolution hyperspectral image with the downsampled multispectral image. Connect the channels to obtain the fused image F2; then upsample the hyperspectral image. The high-resolution multispectral image is concatenated along the channel dimension to obtain the fused image F3; the upsampled hyperspectral image is then... S40: Perform pixel-level subtraction with the high-resolution multispectral image to obtain the difference image F4; S50: Extract features from the fused images F1, F2, F3, and F4 respectively to obtain multiple shallow features; S60: Enhance the multiple shallow features to obtain multiple enhanced features; S70: Extract features from the multiple enhanced features to obtain multiple deep features; S80: Fuse the multiple shallow features and multiple deep features respectively to obtain the fused feature; S90: Based on the upsampled hyperspectral image... By combining features, high-resolution hyperspectral images are obtained.

[0016] The method for super-resolution reconstruction of hyperspectral images of an object provided in this application acquires sampling results of different resolutions from high-resolution hyperspectral images and high-resolution multispectral images, and processes them according to the corresponding channel dimensions to obtain difference images and multiple fused images. This method can fully consider the correlation and multi-scale characteristics between multispectral and hyperspectral images. Feature extraction and enhancement processing of the fused images can improve the accuracy of local and global feature extraction and enhance high-frequency features such as image details and edges. Compared with traditional fusion reconstruction methods and existing super-resolution reconstruction methods that incorporate deep neural networks, this method improves cross-modal information interaction capabilities, possesses better spectral fidelity and spatial detail preservation, and obtains high-quality high-resolution hyperspectral images.

[0017] Figure 1 It is a ground truth diagram of a test image. Figure 2 The method provided in the embodiments of this application is used to... Figure 1 The reconstructed diagram is as follows: Figure 1 and Figure 2 As shown, the method provided in the embodiments of this application is used to... Figure 1 The results after reconstruction and Figure 1 The test images in the images appear to have little difference to the naked eye.

[0018] Figure 3 yes Figure 2 A schematic diagram of the error in the reconstruction results, as shown below. Figure 3 As shown, the dark blue areas represent regions with error values ​​close to 0, indicating that in most areas of the image, the reconstructed results are very close to the true values, with minimal error; the light blue areas represent regions with larger error values, these areas correspond to... Figure 1In the reconstruction of the ground features' edges and contours, the texture of the light blue areas is relatively blurry, and the contrast with the dark blue areas is not obvious. This indicates that the difference between the reconstructed ground feature edges and contours and the true values ​​is small; that is, the absolute value of the reconstruction error is controlled within a small range. This demonstrates that the method provided by the embodiments of this application effectively reconstructs the ground features' edges and contours. Figure 1 High-quality, high-resolution hyperspectral images were obtained after reconstruction.

[0019] In some embodiments, step S40 further includes the following steps: S41: determining the spatial features of the fused image F1, fused image F2, fused image F3, and difference image F4; S42: determining the spectral features of the fused image F1, fused image F2, fused image F3, and difference image F4; S43: concatenating the spatial features and spectral features along the channel dimension to obtain spatial-spectral fusion features; S44: dividing the spatial-spectral fusion features into predetermined groups along the channel dimension; S45: performing enhancement processing on each group of spatial-spectral fusion features to obtain multiple shallow features.

[0020] This approach combines the ability to capture local spatial information with the ability to extract spectral sequences. By integrating feature enhancement processing, it effectively enhances key information in the image, providing more discriminative features for subsequent image reconstruction.

[0021] In some embodiments, in step S40, shallow features of the fused images F1, F2, F3 and F4 and the difference image F4 can be extracted in the same manner to improve feature extraction efficiency.

[0022] In some embodiments, step S45 further includes the following steps: S451: Perform convolutional block attention processing on each group of spatial-spectral fusion features to obtain the corresponding local attention weight matrix; S452: Determine the mean of the local attention weight matrix; S453: Optimize the local attention weight matrix based on the mean; S454: Perform matrix multiplication operation between the optimized local attention weight matrix and the corresponding spatial-spectral fusion feature; S455: Perform pixel-level addition operation between the result of the matrix multiplication operation and the corresponding spatial-spectral fusion feature to obtain the enhanced spatial-spectral fusion feature for each group; S456: Connect all the enhanced spatial-spectral fusion features along the channel dimension to obtain multiple shallow features. Through multi-group attention-coordinated feature enhancement, key information in the image can be effectively enhanced.

[0023] In some embodiments, in step S453, the elements in the local attention weight matrix corresponding to each set of spatial spectral fusion features that are greater than the mean are set to 1, while the elements that are less than or equal to the mean remain unchanged, so as to obtain the optimized local attention weight matrix.

[0024] In some embodiments, step S41 further includes the following steps: S411: performing channel dimensionality reduction processing on the channels; S412: performing 2D convolution operation on the channel-dimensionality-reduced image features using a 3×3 convolution kernel; S413: performing batch normalization and ReLU activation function processing on the convolution result of step S412 to obtain spatial features. 2D convolution improves the ability to capture local spatial information, enhances spatial texture features, and increases the ability to express nonlinear features.

[0025] In some embodiments, in step S411, a 1×1 convolution operation can be performed to reduce the channel dimension, thereby reducing the amount of subsequent computation while preserving key features.

[0026] In some embodiments, step S42 further includes the following steps: S421: expanding the dimensions of the image features after channel dimensionality reduction (e.g., using the Reshape function); S422: performing 3D convolution processing on the processing result of step S421 using a 1×1×3 convolution kernel; S423: compressing the dimensions of the processing result of step S422 (e.g., using the Reshape function) to obtain spectral features. 3D convolution can extract local dependencies in the spectral dimension, improving the ability to extract spectral sequences.

[0027] Feature extraction is performed using 2D and 3D convolution processing. This approach leverages the ability of 3D convolution to extract spectral sequences and combines it with the ability of 2D convolution to capture spatial local information. Furthermore, by incorporating multiple sets of attention-based collaborative feature enhancement, key information in the image is effectively enhanced, providing more discriminative features for subsequent reconstruction tasks.

[0028] In some embodiments, step S50 further includes the following steps: S51: Performing discrete wavelet transform on the shallow features to obtain low-frequency information components, horizontal high-frequency information components, vertical high-frequency information components, and diagonal high-frequency information components; S52: Connecting the low-frequency information components, horizontal high-frequency information components, vertical high-frequency information components, and diagonal high-frequency information components along the channel dimension to obtain fused features; S53: Performing channel compression processing on the channels to obtain fused features. S54: Based on fusion features Determine the fusion features after nonlinear learning S55: Fusing features after convolution processing With fusion features S56: Perform pixel-level addition operations; S57: Perform convolution processing (e.g., 1×1 convolution) on the pixel-level addition results of step S55, and restore the resolution through discrete wavelet inverse transform to obtain multiple enhanced features. By introducing prior information in the frequency dimension and combining it with depthwise separable convolution operations, high-frequency features such as image details and edges can be effectively enhanced, the enhanced feature extraction effect can be optimized, and the extracted spatial information can be fully preserved.

[0029] In some embodiments, the wavelet basis function is not specifically limited in step S51, and the Haar wavelet basis function can be selected.

[0030] In some embodiments, in step S53, a 1×1 convolution operation can be performed on the fused features obtained in step S52 to achieve channel compression and reduce the amount of subsequent parameter calculations.

[0031] In some embodiments, step S54 further includes the following step: S541: merging features Perform depthwise separable convolution processing; S542: Perform nonlinear learning on the processing result of step S541; S543: Perform layer normalization, 1×1 convolution and GeLU activation function processing on the nonlinear learning result of step S542 to obtain the fused features after nonlinear learning. .

[0032] In some embodiments, in step S50, all shallow features can be enhanced in the same manner to obtain enhanced features.

[0033] In some embodiments, step S60 further includes the following steps: S61: Perform layer normalization and linear transformation layer processing on the enhanced features to obtain spatial query features, spatial key features, and spatial value features; S62: Perform matrix multiplication on the transposes of the spatial query features and spatial key features, perform matrix multiplication on the result with the spatial value features, and perform pixel-level addition on the result with the enhanced features to obtain intermediate features with spatial information; S63: Perform layer normalization and convolution processing (e.g., 1×1 convolution) on the intermediate features to obtain spectral query features, spectral key features, and spectral value features; S64: Perform matrix multiplication on the transposes of the spectral query features and spectral key features, perform matrix multiplication on the result with the spectral value features, and perform pixel-level addition on the result with the intermediate features to obtain deep features, further improving the ability to capture long-distance spatial and spectral information.

[0034] In some embodiments, in step S60, all enhanced features can be extracted in the same manner to obtain deep features.

[0035] In some embodiments, step S70 further includes the following steps: S71: Connecting the deep features of the first branch and the shallow features of the first branch along the channel dimension to obtain the first branch fusion feature; S72: Connecting the deep features of the second branch, the shallow features of the second branch, and the fusion feature of the first branch along the channel dimension to obtain the second branch fusion feature; S73: Connecting the deep features of the fourth branch and the shallow features of the fourth branch along the channel dimension to obtain the fourth branch fusion feature; S74: Connecting the deep features of the third branch, the shallow features of the third branch, the fusion feature of the second branch, and the fusion feature of the fourth branch along the channel dimension to obtain the fusion feature.

[0036] By adopting a progressive fusion approach, shallow and deep spatial-spectral features of multiple scales of input images extracted from different branches are merged step by step, ensuring seamless and comprehensive fusion of spatial and spectral information. This fully utilizes the rich information in the input images and effectively enhances detail features and preserves spectral information fidelity.

[0037] In some embodiments, step S80 further includes the following step: S81: Upsample the hyperspectral image S81: Perform pixel-level addition with the fused features; S82: Perform 3×3 convolution on the result of step S81; S83: Apply ReLU activation function and perform 3×3 convolution on the result of convolution in step S82; S84: Perform pixel-level addition with the result of step S83 and the result of step S81 to obtain a high-resolution hyperspectral image. Performing pixel-level addition between the upsampled hyperspectral image and the fused features through cross-connection not only effectively preserves the spatial and spectral features of the original input image, improving the reconstruction quality of the high-resolution hyperspectral image, but also accelerates the network's convergence speed and improves computational efficiency.

[0038] In some embodiments, in step S10, the height of the acquired high-resolution multispectral image is set to twice the height of the low-resolution hyperspectral image; the width of the high-resolution multispectral image is set to twice the width of the low-resolution hyperspectral image; and the number of channels of the low-resolution hyperspectral image is set to be much smaller than the number of channels of the high-resolution multispectral image.

[0039] In some embodiments, the sampling method used to sample the low-resolution hyperspectral image and the high-resolution multispectral image in step S20 is not specifically limited, and can be bilinear interpolation, nearest neighbor interpolation, or bicubic interpolation, etc.

[0040] The following five specific embodiments verify the effectiveness of the method for super-resolution reconstruction of hyperspectral images of objects provided in this application.

[0041] Example 1: Five models were selected and compared on the test dataset: MSDCNN (A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening), SSFCNN (Spatial and Spectral Fusion with CNN for Hyperspectral Image Super-Resolution), ResTFNet (Remote sensing image fusion based on two-stream fusion network), MCTNet (Multi-hierarchical crosstransformer for hyperspectral and multispectral image fusion), and DCTransformer (Reciprocal transformer for hyperspectral and multispectral image fusion).

[0042] Table 1. Reconstruction performance metrics of different methods on the test dataset Table 1 shows the reconstruction performance metrics of different methods on the test dataset. As shown in Table 1, the method for super-resolution reconstruction of hyperspectral images of objects provided in this application shows significant advantages in all evaluation metrics and has better reconstruction performance.

[0043] As shown in Table 1, in the peak signal-to-noise ratio (PSNR) fidelity index, the method provided in this application exhibits the best performance with a value of 51.4254, which is 2.6751 dB higher than the second-best DCTransformer model's PSNR value of 48.7503. This indicates that the reconstruction result is better in terms of detail restoration and image sharpness. The root mean square error (RMSE), relative dimensionless global synthesis error (ERGAS), spectral angle mapping (SAM), and structural similarity index (SSIM) also achieve the best performance, with improvements of 0.2366, 0.0448, 0.0706, and 0.0028 respectively compared to the DCTransformer model. This further verifies the good effects of the method provided in this application in terms of overall image structure restoration, spatial detail preservation, and spectral consistency, resulting in a higher overall quality reconstructed high-resolution hyperspectral image.

[0044] Example 2: Replace steps S41 and S42 of this application with a single-branch 3×3 convolution and remove step S45. The model is denoted as Model. A .

[0045] Table 2 Ablation experiment results of single-branch 3×3 convolution and this application on the test dataset. Table 2 shows the ablation experiment results on the test dataset for two methods provided in this application: extracting spatial and spectral features of the fused image from two branches using 2D and 3D convolutions respectively, and extracting features using traditional single-branch 3×3 convolution. As shown in Table 2, compared with the method provided in this application, after replacing the processing method of steps S41-S42 and removing step S45, the model Model A The PSNR decreased by 0.8240, RMSE increased by 0.0942, ERGAS increased by 0.0213, SAM increased by 0.396, and SSIM decreased by 0.0016. All performance metrics showed varying degrees of decline. This indicates that the use of 2D and 3D convolutions in steps S41-S42 to extract spatial and spectral features from the fused image from the two branches, and the enhancement processing of each set of spatial-spectral fusion features in step S45, can improve the ability to capture spectral and spatial features, strengthen key useful information, and effectively improve the reconstruction performance.

[0046] Example 3: Step S50 of this application is removed, and the model is denoted as Model. B .

[0047] Table 3. Ablation experiment results on the test dataset with or without shallow feature enhancement processing. Table 3 shows the ablation experiment results on the test dataset after enhancing and removing multiple shallow features in step S50 of this application. As shown in Table 3, compared with the method provided in this application, after removing the features in step S50, the model... B The PSNR decreased by 0.4229, RMSE increased by 0.0376, ERGAS increased by 0.0113, SAM increased by 0.0157, and SSIM decreased by 0.0005. All metrics showed varying degrees of performance degradation. This indicates that the S50 step, by introducing prior information in the frequency dimension, effectively enhances and utilizes high-frequency features such as image details and edges, which can help improve the quality of the reconstructed image.

[0048] Example 4: Step S60 of this application is removed, and the model is denoted as Model. C .

[0049] Table 4 Ablation experiment results on the test dataset with and without feature extraction of enhanced features. Table 4 shows the ablation experiment results on the test dataset after feature extraction processing of multiple enhancement features in step S60 of this application, with and without retention. As shown in Table 4, compared with the method provided in this application, after removing the enhancement features in step S60, the model... C The PSNR decreased by 0.5572, RMSE increased by 0.0661, ERGAS increased by 0.0184, SAM increased by 0.0279, and SSIM decreased by 0.0011. All performance metrics showed varying degrees of decline, indicating that the S60 step can effectively utilize long-range spatial-spectral dependencies and significantly improve reconstruction performance.

[0050] Example 5: Remove the fused image F1, fused image F2, and difference image F4 from step S30 of this application, and retain only the fused image F3. The model is denoted as Model. D .

[0051] Table 5. Ablation experiment results on the test dataset for whether only the fused image F3 is retained. Table 5 shows the ablation experiment results on the test dataset after removing the fused image F1, fused image F2, and difference image F4 from step S30 and retaining only the fused image F3. As shown in Table 5, compared with the method provided in this application, after modifying step S30, the model Model... DThe PSNR decreased by 1.8091, RMSE increased by 0.1704, ERGAS increased by 0.0310, SAM increased by 0.0565, and SSIM decreased by 0.0024. All metrics showed varying degrees of performance degradation. This indicates that by generating difference images and multiple fused images, multi-scale fusion and difference features can be fully utilized to effectively preserve image details and edge information, significantly improving reconstruction performance.

[0052] Another embodiment of this application provides an apparatus for super-resolution reconstruction of a hyperspectral image of an object, comprising: an image acquisition module configured to acquire a low-resolution hyperspectral image of the object and a high-resolution multispectral image of the object; and an image preprocessing module configured to upsample the low-resolution hyperspectral image by a factor of 2 to obtain an upsampled hyperspectral image. The low-resolution hyperspectral image is downsampled by a factor of 2 to obtain the downsampled hyperspectral image. The high-resolution multispectral image is downsampled by a factor of 2 to obtain the downsampled multispectral image. The high-resolution multispectral image was downsampled by 4 times to obtain the downsampled multispectral image. The image fusion module is configured to: convert downsampled hyperspectral images... With downsampled multispectral images Connect the channels to obtain the fused image F1; combine the low-resolution hyperspectral image with the downsampled multispectral image. Connect the channels to obtain the fused image F2; then upsample the hyperspectral image. The high-resolution multispectral image is concatenated along the channel dimension to obtain the fused image F3; the upsampled hyperspectral image is then... The system performs pixel-level subtraction with the high-resolution multispectral image to obtain the difference image F4; a shallow feature extraction module is configured to extract features from the fused images F1, F2, F3, and the difference image F4 respectively to obtain multiple shallow features; a feature enhancement module is configured to enhance the multiple shallow features separately to obtain multiple enhanced features; a deep feature extraction module is configured to extract features from the multiple enhanced features separately to obtain multiple deep features; a feature fusion module is configured to fuse the multiple shallow features and multiple deep features separately to obtain a fused feature; and an image reconstruction module is configured to reconstruct the image based on the upsampled hyperspectral image. By combining features, high-resolution hyperspectral images are obtained.

[0053] The apparatus for super-resolution reconstruction of hyperspectral images of an object provided in this application obtains sampling results of different resolutions from high-resolution hyperspectral images and high-resolution multispectral images through an image preprocessing module, and processes the images according to the corresponding channel dimensions to obtain difference images and multiple fused images, which can fully consider the correlation and multi-scale characteristics between multispectral and hyperspectral images. Through a feature enhancement module and a deep feature extraction module, features are extracted and enhanced from the fused images, improving the accuracy of local and global feature extraction and enhancing high-frequency features such as image details and edges. Compared with traditional fusion reconstruction methods and existing super-resolution reconstruction methods that incorporate deep neural networks, this apparatus improves cross-modal information interaction capabilities, has better spectral fidelity and spatial detail preservation, and obtains high-quality high-resolution hyperspectral images.

[0054] Regarding the embodiments of this application, it should also be noted that, without conflict, the embodiments of this application and the features in the embodiments can be combined with each other to obtain new embodiments.

[0055] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for super-resolution reconstruction of a hyperspectral image of an object, characterized in that, It includes the following steps: S10: Acquire a low-resolution hyperspectral image and a high-resolution multispectral image of the object; S20: Upsample the low-resolution hyperspectral image by a factor of 2 to obtain an upsampled hyperspectral image. ; The low-resolution hyperspectral image is downsampled by a factor of 2 to obtain a downsampled hyperspectral image. ; The high-resolution multispectral image is downsampled by 2 times to obtain a downsampled multispectral image. ; The high-resolution multispectral image was downsampled by 4 times to obtain a downsampled multispectral image. ; S30: Transfer the downsampled hyperspectral image With the downsampled multispectral image Connect the channels to obtain the fused image F1; The low-resolution hyperspectral image and the downsampled multispectral image Connect the channels to obtain the fused image F2; The upsampled hyperspectral image The fused image F3 is obtained by concatenating the high-resolution multispectral image along the channel dimension. The upsampled hyperspectral image Perform pixel-level subtraction with the high-resolution multispectral image to obtain the difference image F4; S40: Perform feature extraction on the fused image F1, the fused image F2, the fused image F3, and the difference image F4 respectively to obtain multiple shallow features; S50: Enhance the multiple shallow features respectively to obtain multiple enhanced features; S60: Perform feature extraction on the multiple enhanced features to obtain multiple deep features; S70: The multiple shallow features and multiple deep features are fused to obtain fused features; S80: Based on the upsampled hyperspectral image Combined with the aforementioned fusion features, a high-resolution hyperspectral image is obtained.

2. The method according to claim 1, characterized in that, Step S40 also includes the following steps: S41: Determine the spatial features of the fused image F1, the fused image F2, the fused image F3, and the difference image F4; S42: Determine the spectral features of the fused image F1, the fused image F2, the fused image F3, and the difference image F4; S43: Connect the spatial features and the spectral features in the channel dimension to obtain spatial-spectral fusion features; S44: Divide the spatial-spectral fusion features into predetermined groups along the channel dimension; S45: Enhance each set of spatial spectrum fusion features to obtain the multiple shallow features.

3. The method according to claim 2, characterized in that, Step S45 also includes the following steps: S451: Perform convolutional block attention processing on each group of spatial spectrum fusion features to obtain the corresponding local attention weight matrix; S452: Determine the mean of the local attention weight matrix; S453: Optimize the local attention weight matrix based on the mean; S454: Perform matrix multiplication on the optimized local attention weight matrix and the corresponding spatial spectral fusion feature; S455: Perform pixel-level addition on the result of the matrix multiplication operation and the corresponding spatial spectrum fusion feature to obtain each enhanced spatial spectrum fusion feature; S456: Connect all the enhanced spatial-spectral fusion features along the channel dimension to obtain the multiple shallow features.

4. The method according to claim 1, characterized in that, The S50 step also includes the following steps: S51: Perform discrete wavelet transform on the shallow features to obtain low-frequency information components, horizontal high-frequency information components, vertical high-frequency information components, and diagonal high-frequency information components; S52: Connect the low-frequency information component, the horizontal high-frequency information component, the vertical high-frequency information component, and the diagonal high-frequency information component along the channel dimension to obtain fusion features; S53: Perform channel compression processing on the channel to obtain fusion features. ; S54: Based on the fusion features Determine the fusion features after nonlinear learning ; S55: The fused features after convolution processing With the fusion feature Perform pixel-level addition operations; S56: Perform convolution processing on the pixel-level addition operation results of step S55, and restore the resolution through discrete wavelet inverse transform to obtain the multiple enhanced features.

5. The method according to claim 4, characterized in that, Step S60 also includes the following steps: S61: Perform layer normalization and linear transformation layer processing on the enhanced features to obtain spatial query features, spatial key features and spatial value features; S62: Perform matrix multiplication on the transpose of the spatial query feature and the spatial key feature, perform matrix multiplication on the result with the spatial value feature, and perform pixel-level addition on the result with the enhancement feature to obtain an intermediate feature with spatial information. S63: Perform layer normalization and convolution processing on the intermediate features to obtain spectral query features, spectral key features and spectral value features; S64: Perform matrix multiplication on the transpose of the spectral query feature and the spectral key feature, perform matrix multiplication on the result with the spectral value feature, and perform pixel-level addition on the result with the intermediate feature to obtain the deep feature.

6. The method according to claim 1, characterized in that, Step S70 also includes the following steps: S71: Connect the deep features of the first branch and the shallow features of the first branch along the channel dimension to obtain the fused features of the first branch; S72: Connect the deep features of the second branch, the shallow features of the second branch, and the fused features of the first branch along the channel dimension to obtain the fused features of the second branch; S73: Connect the deep features of the fourth branch and the shallow features of the fourth branch along the channel dimension to obtain the fused features of the fourth branch; S74: Connect the deep features of the third branch, the shallow features of the third branch, the fused features of the second branch, and the fused features of the fourth branch along the channel dimension to obtain the fused features.

7. The method according to claim 1, characterized in that, The S80 step also includes the following steps: S81: Transfer the upsampled hyperspectral image Perform pixel-level addition operations with the fused features; S82: Perform 3×3 convolution on the result of step S81; S83: Apply the ReLU activation function to perform a 3×3 convolution on the convolution result from step S82; S84: Perform pixel-level addition on the processing result of step S83 and the calculation result of step S81 to obtain the high-resolution hyperspectral image.

8. The method according to claim 2, characterized in that, Step S41 also includes the following steps: S411: Perform channel dimensionality reduction processing on the channel; S412: Perform 2D convolution operation on the image features after dimensionality reduction of the channels using a 3×3 convolution kernel; S413: Perform batch normalization and ReLU activation function processing on the convolution results of step S412 to obtain the spatial features.

9. The method according to claim 2, characterized in that, Step S42 also includes the following steps: S421: Perform dimensionality expansion processing on the image features after dimensionality reduction of the channels; S422: Perform 3D convolution processing on the processing result of step S421 using a 1×1×3 convolution kernel; S423: Perform dimensionality compression processing on the processing result of step S422 to obtain the spectral features.

10. The method according to claim 4, characterized in that, Step S54 also includes the following steps: S541: Regarding the fusion feature Perform depthwise separable convolution processing; S542: Perform nonlinear learning on the processing results of step S541; S543: Perform layer normalization, 1×1 convolution, and GeLU activation function processing on the nonlinear learning results of step S542 to obtain the fused features after nonlinear learning. .

11. An apparatus for super-resolution reconstruction of a hyperspectral image of an object, characterized in that, It includes: An image acquisition module, configured to acquire both a low-resolution hyperspectral image and a high-resolution multispectral image of the object; Image preprocessing module, the image preprocessing module is configured as follows: The low-resolution hyperspectral image is upsampled by a factor of 2 to obtain an upsampled hyperspectral image. ; The low-resolution hyperspectral image is downsampled by a factor of 2 to obtain a downsampled hyperspectral image. ; The high-resolution multispectral image is downsampled by 2 times to obtain a downsampled multispectral image. ; The high-resolution multispectral image was downsampled by 4 times to obtain a downsampled multispectral image. ; Image fusion module, wherein the image fusion module is configured as follows: The downsampled hyperspectral image With the downsampled multispectral image Connect the channels to obtain the fused image F1; The low-resolution hyperspectral image and the downsampled multispectral image Connect the channels to obtain the fused image F2; The upsampled hyperspectral image The fused image F3 is obtained by concatenating the high-resolution multispectral image along the channel dimension. The upsampled hyperspectral image Perform pixel-level subtraction with the high-resolution multispectral image to obtain the difference image F4; A shallow feature extraction module is configured to extract features from the fused image F1, the fused image F2, the fused image F3, and the difference image F4 respectively to obtain multiple shallow features; A feature enhancement module is configured to enhance the multiple shallow features respectively to obtain multiple enhanced features; A deep feature extraction module is configured to extract features from the multiple enhanced features respectively to obtain multiple deep features; The feature fusion module is configured to fuse multiple shallow features and multiple deep features respectively to obtain fused features; The image reconstruction module is configured to reconstruct the image based on the upsampled hyperspectral image. Combined with the aforementioned fusion features, a high-resolution hyperspectral image is obtained.

Images