Feature fusion network structure and method and related spectral image reconstruction system

Abstract

The invention discloses a feature fusion network structure and method and a related spectral image reconstruction system, and belongs to the technical field of multispectral images. According to the feature fusion network structure provided by the invention, Fourier transform is adopted to process spectral image features in a frequency domain, full spatial domain information is fully utilized, more spatial resolutions are reserved, detail information in the spectral image is better captured and recovered, in addition, an attention mechanism is introduced at a jump connection position, and an attention effect is improved. The features of the decoding end are fused with upper-layer features through up-sampling, so that effective integration of context information is realized, and the efficiency and quality of spectral image reconstruction are improved; the system for reconstructing the spectral image based on the feature fusion network structure can achieve a better reconstruction effect, restore the real texture of the image and reduce mosaic traces while keeping smaller parameter quantity and FLOPs, and is close to the quality level of an original multi-spectral image visually.

Description

Technical Field

[0001] The present invention belongs to the technical field of multispectral images, and in particular relates to a feature fusion network structure, method and related spectral image reconstruction system. Background Art

[0002] Spectral Imaging (SI) technology cleverly integrates digital imaging and spectroscopy, and can simultaneously capture the spatial distribution information and fine spectral characteristics of materials, thereby achieving in-depth analysis of material composition and microstructure. This technology has shown a wide range of application potential in many fields, including food quality control, medical diagnostic imaging, satellite remote sensing monitoring, precision agricultural management, and advanced computer vision analysis, and its scientific research value cannot be underestimated.

[0003] The core of spectral imaging is that it can construct a three-dimensional (3D) spectral cube containing the spatial position and spectral characteristics of the source object through limited band selection in different wavelength ranges. Given the limitations of two-dimensional (2D) detector arrays under current technical conditions, system designers usually adopt two strategies to meet this challenge: one is to measure and construct the spectral cube one by one through 2D slices of time series, which is called scanning spectral imaging; the other is to split the data cube into multiple 2D elements that can be measured in parallel and reassemble it into a complete cube in subsequent processing. This method is called snapshot spectral imaging.

[0004] Scanning spectral imaging systems are divided into three modes: point scanning, line scanning, and spectral scanning. Point scanning hyperspectral cameras use a spectrometer to obtain one-dimensional spectral information through a single scan, and then combine it with a two-dimensional spatial scan to complete the construction of the entire 3D spectral cube. Line scanning cameras use a scanner to capture both one-dimensional spatial and one-dimensional spectral information in one scan. Spectral scanning cameras use a spectrometer to capture only two-dimensional spatial information within a narrow spectral band in a single scan.

[0005] Snapshot spectral imaging technology can capture the spectral information of a scene in a single observation, significantly shortening the scanning cycle and making high frame rate imaging of dynamic scenes possible. Compared with scanning systems, snapshot spectral imaging equipment is lighter and smaller in size, and is particularly suitable for monitoring rapidly changing scenes. However, mainstream snapshot imaging systems based on multispectral filter arrays often need to compromise on spatial resolution, resulting in a mosaic-like distribution of the acquired spectral images.

[0006] To address this problem, three types of snapshot spectral image demosaicing algorithms have been developed: (1) interpolation-based methods, although simple to implement, efficient and stable, have poor reconstructed image quality, which affects the performance of subsequent applications; (2) low-rank matrix completion methods, which regard spectral image reconstruction as a low-rank matrix completion problem and introduce non-negative constraints, effectively improving the image reconstruction quality; (3) deep learning-based methods, which use advanced algorithms such as residual connections, attention mechanisms, and dense connections to construct convolutional neural networks (CNNs), and achieve high-quality image reconstruction through training and testing of image data. However, this method may also cause false color artifacts in high-contrast bright areas, halo effects in demosaicing results, and convergence problems during network training.

[0007] Therefore, how to provide a solution that can reduce the error of the reconstructed image, improve the image quality, reduce mosaic traces, and make the reconstructed image visually close to the quality level of the original spectral image has become a technical problem that technical personnel in this field urgently need to overcome. Summary of the invention

[0008] The purpose of the present invention is to provide a feature fusion network structure, method and related spectral image reconstruction system to overcome the problem of poor quality of reconstructed images in the prior art.

[0009] The present invention solves the above technical problems through the following technical solutions: A feature fusion network structure includes the following modules: Shallow feature extraction module, used for image features , perform 1×1 convolution to extract shallow features; A feature processing module, used for performing multi-branch processing with different receptive fields on shallow features to obtain features after multi-branch processing, wherein at least one branch includes Fourier transform processing; The feature fusion module is used to merge the features processed by multiple branches, perform 1×1 convolution, and then combine them with the input features. Add together to get the first feature A.

[0010] A method for a feature fusion network structure comprises the following steps: Input image features , perform 1×1 convolution to extract shallow features and obtain shallow features , for shallow features Perform multi-branch processing with different receptive fields to obtain features , specifically: shallow features Without processing, get the features ; Shallow Features After 3×3 convolution, it is divided into two paths, one of which is not processed, and the feature is obtained , another way, the feature Deliver to branch road; Shallow Features After 3×3 convolution, it is divided into two paths, one of which is not processed, and the feature is obtained , another way, the feature Deliver to branch road; Shallow Features and Features After 3×3 convolution, the feature is obtained ; Shallow Features The feature is obtained by Fourier transform, two 1×1 convolutions and Fourier inverse transform. ; Among them, FFT is Fourier forward transform; IFFT is Fourier inverse transform; It is a 1×1 convolution; Merge All Features , perform 1×1 convolution, and then combine with the input features Add together to get the first feature A.

[0011] A method for reconstructing a spectral image based on a feature fusion network structure comprises the following steps: The spectral image features are extracted through 3×3 convolution to obtain the image features ; A method for a feature fusion network structure based on claim 2, obtaining a first feature A; Perform two PixelUnshuffle operations on the first feature A to reduce the spatial size of the image and increase the number of feature channels without losing information, so as to obtain the first feature B to be processed, and then perform two operations; Among them, the first feature B to be processed is subjected to channel attention weighted processing to obtain the first feature after channel attention weighted processing, and then the first feature after channel attention weighted processing is subjected to spatial attention weighted processing to obtain the first feature C after spatial attention weighted processing, and the first feature C after spatial attention weighted processing is added to the first feature B to be processed to obtain a comprehensive feature D, and then the comprehensive feature D after convolution processing is added to the comprehensive feature D without convolution processing to obtain the second feature; The other path performs two PixelShuffle operations on the first feature B to be processed, restores the spatial size and number of feature channels of the image, and upsamples and adds the features of the results of the two PixelShuffle operations to obtain the third feature; The third feature is added to the second feature to obtain the final reconstructed spectral image.

[0012] Furthermore, the channel attention weighted processing specifically includes the following steps: Global average pooling, fully connected network, ReLU activation, fully connected network and Sigmoid activation are performed in sequence to generate channel attention weights. The weight vector of the channel attention weights is element-wise multiplied with the first feature B to be processed to obtain the first feature after channel attention weighted processing.

[0013] Furthermore, the spatial attention weighted processing specifically includes the following steps: The first feature after the channel attention weighted processing is subjected to global pooling and maximum pooling respectively to obtain the first feature of the global pooling and the first feature of the maximum pooling respectively. The first feature of the global pooling and the first feature of the maximum pooling are added to obtain a comprehensive spatial feature representation. The comprehensive spatial feature representation is sequentially subjected to 3×3 convolution and Sigmoid activation to generate the spatial attention weight. The weight vector of the spatial attention weight is element-wise multiplied with the first feature after the channel attention weighted processing to obtain the first feature C after the spatial attention weighted processing.

[0014] Furthermore, the comprehensive feature D is subjected to convolution processing, specifically including 3×3 convolution and 1×1 convolution processing in sequence.

[0015] A system for reconstructing spectral images based on a feature fusion network structure, using the above-mentioned method for reconstructing spectral images based on a feature fusion network structure, specifically includes the following modules: The encoding module is used to extract spectral image features through 3×3 convolution to obtain image features , based on a feature fusion network structure method, the first feature A is obtained; the first feature A is subjected to two PixelUnshuffle operations, the spatial size of the image is reduced without losing information, the number of feature channels is increased, and the first feature B to be processed is obtained; The visual attention module includes a channel attention module, a spatial attention module and a tail processing module, wherein the channel attention module is used to perform channel attention weighted processing on the first feature B to be processed to obtain a first feature after the channel attention weighted processing; the spatial attention module is used to perform spatial attention weighted processing on the first feature after the channel attention weighted processing to obtain a first feature C after the spatial attention weighted processing; the tail processing module is used to add the first feature C after the spatial attention weighted processing to the first feature B to be processed to obtain a comprehensive feature D, and then add the comprehensive feature D after convolution processing to the comprehensive feature D that has not been convolution processed to obtain a second feature; The decoding module is used to perform two PixelShuffle operations on the first feature B to be processed, restore the spatial size and the number of feature channels of the image, upsample and add the features of the results of the two PixelShuffle operations to obtain a third feature; add the third feature to the second feature to obtain the final reconstructed spectral image.

[0016] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for reconstructing a spectral image based on a feature fusion network structure is implemented.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for reconstructing a spectral image based on a feature fusion network structure.

[0018] A computer program product includes a computer program, which implements the above-mentioned method for reconstructing spectral images based on a feature fusion network structure when executed by a processor.

[0019] Compared with the prior art, the positive and progressive effects of the present invention are: The feature fusion network structure provided by the present invention uses Fourier transform to process image features in the frequency domain, makes full use of the full spatial domain information, retains more spatial resolution, and better captures and restores the detail information in the image. In addition, the introduction of the attention mechanism at the jump connection can more accurately capture and highlight relevant information while suppressing the influence of non-critical features; the features at the decoding end are fused with the upper-level features through upsampling to achieve effective integration of contextual information and improve the efficiency and quality of spectral image reconstruction.

[0020] The system for reconstructing spectral images based on a feature fusion network structure provided by the present invention can achieve better reconstruction effect, restore the real texture of the image, reduce mosaic traces, and visually approach the quality level of the original spectral image while maintaining a small number of parameters and FLOPs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings in the specification are used to provide further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0022] Figure 1 Schematic diagram of the feature fusion network structure (Res2FFT); Figure 2 Schematic diagram of the system (RF_Unet) for reconstructing spectral images based on feature fusion network structure; Figure 3Schematic diagram of the visual attention module (VAM); Figure 4 Schematic diagram of the channel attention module (SE); Figure 5 Schematic diagram of the spatial attention module (SA); Figure 6 A schematic diagram of a difference heat map of different algorithms for ARAD_HS_0911 data provided by an embodiment of the present invention; Figure 7 A schematic diagram of comparing spectral curves of different algorithms for ARAD_HS_0911 data provided by an embodiment of the present invention; Figure 8 A schematic diagram of a box diagram under four indicators of a different method provided by an embodiment of the present invention; Fig. 9 A schematic diagram of a difference heat map of an ARAD_HS_0911 data ablation experiment provided by an embodiment of the present invention; Fig.10 A schematic diagram for comparing spectral curves of an image reconstructed by an ARAD_HS_0911 data ablation experiment provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0024] The present invention is further described in detail below in conjunction with the accompanying drawings, which are intended to explain the present invention rather than to limit it.

[0025] See also Figure 1 , the present invention provides a feature fusion network structure, including the following modules: Shallow feature extraction module, used for image features , perform 1×1 convolution to extract shallow features; A feature processing module, used for performing multi-branch processing with different receptive fields on shallow features to obtain features after multi-branch processing, wherein at least one branch includes Fourier transform processing; The feature fusion module is used to merge the features processed by multiple branches, perform 1×1 convolution, and then combine them with the input features. Add together to get the first feature A.

[0026] The present invention also provides a method for feature fusion network structure, comprising the following steps: Input image features , perform 1×1 convolution to extract shallow features and obtain shallow features , for shallow features Perform multi-branch processing with different receptive fields to obtain features , specifically: shallow features Without processing, get the features ; Shallow Features After 3×3 convolution, it is divided into two paths, one of which is not processed, and the feature is obtained , another way, the feature Deliver to branch road; Shallow Features After 3×3 convolution, it is divided into two paths, one of which is not processed, and the feature is obtained , another way, the feature Deliver to branch road; Shallow Features and Features After 3×3 convolution, the feature is obtained ; Shallow Features The feature is obtained by Fourier transform, two 1×1 convolutions and Fourier inverse transform. ; Among them, FFT is Fourier forward transform; IFFT is Fourier inverse transform; It is a 1×1 convolution; Merge All Features , perform 1×1 convolution, and then combine with the input features Add together to get the first feature A.

[0027] For the convenience of description, the feature fusion network structure is defined as Res2FFT, and the method network for reconstructing spectral images based on the feature fusion network structure is defined as RF_Unet network.

[0028] The method of the feature fusion network structure provided by the present invention can capture features at different scales through hierarchical residual connections, which is very useful for understanding the multi-level details in the image. This multi-scale feature extraction capability enables Res2FFT to have better performance when processing image reconstruction tasks. The Fourier transform is integrated into Res2FFT for image reconstruction. This building block benefits from eliminating the high-frequency and low-frequency differences between the mosaic image and the true value image pair, while capturing the interaction of long-range and short-range contexts. The result obtained after Fourier transforming the image has a global receptive field. The Fourier transform converts the image from the spatial domain to the frequency domain, and in this process, it considers the mutual relationship of the entire domain in the image. Therefore, the transformed frequency domain representation is no longer limited to the local area of ​​the image, but contains the global information of the entire image. This means that any frequency component in the frequency domain reflects the contribution of all positions in the original image to this frequency component. Therefore, the Fourier transform provides a global perspective for the image, which is very useful for analyzing the overall structure and pattern of the image.

[0029] Based on the same inventive concept, the present invention also provides a method for reconstructing a spectral image based on a feature fusion network structure, comprising the following steps: extracting spectral image features through 3×3 convolution to obtain image features ; Based on the method of a feature fusion network structure described in claim 2, a first feature A is obtained; Perform two PixelUnshuffle operations on the first feature A to reduce the spatial size of the image and increase the number of feature channels without losing information, so as to obtain the first feature B to be processed, and then perform two operations; Among them, the first feature B to be processed is subjected to channel attention weighted processing to obtain the first feature after channel attention weighted processing, and then the first feature after channel attention weighted processing is subjected to spatial attention weighted processing to obtain the first feature C after spatial attention weighted processing, and the first feature C after spatial attention weighted processing is added to the first feature B to be processed to obtain a comprehensive feature D, and then the comprehensive feature D after convolution processing is added to the comprehensive feature D without convolution processing to obtain the second feature; The other path performs two PixelShuffle operations on the first feature B to be processed, restores the spatial size and number of feature channels of the image, and upsamples and adds the features of the results of the two PixelShuffle operations to obtain the third feature; The third feature is added to the second feature to obtain the final reconstructed spectral image.

[0030] Specifically, the channel attention weighted processing includes the following steps: Global average pooling, fully connected network, ReLU activation, fully connected network and Sigmoid activation are performed in sequence to generate channel attention weights. The weight vector of the channel attention weights is element-wise multiplied with the first feature B to be processed to obtain the first feature after channel attention weighted processing.

[0031] Specifically, the spatial attention weighted processing includes the following steps: The first feature after the channel attention weighted processing is subjected to global pooling and maximum pooling respectively to obtain the first feature of the global pooling and the first feature of the maximum pooling respectively. The first feature of the global pooling and the first feature of the maximum pooling are added to obtain a comprehensive spatial feature representation. The comprehensive spatial feature representation is sequentially subjected to 3×3 convolution and Sigmoid activation to generate the spatial attention weight. The weight vector of the spatial attention weight is element-wise multiplied with the first feature after the channel attention weighted processing to obtain the first feature C after the spatial attention weighted processing.

[0032] Specifically, the comprehensive feature D is subjected to convolution processing, specifically including 3×3 convolution and 1×1 convolution processing in sequence.

[0033] See also Figure 2 The encoder extracts image features through convolution and Res2FFT, and gradually reduces the spatial size of the image through two PixelUnshuffle operations. Correspondingly, the decoder gradually restores the size of the image through two PixelShuffle operations. The feature map of the encoder is further refined through the visual attention module (VAM), combined with the encoder features through jump connections, and the features in the decoder are upsampled and added to the first layer features to obtain the final reconstructed spectral image, which can retain more low-level information and help accurate pixel-level prediction.

[0034] See also Figure 3 The visual attention module combines the channel attention module and the spatial attention module. The channel attention module dynamically adjusts the importance weights of each channel feature to enhance the model's ability to capture key factors of reconstruction quality while suppressing the interference of non-core features. The spatial attention module weights features at the pixel level to achieve precise attention to key areas of the image, such as edges and textures. This process helps the model accurately reconstruct these key details during the demosaicing process, thereby improving the overall clarity and visual coherence of the image.

[0035] See also Figure 4The channel attention module enhances the representation ability of the network by recalibrating the feature response of the channel. First, the spatial information of each channel is compressed into a number through global average pooling (GAP); then a fully connected network (FC) is used to implement linear transformation, with ReLU activation function and Sigmoid activation function added in the middle to learn the nonlinear relationship between channels, and finally generate attention weights; this weight vector is element-wise multiplied with the first feature B to be processed to recalibrate the feature response of each channel.

[0036] See also Figure 5 The spatial attention module extracts global spatial information by performing global pooling and maximum pooling operations on the input feature map, and then adds the information obtained from the two pooling operations to obtain a comprehensive spatial feature representation; then, this representation is further processed by a convolutional layer, and finally, the spatial attention weight is generated by a Sigmoid activation layer. Finally, this weight vector is element-wise multiplied with the first feature after channel attention weighting to recalibrate the feature response of each channel.

[0037] As a specific embodiment of the present invention, the ARAD_1K dataset provided by the NTIRE2022 challenge is used in this example, which includes 900 training images, 50 verification images and 50 test images. Since the test images are not public, this experiment uses the 50 verification images as test images.

[0038] In order to verify the effectiveness of the proposed method, the present invention is compared with existing interpolation-based algorithms and deep learning-based methods, including weighted bilinear interpolation (WB), binary tree-based edge perception (BTES), hyperspectral demosaicing using pseudo-panchromatic images (PPID), graph and rank regularization (GRMR) demosaicing algorithm, directional interpolation (DI), mosaic convolution-attention network (MCAN), NTIRE2022 participating network Res2-Unet, and a multi-scale densely connected large kernel attention network (MDLKA).

[0039] In this example, mean relative absolute error (MRAE), root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and mean of spectral angle mapper (MSAM) are used as quantitative comparison indicators. PSNR calculates the difference in pixel values ​​between the reference image and the estimated image, and PEAK represents the maximum pixel value of the image (for the ARAD_1K dataset, PEAK=1). The four quantitative evaluation indicators are defined as:

[0040]

[0041]

[0042]

[0043] in, and Represent the true image and the reconstructed image respectively. Pixels The values ​​of the spectral channels, is the size of the ground truth image (number of pixels × number of spectral channels).

[0044] The test results on the ARAD_1K data test set are shown in Table 1. It can be seen from Table 1 that the test results of the RF_Unet network are better than other algorithms in all four evaluation indicators. PSNR reflects the similarity between the original image and the reconstructed image. Compared with WB, BTES, PPID, GRMR, DI, MCAN, Res2Unet and MDLKA, the PSNR average of the RF_Unet network is increased by 14.206dB, 15.8692dB, 11.6867dB, 13.9484dB, 13.0557dB, 5.7271dB, 2.2787dB and 0.3271dB, respectively, indicating that the quality of the reconstructed image has been significantly improved. RMSE is a common indicator to measure the difference between the reconstructed image and the original image, which can intuitively reflect the error size of the reconstructed image. The experimental results show that the RMSE value of the RF_Unet network is the smallest, which means that the reconstructed image has a smaller error and higher quality.

[0045] Table 1 Comparison of test results of different algorithms on the ARAD_1K dataset

[0046] See Table 2, which records the time required for each method to demosaic a single snapshot spectral image. By comparing the existing algorithms, it can be seen that the WB method takes the shortest time, the BTES method takes the longest time, and the deep learning-based methods take about 1 second, which is shorter than the PPID, GRMR and DI methods. The proposed method achieves faster demosaicing while ensuring image quality. Table 2 Comparison of the time required for demosaicing a single snapshot spectral image using different algorithms

[0047] The standard convolutional layer is replaced with the Res2Net module in the encoder and decoder of U-Net as the baseline model. By calculating the mean and variance of the four evaluation indicators of MRAE, RMSE, PSNR, and MSAM, the effectiveness of the Res2FFT module, visual attention module (VAM module) and multi-scale connection designed by the present invention are gradually verified. The results are shown in Table 3. It can be seen that after adding the FFT branch to Res2Net, the PSNR increases by about 1dB. It shows that the Res2FFT module combines convolution operation and Fourier transform, and the receptive field gradually increases to the global, extracts features locally and globally, and enhances the model's ability to capture high-frequency details. Then, the VAM module is added at the jump connection, and the PSNR increases by 1.0044dB, indicating that the VAM module better integrates low-level and high-level features, so that the reconstructed image retains details and has a good overall structure. Finally, the outputs of other layers of the decoder are upsampled and combined with the first layer feature maps to obtain the final RF_Unet network. The PSNR is increased by 0.4654dB. By merging the upsampled feature maps with the upper layer feature maps, the model can continuously reuse and update important feature information during the reconstruction process, and better reconstruct the full-resolution spectral image.

[0048] Table 3 Comparison of network ablation experiment test results

[0049] In this embodiment, in order to further verify the above conclusion, Figure 6 The heat map results of the difference between the reconstructed images of different algorithms and the reference spectral images are shown. It can be seen that the spectral images reconstructed by other methods have a large error with the reference spectral images, especially at the edge positions, and the high-frequency information of the image is not well restored. The de-mosaicing method proposed in the present invention has the smallest pixel difference on the difference map, indicating that the processed image has the highest visual consistency with the original image. This result means that this method recovers more original image information during the de-mosaicing process and reduces distortion and errors.

[0050] In addition, the spectral curves of the reconstructed spectral images of different methods and the reference spectral images at 5 evenly distributed positions were plotted for comparative analysis. Figure 7 As shown in the figure, by comparing these spectral curves, it can be observed that the reconstructed image of the method is highly consistent with the reference spectral image in the spectral curves of the pixels at the five positions, which shows that the method has excellent performance in noise suppression and detail fidelity.

[0051] In the de-mosaicing scenario, the box plot can reflect the performance stability and error range of each method on different test samples. Figure 8 As shown in the figure, according to the experimental results, it can be seen that the box plot of this method shows the smallest error range in these four indicators, indicating that this method can restore image information more accurately and has higher stability in the process of removing mosaics. In terms of the PSNR indicator, the box plot of this method has the shortest length and the highest median line, indicating that the overall PSNR value is relatively high. In terms of the other three indicators, the median line of the box plot of this method is closer to zero, proving that this method has a smaller error and better image quality.

[0052] In this example, the effectiveness of each module in improving model performance in the ablation experiment is verified through the visualization analysis of the difference heat map. Fig. 9 It can be seen that with the increase of model complexity, the reconstruction results are getting better and better, and the error is getting smaller and smaller. In the process of gradually introducing more modules, the quality of the reconstructed image has been significantly improved. After the introduction of FFT, the model is able to better capture the details of the image, which is manifested in the difference heat map as a significant reduction in local differences. After adding the attention module, the overall error is further reduced. To further evaluate the impact of each module on the image reconstruction quality in the ablation experiment, the spectral curves of the reconstructed spectral image and the reference spectral image at 5 evenly distributed positions are compared. The experimental results are shown in the attached figure. Fig.10 As shown in Figure 2, with the increase of model complexity and the gradual integration of various modules, the spectral curve of the reconstructed image tends to be consistent with the spectral curve of the reference image, and the error gradually decreases.

[0053] Based on the same inventive concept, the present application also provides a system for reconstructing spectral images based on a feature fusion network structure, which adopts the above-mentioned method for reconstructing spectral images based on a feature fusion network structure, and specifically includes the following modules: The encoding module is used to extract spectral image features through 3×3 convolution to obtain image features , based on a feature fusion network structure method, the first feature A is obtained; the first feature A is subjected to two PixelUnshuffle operations, the spatial size of the image is reduced without losing information, the number of feature channels is increased, and the first feature B to be processed is obtained; The visual attention module includes a channel attention module, a spatial attention module and a tail processing module, wherein the channel attention module is used to perform channel attention weighted processing on the first feature B to be processed to obtain a first feature after the channel attention weighted processing; the spatial attention module is used to perform spatial attention weighted processing on the first feature after the channel attention weighted processing to obtain a first feature C after the spatial attention weighted processing; the tail processing module is used to add the first feature C after the spatial attention weighted processing to the first feature B to be processed to obtain a comprehensive feature D, and then add the comprehensive feature D after convolution processing to the comprehensive feature D that has not been convolution processed to obtain a second feature; The decoding module is used to perform two PixelShuffle operations on the first feature B to be processed, restore the spatial size and the number of feature channels of the image, upsample and add the features of the results of the two PixelShuffle operations to obtain a third feature; add the third feature to the second feature to obtain the final reconstructed spectral image.

[0054] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for reconstructing a spectral image based on a feature fusion network structure are implemented, wherein the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0055] Based on the same inventive concept, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method for reconstructing a spectral image based on a feature fusion network structure are implemented. Specifically, the computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0056] Based on the same inventive concept, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned method for reconstructing a spectral image based on a feature fusion network structure.

[0057] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0059] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0061] Finally, it should be noted that the above-listed embodiments exist only as one or more specific forms of expression of the technical solution of the present invention. Their purpose is to clearly explain the concept, principle and application of the present invention through specific examples, and it is by no means intended to limit the protection scope of the present invention to these specific embodiments. In fact, the real value of the present invention lies in the technical ideas and innovations it proposes, rather than its form of expression or means of implementation.

[0062] For ordinary technicians in the relevant technical field, after in-depth reading and understanding of the technical solution of the present invention, they are fully capable of making various forms of changes, modifications or equivalent substitutions to the specific implementation methods of the invention based on their own professional knowledge and skills. These changes may include but are not limited to: adjusting the value range of technical parameters, optimizing algorithm processes to improve efficiency, replacing some technical components to achieve better compatibility or reduce costs, etc. As long as these changed technical solutions still substantially maintain the technical features claimed for protection by the original invention, that is, they can still achieve the core functions and effects of the present invention, then these changes should be deemed to fall within the scope of protection of the pending claims of the present invention.

[0063] In addition, with the continuous progress and development of technology, new technical means and methods continue to emerge, which also provides broad space for further improvement and perfection of the present invention. Therefore, the protection scope of the present invention should also include those reasonably foreseeable improvements and extensions based on the existing technology. As long as these improvements and extensions do not deviate from the basic principles and core concepts of the present invention, they should be regarded as equivalents of the present invention and are also protected by patent rights.

Claims

1. A feature fusion network structure, characterized in that: Includes the following modules: Shallow feature extraction module, used for image features , perform 1×1 convolution to extract shallow features; A feature processing module, used for performing multi-branch processing with different receptive fields on shallow features to obtain features after multi-branch processing, wherein at least one branch includes Fourier transform processing; The feature fusion module is used to merge the features processed by multiple branches, perform 1×1 convolution, and then combine them with the input features. Add together to get the first feature A.

2. A method for feature fusion network structure, characterized in that: The following steps are involved: Input image features , perform 1×1 convolution to extract shallow features and obtain shallow features , for shallow features Perform multi-branch processing with different receptive fields to obtain features , specifically: shallow features Without processing, get the features ; Shallow Features After 3×3 convolution, it is divided into two paths, one of which is not processed, and the feature is obtained , another way, the feature Deliver to branch road; Shallow Features After 3×3 convolution, it is divided into two paths, one of which is not processed, and the feature is obtained , another way, the feature Deliver to branch road; Shallow Features and Features After 3×3 convolution, the feature is obtained ; Shallow Features The feature is obtained by Fourier forward transform, two 1×1 convolutions and Fourier inverse transform. ; Among them, FFT is Fourier forward transform; IFFT is Fourier inverse transform; It is a 1×1 convolution; Merge All Features , perform 1×1 convolution, and then combine with the input features Add together to get the first feature A.

3. A method for reconstructing spectral images based on a feature fusion network structure, characterized in that: The following steps are involved: The spectral image features are extracted through 3×3 convolution to obtain the image features ; Based on the method of a feature fusion network structure described in claim 2, a first feature A is obtained; Perform two PixelUnshuffle operations on the first feature A to reduce the spatial size of the image and increase the number of feature channels without losing information, so as to obtain the first feature B to be processed, and then perform two operations; Among them, the first feature B to be processed is subjected to channel attention weighted processing to obtain the first feature after channel attention weighted processing, and then the first feature after channel attention weighted processing is subjected to spatial attention weighted processing to obtain the first feature C after spatial attention weighted processing, and the first feature C after spatial attention weighted processing is added to the first feature B to be processed to obtain a comprehensive feature D, and then the comprehensive feature D after convolution processing is added to the comprehensive feature D without convolution processing to obtain the second feature; The other path performs two PixelShuffle operations on the first feature B to be processed, restores the spatial size and number of feature channels of the image, and upsamples and adds the features of the results of the two PixelShuffle operations to obtain the third feature; The third feature is added to the second feature to obtain the final reconstructed spectral image.

4. The method for reconstructing spectral images based on a feature fusion network structure according to claim 3, characterized in that: The channel attention weighted processing specifically includes the following steps: Global average pooling, fully connected network, ReLU activation, fully connected network and Sigmoid activation are performed in sequence to generate channel attention weights. The weight vector of the channel attention weights is element-wise multiplied with the first feature B to be processed to obtain the first feature after channel attention weighted processing.

5. The method for reconstructing spectral images based on a feature fusion network structure according to claim 3, characterized in that: The spatial attention weighted processing specifically includes the following steps: The first feature after the channel attention weighted processing is subjected to global pooling and maximum pooling respectively to obtain the first feature of the global pooling and the first feature of the maximum pooling respectively. The first feature of the global pooling and the first feature of the maximum pooling are added to obtain a comprehensive spatial feature representation. The comprehensive spatial feature representation is sequentially subjected to 3×3 convolution and Sigmoid activation to generate the spatial attention weight. The weight vector of the spatial attention weight is element-wise multiplied with the first feature after the channel attention weighted processing to obtain the first feature C after the spatial attention weighted processing.

6. The method for reconstructing spectral images based on a feature fusion network structure according to claim 3, characterized in that: The convolution processing of the comprehensive feature D specifically includes 3×3 convolution and 1×1 convolution processing in sequence.

7. A system for reconstructing spectral images based on a feature fusion network structure, characterized in that: The method for reconstructing a spectral image based on a feature fusion network structure as described in claim 3 specifically includes the following modules: The encoding module is used to extract spectral image features through 3×3 convolution to obtain image features , based on a feature fusion network structure method, the first feature A is obtained; the first feature A is subjected to two PixelUnshuffle operations, the spatial size of the image is reduced without losing information, the number of feature channels is increased, and the first feature B to be processed is obtained; The visual attention module includes a channel attention module, a spatial attention module and a tail processing module, wherein the channel attention module is used to perform channel attention weighted processing on the first feature B to be processed to obtain a first feature after the channel attention weighted processing; the spatial attention module is used to perform spatial attention weighted processing on the first feature after the channel attention weighted processing to obtain a first feature C after the spatial attention weighted processing; the tail processing module is used to add the first feature C after the spatial attention weighted processing to the first feature B to be processed to obtain a comprehensive feature D, and then add the comprehensive feature D after convolution processing to the comprehensive feature D that has not been convolution processed to obtain a second feature; The decoding module is used to perform two PixelShuffle operations on the first feature B to be processed, restore the spatial size and the number of feature channels of the image, upsample and add the features of the results of the two PixelShuffle operations to obtain a third feature; add the third feature to the second feature to obtain the final reconstructed spectral image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for reconstructing a spectral image based on a feature fusion network structure as described in any one of claims 3 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for reconstructing a spectral image based on a feature fusion network structure as described in any one of claims 3 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for reconstructing a spectral image based on a feature fusion network structure as described in any one of claims 3 to 6 is implemented.

Images