High-resolution multispectral video imaging method and device

Abstract

The invention discloses a high-resolution multispectral video imaging method and a high-resolution multispectral video imaging device. The high-resolution multispectral video imaging method comprises the following steps: acquiring an embedded low-resolution multispectral image # imgabs2 # and an embedded high-resolution panchromatic image # imgabs3 # from a low-resolution multispectral image # imgabs0 # and a high-resolution panchromatic image # imgabs1 #; performing # imgabs6 grade spectrum and spatial difference analysis on the # imgabs4 # by combining the # imgabs5 # to obtain an output characteristic # imgabs7 #; the # imgabs8 # is combined with the # imgabs9 # to carry out # imgabs10 # level space and spectrum difference analysis so as to extract an output feature # imgabs11 #; and fusing the characteristics # imgabs12 # and # imgabs13 # to obtain an initial fusion characteristic # imgabs14 #, and decoding by using a decoder to reconstruct a final high-resolution multispectral image. The invention aims to realize the acquisition of the high-resolution multispectral video by fusing the panchromatic video and the low-spatial-resolution multispectral video obtained by different sensors in the same scene, and improve the detection efficiency and the image quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a high-resolution multi-spectral video imaging method and device. Background Art

[0002] Multispectral images contain rich spectral information, so they have been widely used in classification, target detection, environmental monitoring, agricultural monitoring and other fields. However, due to the limitation of hardware technology, it is difficult for a single sensor to achieve high spectral and high spatial resolution at the same time. Therefore, in multispectral images, in order to obtain spectral information, spatial resolution often needs to be sacrificed. Existing spectral imaging technology has constraints between spatial resolution and spectral resolution, and it is challenging to directly obtain multispectral images with high spatial resolution, which reduces the practical application value of multispectral images. In order to solve the above problems, one feasible way is to design advanced sensors, but this involves technical difficulties and expensive costs; another feasible way is to fuse images obtained by different sensors to achieve information complementarity, thereby effectively improving detection efficiency and improving image quality. This fusion method provides a feasible solution to overcome the limitations of multispectral images, so that the spatial resolution of multispectral images is improved while maintaining spectral information. However, how to fuse images obtained by different sensors to achieve information complementarity, thereby effectively improving detection efficiency and improving image quality, has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a high-resolution multispectral video imaging method and device are provided. The present invention aims to acquire high-resolution multispectral video by fusing full-color video and low-spatial-resolution multispectral video obtained by different sensors of the same scene, thereby improving detection efficiency and image quality.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A high-resolution multispectral video imaging method includes inputting a low-resolution multispectral image in a multispectral image video and a high-resolution panchromatic image in a panchromatic image video into a pre-trained multispectral image fusion network to reconstruct a final high-resolution multispectral image, and the multispectral image fusion network reconstructs the final high-resolution multispectral image including the following steps: S101, low-resolution multispectral images Input embedding layer to get embedded low-resolution multispectral image ; High-resolution full-color image Input the embedding layer to get the embedded high-resolution full-color image ; S102, embed high-resolution full-color images By combining low-resolution multispectral images conduct The spectral and spatial differences are analyzed in stages to extract spectral prior experience and spatial prior experience, and the output features are output by downsampling when each stage analyzes the spectral and spatial differences, and the output features are obtained by downsampling at the last stage. ; Embed high-resolution full-color images By combining low-resolution multispectral images conduct The spatial and spectral differences are analyzed at each level to extract spectral prior experience and spatial prior experience, and the output features are output by downsampling when analyzing the spectral and spatial differences at each level, and the output features are obtained by downsampling at the last level. ; S103, the feature and Features Fusion gets the initial fusion features ; S104, the initial fusion features The spectral prior experience and spatial prior experience are combined and the decoder is used to decode and reconstruct the final high-resolution multispectral image.

[0005] Optionally, in step S102, the embedded high-resolution full-color image By combining low-resolution multispectral images Multi-level analysis of spectral and spatial differences to extract spectral priors and spatial priors is performed through the spectral difference analysis module Implementation; The spectrum difference analysis module It includes the first linear normalization layer LN, the spectral low-rank cross attention module connected in sequence , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, and the output of the first linear normalization layer LN is also used as the input of the first superposition module, and the output of the functional layer is also used as the input of the second superposition module; the spectral low-rank cross attention module The function expression of the functional layer is: , , In the above formula, For the spectral low-rank cross attention module The output features, Indicates stacking operation, is the number of attention heads, is the result of the cross attention of the jth attention head, are learnable parameters, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer; is the value of the jth attention head, is the attention score of the jth attention head, and: , In the above formula, is the softmax activation function, is the key obtained by linear mapping of the input feature, The query is obtained by linear mapping of the input features. The value of the input features after linear mapping is ,in is the high-resolution full-color image embedded in the previous stage, For low-resolution multispectral images The image obtained by resizing, is the width of the image, is the height of the image, is the number of channels of the image, is the number of stages, , and are learnable parameters.

[0006] Optionally, in step S102, the embedded high-resolution full-color image By combining low-resolution multispectral images Multi-level analysis of spatial and spectral differences to extract spectral priors and spatial priors is performed through the spatial difference analysis module Implementation; the spatial difference analysis module It includes the first linear normalization layer LN and the spatial low-rank cross attention module connected in sequence. , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, and the output of the first linear normalization layer LN is also used as the input of the first superposition module, and the output of the functional layer is also used as the input of the second superposition module; the spatial low-rank cross attention module The function expression is: , In the above formula, For the spatial low-rank cross attention module The output features, Indicates stacking operation, is the i-th channel and Features of spatial cross-attention output, for and Features of spatial cross-attention output, This is the spatial difference analysis module for the current stage The output result is, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer. Indicates the first of N encoding stages The encoding stage, represents the number of spectral bands of the image; and:

[0007]

[0008] In the above formula, for and The output of spatial criss-cross attention, for and The output of spatial cross attention, and is a learnable parameter, is a hyperparameter used to represent the number of subspace bases, For and The attention score of the j-th head obtained in for and The attention score of the j-th head obtained in is the subspace of features extracted from low-resolution multispectral images by the previous network, is the feature subspace of the embedded high-resolution full-color image, is the spatial coefficient of the feature extracted by the previous network from the low-resolution multispectral image, is the spatial coefficient of the feature of the embedded high-resolution panchromatic image, where: , , , , In the above formula, is the feature of the i-th channel extracted from the low-resolution multispectral image by the previous network, is the feature of the i-th channel of the embedded high-resolution full-color image, For the previous stage The module outputs the result, , , are learnable parameters.

[0009] Optionally, in step S103, the feature and Features Fusion gets the initial fusion features The function expression is: , In the above formula, represents the bottleneck structure module, Indicates a stacking operation.

[0010] Optionally, the bottleneck structure module consists of a convolution module with a convolution kernel size of 1×1 and a fusion module guided by a spectral-spatial prior The spectral-spatial prior guided fusion module It includes the first linear normalization layer LN, the spectral-spatial prior-guided self-attention module connected in sequence , a first stacking module, a second linear normalization layer LN, a feed-forward network FFN and a second stacking module, and the spectral-spatial prior-guided self-attention module The function expression is: , In the above formula, Self-attention module guided by spectral-spatial priors The output features of Indicates stacking operation, is the number of attention heads, is the result of the cross attention of the jth attention head, are learnable parameters, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer; is the spatial spectrum prior experience, and: , In the above formula, A fusion module guided by the n-th level spectral-spatial prior using an attention mechanism Input features The extracted value value, and Represent the embedded low-resolution multispectral images and embedded high-resolution full-color images spectral and spatial priors extracted from the

[0011] Optionally, in step S104, the initial fusion feature Combining spectral prior experience and spatial prior experience, the decoder is used to decode and reconstruct the final high-resolution multispectral image, including: S201, the initial fusion feature pass Fusion module guided by spectral-spatial priors Combining spectral prior experience and spatial prior experience decoding to obtain decoding features ; and each level of spectral-spatial prior-guided fusion module The input end includes a channel upsampling module and stacking operations, and each level of spectral-spatial prior-guided fusion module The function expression of the output feature is: , In the above formula, is the nth level spectral-spatial prior guided fusion module The output features of represents the n-th level spectral-spatial prior guided fusion module , is the nth level spectral-spatial prior guided fusion module The input features of and Represent the embedded low-resolution multispectral images and embedded high-resolution full-color images The spectral priors and spatial priors extracted from , and there are: , , In the above formula, Indicates stacking operation, Indicates the nth level channel upsampling module The output features of Indicates the n+1th level channel upsampling module The output features of Indicates the nth level channel upsampling module , For the N-n+1 stage The output of the module, ; S202, reconstruct the final high-resolution multispectral image according to the following formula: , In the above formula, For the final high-resolution multispectral image, is the convolution operation, Indicates stacking operation, is the low-resolution multispectral image embedded for the first time, This is the first high-resolution full-color image embedded.

[0012] In addition, the present invention also provides a high-resolution multi-spectral video imaging device, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the high-resolution multi-spectral video imaging method.

[0013] Optionally, the microprocessor is also connected to an optical and sensing module, which includes an objective lens, a beam splitter prism, an eyepiece, a full-color imaging sensor and a multi-spectral coding sensor. The eyepiece includes a first eyepiece and a second eyepiece. The light entering from the outside passes through the objective lens and then enters the beam splitter prism and is divided into two beams, one beam enters the full-color imaging sensor through the first eyepiece, and the other beam enters the multi-spectral coding sensor through the second eyepiece. The full-color imaging sensor and the multi-spectral coding sensor are respectively connected to the microprocessor.

[0014] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the high-resolution multi-spectral video imaging method through a processor.

[0015] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the high-resolution multi-spectral video imaging method through a processor.

[0016] Compared with the prior art, the present invention mainly has the following advantages: 1. The high-resolution multispectral video imaging method of the present invention comprises converting a low-resolution multispectral image and high-resolution full-color images Get embedded low-resolution multispectral images and high-resolution full-color images ;Will By combining conduct Level analysis of spectral and spatial differences to obtain output features ;Will By combining conduct Analyze spatial and spectral differences to extract output features ; The feature and Features Fusion gets the initial fusion features , and then the initial fusion features The decoder is used to decode and reconstruct the final high-resolution multispectral image. The high-resolution multispectral video imaging method of the present invention achieves the acquisition of high-resolution multispectral video by fusing the full-color video and the multispectral video with low spatial resolution obtained by different sensors of the same scene. It can fuse the images obtained by different sensors to achieve information complementarity, thereby effectively improving the detection efficiency and improving the image quality.

[0017] 2. The high-resolution multispectral video imaging method of the present invention is applicable to the fusion of various types of panchromatic images and multispectral image data and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the network structure of the multispectral image fusion network in an embodiment of the present invention.

[0019] Figure 2 The spectrum difference analysis module in the embodiment of the present invention Schematic diagram of the network structure.

[0020] Figure 3 Schematic diagram of the network structure of the feedforward network FFN in an embodiment of the present invention.

[0021] Figure 4 The spatial difference analysis module in the embodiment of the present invention Schematic diagram of the network structure.

[0022] Figure 5 It is a spatial low-rank cross attention module in the embodiment of the present invention. Schematic diagram of the network structure.

[0023] Figure 6 It is a fusion module guided by a priori spectrum-space in the embodiment of the present invention. Schematic diagram of the network structure.

[0024] Figure 7 The self-attention module guided by spectral-spatial prior in the embodiment of the present invention Schematic diagram of the network structure.

[0025] Figure 8 Spectral angle error diagram of different methods in the embodiments of the present invention.

[0026] Fig. 9 Spectral angle error diagram of the reconstructed scene using different methods in the embodiments of the present invention.

[0027] Fig.10 Spectral angle error diagrams reconstructed by different methods in the embodiments of the present invention.

[0028] Fig.11This is a spectral angle error diagram of HR-HSI degraded HSI predicted by different fusion methods in the embodiments of the present invention.

[0029] Fig.12 1 and 2 are the classification results before and after fusion in the embodiment of the present invention.

[0030] Fig.13 Schematic diagram of the optical and sensor module structure of the high-resolution multi-spectral video imaging device in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] like Figure 1 As shown, this embodiment provides a high-resolution multispectral video imaging method, including inputting a low-resolution multispectral image in a multispectral image video and a high-resolution panchromatic image in a panchromatic image video into a pre-trained multispectral image fusion network to reconstruct a final high-resolution multispectral image, and the multispectral image fusion network reconstructs the final high-resolution multispectral image including the following steps: S101, low-resolution multispectral images Input embedding layer to get embedded low-resolution multispectral image ; High-resolution full-color image Input the embedding layer to get the embedded high-resolution full-color image ; S102, embed high-resolution full-color images By combining low-resolution multispectral images conduct The spectral and spatial differences are analyzed in stages to extract spectral prior experience and spatial prior experience, and the output features are output by downsampling when each stage analyzes the spectral and spatial differences, and the output features are obtained by downsampling at the last stage. ; Embed high-resolution full-color images By combining low-resolution multispectral images conduct The spatial and spectral differences are analyzed at each level to extract spectral prior experience and spatial prior experience, and the output features are output by downsampling when analyzing the spectral and spatial differences at each level, and the output features are obtained by downsampling at the last level. ; S103, the feature and Features Fusion gets the initial fusion features ; S104, the initial fusion features The spectral prior experience and spatial prior experience are combined and the decoder is used to decode and reconstruct the final high-resolution multispectral image.

[0032] like Figure 1As shown, the low-resolution multispectral image in this embodiment is the image frame in the low-resolution multispectral image (LR-MSI) video frame, and the high-resolution panchromatic image The image frame in the high-resolution panchromatic image (HR-PAN) video frame can also be used as an independent image instead of the image frame in the video frame. The principle is similar, so it will not be repeated below. and high-resolution panchromatic image (HR-PAN) Input the embedding layer separately to obtain the embedded low-resolution multispectral image , embedded high-resolution full-color image .in is a low-resolution multispectral image. For high-resolution full-color images, and represents the number of spectral bands of the image, and Represents the spatial scale information of the image. and represents the function of the embedding layer, where " " is much less than, for example, a value less than 1 / 10 means much less than. The function expression of the embedding layer is: , , The embedding layer functions in this embodiment are composed of N pairs of bilinear interpolation layers and 3×3 convolutional layers, where N is the total order of the encoding stage. Indicates the first of N encoding stages The encoding stage, .

[0033] In this embodiment, the high-resolution full-color image is embedded in step S102. By combining low-resolution multispectral images Multi-level analysis of spectral and spatial differences to extract spectral priors and spatial priors is performed through the spectral difference analysis module realized; such as Figure 2 As shown, the spectrum difference analysis module in this embodiment It includes the first linear normalization layer LN, the spectral low-rank cross attention module connected in sequence , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, and the output of the first linear normalization layer LN is also used as the input of the first superposition module, and the output of the functional layer is also used as the input of the second superposition module; the spectral low-rank cross attention module The function expression of the functional layer is: , , In the above formula, For the spectral low-rank cross attention module The output features, Indicates stacking operation, is the number of attention heads, is the result of the cross attention of the jth attention head, are learnable parameters, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer; is the value of the jth attention head, is the attention score of the jth attention head, and: , In the above formula, is the softmax activation function, is the key obtained by linear mapping of the input feature, The query is obtained by linear mapping of the input features. The value of the input features after linear mapping is ,in is the high-resolution full-color image embedded in the previous stage, For low-resolution multispectral images The image obtained by resizing, is the width of the image, is the height of the image, is the number of channels of the image, is the number of stages, , and are learnable parameters.

[0034] like Figure 3 As shown in FIG. 1 , the feed-forward network FFN is composed of a 1×1 convolutional layer, a LeakReLU activation function, a 1×1 convolutional layer, a LeakReLU activation function, and a 1×1 convolutional layer connected in sequence.

[0035] In this embodiment, the high-resolution full-color image is embedded in step S102. By combining low-resolution multispectral images Multi-level analysis of spatial and spectral differences to extract spectral and spatial priors is performed through the spatial difference analysis module realized; such as Figure 4 As shown, the spatial difference analysis module in this embodiment It includes the first linear normalization layer LN and the spatial low-rank cross attention module connected in sequence. , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, and the output of the first linear normalization layer LN is also used as the input of the first superposition module, and the output of the functional layer is also used as the input of the second superposition module; Figure 5 As shown, the spatial low-rank cross attention module The function expression is: , In the above formula, For the spatial low-rank cross attention module The output features, Indicates stacking operation, is the i-th channel and Features of spatial cross-attention output, for and Features of spatial cross-attention output, This is the spatial difference analysis module for the current stage The output result is, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer. Indicates the first of N encoding stages The encoding stage, represents the number of spectral bands of the image; and:

[0036]

[0037] In the above formula, for and The output of spatial criss-cross attention, for and The output of spatial cross attention, and is a learnable parameter, is a hyperparameter used to represent the number of subspace bases, For and The attention score of the j-th head obtained in for and The attention score of the j-th head obtained in is the subspace of features extracted from low-resolution multispectral images by the previous network, is the feature subspace of the embedded high-resolution full-color image, is the spatial coefficient of the feature extracted by the previous network from the low-resolution multispectral image, is the spatial coefficient of the feature of the embedded high-resolution panchromatic image, where: , , , , In the above formula, is the feature of the i-th channel extracted from the low-resolution multispectral image by the previous network, is the feature of the i-th channel of the embedded high-resolution full-color image, For the previous stage The module outputs the result, , , are learnable parameters.

[0038] The encoder output is input to and In the module ( Modules, The modules are all Transformer structures composed of attention modules and feed-forward networks (FFN). Analyze the spectral and spatial differences between multispectral images and panchromatic images, and extract the spectral prior experience and spatial prior experience. and For the Stages and The output corresponding to the module, and For the previous stage and The module outputs the result, when hour, .

[0039] , , exist In the module, The purpose of this module is to analyze the differences between PAN and HR-MSI. In the attention module, the embedded low-resolution multispectral image Size changed to After that, the features extracted from MSI by the previous network are firstly linearly mapped to obtain query (Q), key (K) and value (V). , and is the learnable parameter in the linear projection layer. , and Split into J heads respectively, and then calculate the similarity score of each head: , , , , in Representative Attention score of each head , (·) T Represents a transpose operation.

[0040] On this basis, Representative The value of the header , the result of cross attention is calculated as : , , in, Representative Head . yes Output of the module. are learnable parameters. is the position embedding function, which consists of two 3×3 convolutional layers and GELU activation. In the module, is from the embedded image What is obtained can be regarded as spectral prior experience. and is from Calculated, which contains information about PAN. Since MSI contains real spectral information, The module enables the model to measure the spectral differences between PAN and MSI by obtaining spectral priors from MSI.

[0041] exist In the module, in order to ensure that the model can capture spatial global information while reducing computational cost, we introduced matrix decomposition in the attention calculation process. As shown in formula (11), we transform the two-dimensional matrix Decomposition into subspace coefficients and the spatial coefficient We use deep learning methods to implement matrix factorization and define four parameter matrices to achieve the decomposition of the MSI features extracted from the previous network and embed them into PAN: , , , , , in , , are learnable parameters. and Characteristics The subspace and subspace coefficients of . and Characteristics The subspace and subspace coefficients of . It is a hyperparameter that indicates the number of subspace bases. When it is set to 1, using deep learning to implement matrix decomposition can make the calculation faster and more flexible, and also enhance the expressiveness of the model.

[0042] The attention score of the jth head from and The attention score of the jth head obtained in is from and In this module, Q S and Q A Respectively from and Calculated. S ,V S ] and [K A ,V A ] respectively from and Next, connect all the heads as shown below: , , in, and yes[ , ]and[ , ]The result obtained by cross attention calculation, and is a learnable parameter. Then, and Multiply and embed the position information into the final attention result As shown in the following formula: , in, and The positional embedding functions in the module are the same.

[0043] Finally, the outputs of the two modules are input into the downsampling module as shown below: , , in, and Both are single 4×4 convolutional layers, which can achieve spatial downsampling and channel upsampling with a sampling factor of 2. Then the final output of the two branches needs to be and Perform initial fusion. In step S103 of this embodiment, the feature and Features Fusion gets the initial fusion features The function expression is: , In the above formula, represents the bottleneck structure module, Represents a stacking operation (along the channel dimension and point-by-point convolutional layers). The bottleneck structure module in this embodiment consists of a convolution module with a convolution kernel size of 1×1 and a fusion module guided by a spectral-spatial prior. composition.

[0044] like Figure 6 As shown, the spectral-spatial prior guided fusion module It includes the first linear normalization layer LN, the spectral-spatial prior-guided self-attention module connected in sequence , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, such as Figure 7 As shown, the spectral-spatial prior guided self-attention module The function expression is: , In the above formula, Self-attention module guided by spectral-spatial priors The output features of Indicates stacking operation, is the number of attention heads, is the result of the cross attention of the jth attention head, are learnable parameters, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer; is the spatial spectrum prior experience, and: , In the above formula, A fusion module guided by the n-th level spectral-spatial prior using an attention mechanism Input features The extracted value value, and Represent the embedded low-resolution multispectral images and embedded high-resolution full-color images The spectral prior experience and spatial prior experience (represented as Spe and Spa respectively in the figure) extracted from the

[0045] In step S104 of this embodiment, the initial fusion feature Combining spectral prior experience and spatial prior experience, the decoder is used to decode and reconstruct the final high-resolution multispectral image, including: S201, the initial fusion feature pass Fusion module guided by spectral-spatial priors Combining spectral prior experience and spatial prior experience decoding to obtain decoding features ; and each level of spectral-spatial prior-guided fusion module The input end includes a channel upsampling module and stacking operations, and each level of spectral-spatial prior-guided fusion module The function expression of the output feature is: , In the above formula, is the nth level spectral-spatial prior guided fusion module The output features of represents the n-th level spectral-spatial prior guided fusion module , is the nth level spectral-spatial prior guided fusion module The input features of and Represent the embedded low-resolution multispectral images and embedded high-resolution full-color images The spectral priors and spatial priors extracted from , and there are: , , In the above formula, Indicates stacking operation, Indicates the nth level channel upsampling module The output features of Indicates the n+1th level channel upsampling module The output features of Indicates the nth level channel upsampling module , For the N-n+1 stage The output of the module, ; S202, reconstruct the final high-resolution multispectral image according to the following formula: , In the above formula, For the final high-resolution multispectral image, is the convolution operation, Indicates stacking operation, is the low-resolution multispectral image embedded for the first time, This is the first high-resolution full-color image embedded.

[0046] In order to enhance the transmission of shallow feature information, the corresponding encoding features before downsampling are also fused in this embodiment, wherein the upsampling module ( ) is used to perform Pixel-Shuffle (PS). In this embodiment, a 5×5 convolutional layer is specifically used to implement channel upsampling of PS. It is indivual Function of the module (a Transformer structure consisting of an attention block and a feed-forward network (FFN)) in the decoder . and Respectively indicate from and The prior experience extracted from hour, , is the corresponding output.

[0047] , , , exist In the module, according to the fusion features By calculation , , and , and then introduced the spatial spectrum prior experience ,in and Respectively indicate from and Prior experience extracted from.

[0048] , Next, we introduce the embedded multispectral image and embedded full-color image , which includes spectral prior experience and spatial prior experience, and the spatial spectrum prior guide mechanism drives The module pays more attention to the spatial spectrum priors hidden in PAN and MSI: , , , in and are the spatial mask and the spectral mask. (·)and (·) represents the function of the spatial attention module and the spectral attention module. Finally, self-attention is performed The result of calculation is: , , Finally, a global residual connection is introduced to reconstruct the final high-resolution multispectral image: , In the above formula, For the final high-resolution multispectral image, is the convolution operation (3×3 convolution layer), Indicates stacking operation, is the low-resolution multispectral image embedded for the first time, This is the first high-resolution full-color image embedded.

[0049] In order to verify the high-resolution multispectral video imaging method of this embodiment, the high-resolution multispectral video imaging method of this embodiment (Ours) is compared with the NSSR, LTTR, SSRnet, Fformer and SSTFUNET models used in the existing methods. For details of NSSR, please refer to the literature: Dong W, Fu F, Shi G, et al. Hyperspectral image super-resolution via non-negative structured sparse representation[J]. IEEE Transactions on Image Processing, 2016, 25(5): 2337-2352.; for details of LTTR, please refer to the literature: DianR, Li S, Fang L. Learning a low tensor-train rank representation for hyperspectral image super-resolution[J]. IEEE transactions on neural networksand learning systems, 2019, 30(9): 2672-2683.; for details of SSRnet, please refer to the literature: Zhang X, HuangW, Wang Q, et al. SSR-NET: Spatial–spectral reconstruction network for hyperspectral and multispectral image fusion[J]. IEEE Transactions onGeoscience and Remote Sensing, 2020, 59(7): 5953-5965.; For details on Fformer, see the literature: Hu JF, Huang TZ, Deng LJ, et al. Fusformer: A transformer-based fusion network for hyperspectral image super-resolution[J]. IEEE Geoscience and RemoteSensing Letters, 2022, 19: 1-5.; For details of SSTF-Unet, please see the literature: Liu H, Feng C, Dian R, etal.Sstf-unet: Spatial–spectral transformer-based u-net for high-resolution hyperspectral image acquisition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023. The datasets used are CAVE, KAIST and ICVL datasets. Some of the test results obtained are as follows. Figure 8 , Fig. 9 , Fig.10 , Fig.11 as well as Fig.12 As shown, Figure 8 Spectral angle error diagrams of the reconstructed jelly bean MS (1st row), real apple (2nd row) and fake pepper MS (3rd row) of three test images of the CAVE dataset using different methods in this embodiment. Fig. 9 Figure 2 is a spectral angle error diagram of the reconstructed scene 21 reflectance (1st row), scene 22 reflectance (2nd row) and scene 26 reflectance (3rd row) of three test images of the KAIST dataset using different methods in this embodiment. Fig.10 Spectral angle error diagrams of the reconstructions of three test images of the ICVL dataset, bgu_0403-1511 (1st row), papper_0503-1330 (2nd row), and sami_0331-1019 (3rd row), using different methods in this embodiment. Fig.11 (a) to (g) are spectral angle error diagrams of HR-HSI degraded HSI predicted by different fusion methods on the Gaofen-5 and Gaofen-1 datasets in this embodiment. Fig.12 ] are the classification results before and after fusion in this embodiment, where (a) is the classification result of LR-HSI, (b) is the predicted classification result of HR-HSI, and (c) is the reference classification result. Figure 8 to Figure 12 The effectiveness of the high-resolution multi-spectral video imaging method of this embodiment is verified. In addition, the statistical results obtained in this embodiment are shown in Tables 1 to 6.

[0050] Table 1: Quantitative indicators of the test methods on the CAVE dataset, KAIST dataset, and ICVL dataset

[0051] The best results in Table 1 are marked in bold. As shown in Table 1, our LRTN achieves the best results in three indicators: PSNR (peak signal-to-noise ratio), ERGAS (error root mean square / signal-to-noise ratio), and SAM (spectral angle mapper). In the UIQI (global image quality index) test on the CAVE dataset, the result of LRTN ranks second. For the KAIST and ICVL datasets, LRTN shows the best performance in all evaluation indicators. In addition, the performance of SSTF-Unet is also very close to LRTN. From the table, we can see that deep learning-based methods have obvious advantages over traditional methods. Among all the methods, although SSRnet has a clear advantage in speed, its performance in fusion quality is limited and lower than other deep learning-based methods. In general, our method LRTN achieves the best fusion results while maintaining a faster processing speed, and has superior fusion quality compared to other methods.

[0052] Table 2: External validation experimental results of deep learning-based methods

[0053] The best results are marked in bold in Table 2. As shown in Table 2, all methods were trained on the KAIST dataset and tested on the CAVE dataset. From the results, we can see that although the SAM (spectral angle mapper) of our method LRTN is slightly lower than that of SSTF-Unet in this case, LRTN shows the best results in all other evaluation indicators. This shows that LRTN has the best performance in most evaluation indicators.

[0054] Table 3: Running times of the methods tested on the Gaoscore-5 and Gaoscore-1 datasets, with the best results marked in bold

[0055] The best results are marked in bold in Table 3. As shown in Table 3, our method LRTN ranks second among all the comparison methods, second only to SSRnet, while SSRnet maintains the fastest reconstruction speed among these methods. In general, although our method is slightly inferior to SSRnet in fusion speed, it achieves the highest level in fusion quality.

[0056] Table 4: Ablation experiments of SEDL, SADL and SSGF modules on CAVE dataset and KAIST dataset

[0057] The best results are marked in bold in Table 4. As shown in Table 4, when we try to replace the key modules with residual blocks or replace the prior knowledge in SSGF, all evaluation indicators on both datasets decrease. For example, for the CAVE dataset, the PSNR (peak signal-to-noise ratio) decreases by 0.22dB, 0.37dB, 0.90dB, and 0.16dB, respectively. In addition, although the “No-MF” model is similar to LRTN in performance, its parameters increase by 71.26Mb. Considering the computational complexity analysis, we can draw the conclusion that making full use of the intrinsic properties of HSI can reduce the parameters and computational complexity of the model while ensuring the fusion accuracy.

[0058] Table 5: Ablation experiments of hyperparameters N and L on CAVE dataset and KAIST dataset

[0059] The best results in Table 5 are marked in bold. As shown in Table 5, we can see that on the CAVE and KAIST datasets, when N = 2, LRTN has the best performance. We set the hyperparameter L in three different ways, namely L = 1, 2, and 3. The results show that the performance of LRTN has almost no change in these three cases, which shows that our method is not sensitive to the setting of the hyperparameter L and has good robustness. When selecting the hyperparameter N, we chose N = 2 based on the best performance results. As for the hyperparameter L, although the performance of LRTN does not change much when L = 1, 2, and 3, in order to minimize the computational cost, we chose L = 1 to reduce the amount of computation while maintaining performance.

[0060] Table 6: Classification results of LR-HSI and predicted HSI, the best results are marked in bold

[0061] The best results are marked in bold in Table 6. As shown in Table 6, most of the classification results of high resolution (HR-HSI) are better than those of low resolution (LR-HSI). At the same time, the average accuracy of the predicted high resolution (HR-HSI) is 6.8% higher than that of the low resolution (LR-HSI), which shows that the high resolution multispectral video imaging method of this embodiment has higher accuracy in processing high resolution data.

[0062] In summary, the high-resolution multispectral video imaging method of this embodiment includes converting the low-resolution multispectral image and high-resolution full-color images Get embedded low-resolution multispectral images and high-resolution full-color images ;Will By combining conduct Level analysis of spectral and spatial differences to obtain output features ;Will By combining conduct Analyze spatial and spectral differences to extract output features ; The feature and Features Fusion gets the initial fusion features , and then the initial fusion features The decoder is used to decode and reconstruct the final high-resolution multispectral image. The high-resolution multispectral video imaging method of this embodiment achieves the acquisition of high-resolution multispectral video by fusing the panchromatic video and low-spatial-resolution multispectral video obtained by different sensors of the same scene, and can achieve the fusion of images obtained by different sensors to achieve information complementarity, thereby effectively improving the detection efficiency and image quality. The high-resolution multispectral video imaging method of this embodiment is suitable for the fusion of various types of panchromatic images and multispectral image data, and has a wide range of applications.

[0063] In addition, this embodiment also provides a high-resolution multi-spectral video imaging device, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the high-resolution multi-spectral video imaging method described above in this embodiment. The microprocessor in this embodiment is also connected to an optical and sensor module. Fig.13As shown, the optical and sensing module in this embodiment includes an objective lens, a beam splitter prism, an eyepiece, a full-color imaging sensor (PAN imaging sensor) and a multi-spectral encoding sensor (MSI encoding sensor). The eyepiece includes a first eyepiece and a second eyepiece. The light entering from the outside passes through the objective lens and enters the beam splitter prism and is divided into two beams. One beam enters the full-color imaging sensor through the first eyepiece, and the other beam enters the multi-spectral encoding sensor through the second eyepiece. The full-color imaging sensor and the multi-spectral encoding sensor are respectively connected to a microprocessor. In this embodiment, a data acquisition system is established, and a method combining software and hardware is proposed to achieve effective acquisition of high-resolution multi-spectral video. In this embodiment, an innovative multi-spectral imaging model is established. By using the proposed method, spectral priors and spatial priors are fully utilized to guide feature extraction and fusion. This method promotes the model to pay more attention to the missing information of different images and more efficiently extract feature information from full-color images and multi-spectral images. In this embodiment, a low-rank Transformer model is designed, and the Transformer module is combined with the matrix decomposition theory, which significantly reduces the model parameter quantity and computational cost, and designs a spatial spectral prior-guided fusion module in the feature fusion stage, which improves the spatial and spectral fidelity of the image and improves the performance of network fusion. In this embodiment, it is suitable for the fusion of various types of panchromatic images and multispectral image data, and has a wide range of applications. In this embodiment, a high-resolution multispectral video imaging device is disclosed, which includes a hardware system for high-resolution multispectral computational imaging, mainly composed of a spectroscopic prism, a panchromatic (PAN) image imaging sensor, and a multispectral image (MSI) encoding sensor. By adopting the MSI and panchromatic image fusion imaging method, the present invention fuses a multispectral image with high spectral resolution and low spatial resolution with a panchromatic image with high spatial resolution and low spectral resolution, thereby realizing the acquisition of high-resolution multispectral video. The scheme of this embodiment has many advantages, such as high temporal and spatial resolution, low cost, high signal-to-noise ratio and high computational efficiency, and breaks through the problem of mutual constraints of existing multispectral cameras in spatial and spectral resolution. Through this device, high-quality multispectral video images can be acquired more effectively, providing more superior solutions for various application scenarios.

[0064] In addition, the present embodiment further provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the high-resolution multi-spectral video imaging method through a processor. In addition, the present embodiment further provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the high-resolution multi-spectral video imaging method through a processor.

[0065] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A high-resolution multispectral video imaging method, characterized in that: The method includes inputting a low-resolution multispectral image in a multispectral image video and a high-resolution panchromatic image in a panchromatic image video into a pre-trained multispectral image fusion network to reconstruct a final high-resolution multispectral image, and the multispectral image fusion network reconstructs the final high-resolution multispectral image including the following steps: S101, low-resolution multispectral images Input embedding layer to get embedded low-resolution multispectral image ; High-resolution full-color image Input the embedding layer to get the embedded high-resolution full-color image ; S102, embed high-resolution full-color images By combining low-resolution multispectral images conduct The spectral and spatial differences are analyzed in stages to extract spectral prior experience and spatial prior experience, and the output features are output by downsampling when each stage analyzes the spectral and spatial differences, and the output features are obtained by downsampling at the last stage. ; Embed high-resolution full-color images By combining low-resolution multispectral images conduct The spatial and spectral differences are analyzed at each level to extract spectral prior experience and spatial prior experience, and the output features are output by downsampling when analyzing the spectral and spatial differences at each level, and the output features are obtained by downsampling at the last level. ; S103, the feature and Features Fusion gets the initial fusion features ; S104, the initial fusion features The spectral prior experience and spatial prior experience are combined and the decoder is used to decode and reconstruct the final high-resolution multispectral image.

2. The high-resolution multispectral video imaging method according to claim 1, characterized in that: In step S102, the high-resolution full-color image to be embedded By combining low-resolution multispectral images Multi-level analysis of spectral and spatial differences to extract spectral priors and spatial priors is performed through the spectral difference analysis module Implementation; The spectrum difference analysis module It includes the first linear normalization layer LN, the spectral low-rank cross attention module connected in sequence , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, and the output of the first linear normalization layer LN is also used as the input of the first superposition module, and the output of the functional layer is also used as the input of the second superposition module; the spectral low-rank cross attention module The function expression of the functional layer is: ， ， In the above formula, For the spectral low-rank cross attention module The output features, Indicates stacking operation, is the number of attention heads, is the result of the cross attention of the jth attention head, are learnable parameters, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer; is the value of the jth attention head, is the attention score of the jth attention head, and: ， In the above formula, is the softmax activation function, is the key obtained by linear mapping of the input feature, The query is obtained by linear mapping of the input features. The value of the input features after linear mapping is ,in is the high-resolution full-color image embedded in the previous stage, For low-resolution multispectral images The image obtained by resizing, is the width of the image, is the height of the image, is the number of channels of the image, is the number of stages, , and are learnable parameters.

3. The high-resolution multispectral video imaging method according to claim 1, characterized in that: In step S102, the high-resolution full-color image to be embedded By combining low-resolution multispectral images Multi-level analysis of spatial and spectral differences to extract spectral priors and spatial priors is performed through the spatial difference analysis module Implementation; the spatial difference analysis module It includes the first linear normalization layer LN and the spatial low-rank cross attention module connected in sequence. , a first superposition module, a second linear normalization layer LN, a feedforward network FFN and a second superposition module, and the output of the first linear normalization layer LN is also used as the input of the first superposition module, and the output of the functional layer is also used as the input of the second superposition module; the spatial low-rank cross attention module The function expression is: ， In the above formula, For the spatial low-rank cross attention module The output features, Indicates stacking operation, is the i-th channel and Features of spatial cross-attention output, for and Features of spatial cross-attention output, This is the spatial difference analysis module for the current stage The output result is, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer. Indicates the first of N encoding stages The encoding stage, represents the number of spectral bands of the image; and: In the above formula, for and The output of spatial criss-cross attention, for and The output of spatial cross attention, and is a learnable parameter, is a hyperparameter used to represent the number of subspace bases, For and The attention score of the j-th head obtained in for and The attention score of the j-th head obtained in is the subspace of features extracted from low-resolution multispectral images by the previous network, is the feature subspace of the embedded high-resolution full-color image, is the spatial coefficient of the feature extracted by the previous network from the low-resolution multispectral image, is the spatial coefficient of the feature of the embedded high-resolution panchromatic image, where: ， ， ， ， In the above formula, is the feature of the i-th channel extracted from the low-resolution multispectral image by the previous network, is the feature of the i-th channel of the embedded high-resolution full-color image, For the previous stage The module outputs the result, , , are learnable parameters.

4. The high-resolution multispectral video imaging method according to claim 1, characterized in that: In step S103, the feature and Features Fusion gets the initial fusion features The function expression is: ， In the above formula, represents the bottleneck structure module, Indicates a stacking operation.

5. The high-resolution multi-spectral video imaging method according to claim 4, characterized in that: The bottleneck structure module consists of a convolution module with a convolution kernel size of 1×1 and a fusion module guided by spectral-spatial priors. The spectral-spatial prior guided fusion module It includes the first linear normalization layer LN, the spectral-spatial prior-guided self-attention module connected in sequence , a first stacking module, a second linear normalization layer LN, a feed-forward network FFN and a second stacking module, and the spectral-spatial prior-guided self-attention module The function expression is: ， In the above formula, Self-attention module guided by spectral-spatial priors The output features of Indicates stacking operation, is the number of attention heads, is the result of the cross attention of the jth attention head, are learnable parameters, is the position embedding function, the position embedding function It consists of two 3×3 convolutional layers and a GELU activation layer; is the spatial spectrum prior experience, and: ， In the above formula, A fusion module guided by the n-th level spectral-spatial prior using an attention mechanism Input features The extracted value value, and Represent the embedded low-resolution multispectral images and embedded high-resolution full-color images spectral and spatial priors extracted from the 6. The high-resolution multi-spectral video imaging method according to claim 5, characterized in that: In step S104, the initial fusion feature Combining spectral prior experience and spatial prior experience, the decoder is used to decode and reconstruct the final high-resolution multispectral image, including: S201, the initial fusion feature pass Fusion module guided by spectral-spatial priors Combining spectral prior experience and spatial prior experience decoding to obtain decoding features ; and each level of spectral-spatial prior-guided fusion module The input end includes a channel upsampling module and stacking operations, and each level of spectral-spatial prior-guided fusion module The function expression of the output feature is: ， In the above formula, is the nth level spectral-spatial prior guided fusion module The output features of represents the n-th level spectral-spatial prior guided fusion module , is the nth level spectral-spatial prior guided fusion module The input features of and Represent the embedded low-resolution multispectral images and embedded high-resolution full-color images The spectral priors and spatial priors extracted from , and there are: ， ， In the above formula, Indicates stacking operation, Indicates the nth level channel upsampling module The output features of Indicates the n+1th level channel upsampling module The output features of Indicates the nth level channel upsampling module , For the N-n+1 stage The output of the module, ; S202, reconstruct the final high-resolution multispectral image according to the following formula: ， In the above formula, For the final high-resolution multispectral image, is the convolution operation, Indicates stacking operation, is the low-resolution multispectral image embedded for the first time, This is the first high-resolution full-color image embedded.

7. A high-resolution multispectral video imaging device, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the high-resolution multi-spectral video imaging method according to any one of claims 1 to 6.

8. The high-resolution multi-spectral video imaging device according to claim 7, characterized in that: The microprocessor is also connected to an optical and sensing module, which includes an objective lens, a beam splitter prism, an eyepiece, a full-color imaging sensor and a multi-spectral coding sensor. The eyepiece includes a first eyepiece and a second eyepiece. The light entering from the outside passes through the objective lens and then enters the beam splitter prism and is divided into two beams, one beam enters the full-color imaging sensor through the first eyepiece, and the other beam enters the multi-spectral coding sensor through the second eyepiece. The full-color imaging sensor and the multi-spectral coding sensor are respectively connected to the microprocessor.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the high-resolution multi-spectral video imaging method according to any one of claims 1 to 6 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the high-resolution multi-spectral video imaging method according to any one of claims 1 to 6 through a processor.

Images