A multispectral image reconstruction system

Abstract

The application relates to a multispectral image reconstruction system, and belongs to the field of image processing. The multispectral image reconstruction system comprises an intensity difference mask convolution demosaicking module, a multi-level convolution feature extraction module, a multi-dimensional attention feature fusion module and a CNN residual reconstruction module; an intensity difference mosaic image is obtained by subtracting a multispectral mosaic image from a single-channel intensity image; the same convolution weight is assigned to pixels belonging to the same MSFA position by adopting a position mask convolution method, different convolution weights are assigned to pixels belonging to different MSFA positions, and a convolution feature map is obtained; multi-level convolution fusion features are obtained based on the convolution feature map; high-frequency convolution features and cross-attention fusion features are extracted based on the multi-level convolution fusion features, and multi-dimensional adaptive fusion features are obtained; residual features are extracted from the multi-dimensional adaptive fusion features, and a multispectral image is obtained. The application can comprehensively recover the missing information of the multispectral mosaic image and improve the reconstruction accuracy.

Description

Technical Field

[0001] This application relates to the field of image signal processing technology, and in particular to a multispectral image reconstruction system. Background Technology

[0002] Snapshot multispectral imaging technology periodically and repeatedly arranges a multi-spectral filter array (MSFA) composed of filters with different center wavelengths on an image sensor. This allows for the simultaneous acquisition of two-dimensional spatial information and one-dimensional spectral information of the target in a single image, meeting the application requirements of low cost, miniaturization, and real-time performance. However, its imaging principle means that each pixel area on the sensor plane can only obtain spatial information of the imaged scene under a certain center band, missing imaging results under other bands. To obtain a multispectral image with complete spatial information across multiple spectral bands, it is necessary to reconstruct the original single multispectral mosaic image, restoring the missing band components at each pixel location. This crucial process is called multispectral image de-mosaicing. Figure 1 As shown.

[0003] Current multispectral image demosaic techniques primarily rely on CNN neural networks (such as ResNet-based CNN algorithms). A single multispectral mosaic image is input into a deep neural network to learn and obtain its corresponding complete multispectral band image. However, the characteristic of ordinary convolution operations sharing weight parameters across all pixels in the image plane easily leads to spectral aliasing when applied to multispectral mosaic images where each pixel has different spectral band information. This results in inaccurate multispectral image information recovery. Furthermore, existing techniques cannot fully utilize long-distance spatial and spectral band correlations, losing low-frequency information from the image, making it difficult to improve the accuracy of multispectral image reconstruction. Summary of the Invention

[0004] The purpose of this application is to provide a multispectral image reconstruction system that can more comprehensively recover the missing information of multispectral mosaic images from multiple dimensions such as space, spectral bands and channels, and achieve high-quality restoration and reconstruction of multispectral mosaic images with low computational complexity, thereby effectively improving the reconstruction accuracy of multispectral images.

[0005] This application provides a multispectral image reconstruction system, which includes: an intensity difference mask convolution demosaic module, a multi-level convolution feature extraction module, a multi-dimensional attention feature fusion module, and a CNN residual reconstruction module. The intensity difference mask convolution demosaic module is configured to: subtract the multispectral mosaic image from the single-channel intensity image to obtain an intensity difference mosaic image; and use positional mask convolution to assign the same convolution weight to pixels belonging to the same MSFA position in the intensity difference mosaic image, and to assign the same convolution weight to pixels belonging to different MSFA positions in the intensity difference mosaic image. Different convolution weights are assigned to pixels at position FA to obtain convolutional feature maps; the multi-level convolutional feature extraction module is configured to process the convolutional feature maps to obtain multi-level convolutional fusion features; the multi-dimensional attention feature fusion module is configured to extract high-frequency convolutional features and cross-attention fusion features based on multi-level convolutional fusion features to obtain multi-dimensional adaptive fusion features; the CNN residual reconstruction module is configured to extract residual features from the multi-dimensional adaptive fusion features, and use a non-linear activation function to process the residual features and single-channel intensity images to obtain multispectral images.

[0006] In an optional implementation of this application, the position mask convolution method includes: using a general standard convolution method, using a different set of standard convolution weights to perform preliminary feature extraction with global parameter sharing on the intensity difference mosaic image, each set of standard convolution weights is only responsible for the reconstruction of one MSFA relative pixel position feature on the intensity difference mosaic image, and the MSFA relative pixel position features handled by the a sets of standard convolution weights are different, resulting in a preliminary feature maps with errors in some pixel position information; using a binary position mask method to process the a preliminary feature maps, so that the i-th preliminary feature map retains only the accurate feature information belonging to the i-th relative pixel position of MSFA, while the inaccurate feature information of other relative pixel positions becomes 0, resulting in a sparse convolution feature maps; adding the a sparse convolution feature maps pixel by pixel to obtain a convolution feature map; where the value of i ranges from 0 to a-1 natural numbers.

[0007] In an optional implementation of this application, a preliminary feature map is obtained using formula (1), as follows:

[0008] in, Preliminary feature map; Represents the convolutional layer function; Standard convolution weights; A mosaic image based on intensity difference; i The value of is a natural number from 0 to a-1.

[0009] In an optional implementation of this application, the sparse convolution feature map is obtained using formula (2), which is as follows:

[0010] in, This is a sparse convolution feature map; This is the initial feature map; 'e' indicates pixel-by-pixel multiplication. , m i A binary location mask of the same size as the multispectral mosaic image, with a value of 1 or 0 for each pixel location. R Represents the set of real numbers. m , n These represent the length and width of the multispectral mosaic image, respectively.

[0011] In an optional implementation of this application, multi-level convolutional fusion features are obtained using formula (3), which is as follows:

[0012] in, This is a multi-level convolutional feature extraction module; This represents a multi-level convolutional fusion feature. Convolutional feature maps; multi-level convolutional feature extraction module Includes multiple convolutional cascaded submodules ; Continuous convolutional cascaded submodules Configured to repeatedly execute formulas (4) and (5) in sequence, the output result is... Formula (4) is as follows:

[0013] Formula (5) is as follows:

[0014] in, x For input tensors; x 1 is to obtain the tensor; Represents the convolutional layer function; Concatenate ( ) represents the concatenation function of vectors along the feature dimension.

[0015] In optional embodiments of this application, the multi-dimensional attention feature fusion module includes a multi-dimensional adaptive feature extraction component; the multi-dimensional adaptive feature extraction component includes a convolutional high-frequency enhancement component, a cross-attention feature extraction component, and a feature fusion component based on the Transformer architecture; the convolutional high-frequency enhancement component is configured to: extract local high-frequency convolutional features of the multispectral mosaic image, divide the input tensor into two equal parts in the feature dimension, and then feed them into convolutional layers and pooling layers for processing to obtain the corresponding feature quantities; re-concatenate the two processed feature quantities in the feature dimension and perform convolution processing, and then add the input tensor with residual connections to obtain high-frequency convolutional features; the cross-attention feature extraction component is configured to: adaptively extract adaptive features of the input tensor from the spatial dimension, spectral dimension, and feature channel dimension, and fuse the adaptive features to obtain cross-attention fusion features; the feature fusion component based on the Transformer architecture is configured to: fuse the high-frequency convolutional features and the cross-attention fusion features to obtain multi-dimensional adaptive fusion features.

[0016] In optional embodiments of this application, the adaptive features include spatial adaptive features, spectral adaptive features, and channel adaptive features; wherein, the steps for obtaining spatial adaptive features include: dividing the input tensor into two paths, using extraction operations to extract salient features at different levels of the input tensor for each corresponding path, generating a spatial adaptive mask for the input tensor; and then multiplying the spatial adaptive mask with the input tensor to obtain the corrected spatial adaptive features; the extraction operations include sequential convolution and post-extraction processing; the post-extraction is sequential local average pooling and convolution, or the post-extraction processing is local max pooling; The steps for obtaining spectral adaptive features include: dividing the input tensor into three paths, performing convolution and depthwise convolution on the first two paths respectively, and then performing autocorrelation on the first two paths to obtain a spectral correlation matrix mask; then multiplying the spectral correlation matrix mask with the input tensor in the spectral dimension to obtain the corrected spectral adaptive features. The steps for obtaining channel adaptive features include: compressing the input tensor into a one-dimensional channel vector through global average pooling, and using a nonlinear function to obtain a feature channel-dimensional adaptive mask; then multiplying the feature channel-dimensional adaptive mask with the input tensor in the feature channel dimension to obtain the corrected channel adaptive features.

[0017] In an optional implementation of this application, the feature fusion component based on the Transformer architecture is configured to: use high-frequency convolutional features as query information, perform cross-correlation and feature extraction optimization on the cross-attention fusion features, and then obtain multi-dimensional adaptive fusion features through a forward feedback structure.

[0018] In an optional implementation of this application, the feature fusion component based on the Transformer architecture is configured to: transpose the high-frequency convolutional features and the cross-attention fusion features and then calculate the cross-correlation matrix in the channel dimension.

[0019] In an optional embodiment of this application, a multispectral image is obtained using formula (6), which is as follows:

[0020] in, Multispectral image; It is a non-linear activation function; Residual characteristics; This is a single-channel intensity image.

[0021] The multispectral image reconstruction system provided in this application has at least the following beneficial effects: The multispectral image reconstruction system provided in this application employs positional mask convolution, which allows pixels belonging to the same MSFA relative spatial position (i.e., the same spectral band) in a multispectral mosaic image to share weights, while assigning different weights to pixels belonging to different MSFA relative spatial positions (i.e., different spectral bands). This enables dynamic weight allocation based on pixel spatial position without increasing the model's computational load, avoiding the spatial-spectral information aliasing problem caused by directly using standard convolution operations to share global parameters across all spatially located pixels in a multispectral mosaic image. This effectively improves the accuracy of image reconstruction and restoration across different spectral bands. Furthermore, by combining CNN local high-frequency feature extraction and Transformer global long-range feature extraction, the system more comprehensively recovers the missing information of the multispectral mosaic image from multiple dimensions—spatial, spectral, and channel—further enhancing the reconstruction accuracy. Therefore, this application achieves high-quality restoration and reconstruction of multispectral mosaic images with relatively low computational complexity. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the demosaic process for a multispectral mosaic image based on 4×4 MSFA.

[0024] Figure 2 This is a general framework diagram of the multispectral image reconstruction system of this application.

[0025] Figure 3 This is a schematic diagram of the dynamic weight allocation in the multispectral image reconstruction system of this application.

[0026] Figure 4 This is an architecture diagram of the multidimensional adaptive feature extraction component in the multispectral image reconstruction system of this application.

[0027] Figure 5 This is an architecture diagram of the cross-attention feature extraction component in the multispectral image reconstruction system of this application.

[0028] Figure 6 This is an architecture diagram of the feature fusion component based on the Transformer architecture in the multispectral image reconstruction system of this application. Detailed Implementation

[0029] This application provides a multispectral image reconstruction system. Figure 2 For an overall framework diagram of the multispectral image reconstruction system of this application, please refer to [link / reference]. Figure 2 The multispectral image reconstruction system includes: an intensity difference mask convolution demosaic module, a multi-level convolution feature extraction module, a multi-dimensional attention feature fusion module, and a CNN residual reconstruction module.

[0030] The intensity difference mask convolution demosaic module is configured to: subtract the multispectral mosaic image from the single-channel intensity image to obtain an intensity difference mosaic image; and use position mask convolution to assign the same convolution weight to pixels belonging to the same MSFA position on the intensity difference mosaic image, and assign different convolution weights to pixels belonging to different MSFA positions on the intensity difference mosaic image to obtain a convolution feature map.

[0031] The multi-level convolutional feature extraction module is configured to process the convolutional feature map to obtain multi-level convolutional fusion features.

[0032] The multi-dimensional attention feature fusion module is configured to extract high-frequency convolution features and cross-attention fusion features based on multi-level convolution fusion features to obtain multi-dimensional adaptive fusion features.

[0033] The CNN residual reconstruction module is configured to: extract residual features from multi-dimensional adaptive fusion features, and process the residual features and single-channel intensity images using a non-linear activation function to obtain a multispectral image.

[0034] The multispectral image reconstruction system provided in this application employs positional mask convolution, which allows pixels belonging to the same MSFA relative spatial position (i.e., the same spectral band) in a multispectral mosaic image to share weights, while assigning different weights to pixels belonging to different MSFA relative spatial positions (i.e., different spectral bands). This enables dynamic weight allocation based on pixel spatial position without increasing the model's computational load, avoiding the spatial-spectral information aliasing problem caused by directly using standard convolution operations to share global parameters across all spatially located pixels in a multispectral mosaic image. This effectively improves the accuracy of image reconstruction and restoration across different spectral bands. Furthermore, by combining CNN local high-frequency feature extraction and Transformer global long-range feature extraction, the system more comprehensively recovers the missing information of the multispectral mosaic image from multiple dimensions—spatial, spectral, and channel—further enhancing the reconstruction accuracy. Therefore, this application achieves high-quality restoration and reconstruction of multispectral mosaic images with relatively low computational complexity.

[0035] The following is a detailed description of the multispectral image reconstruction system.

[0036] (a) Intensity difference mask convolution demosaic module: Since the intensity difference mosaic image obtained by subtracting the multispectral mosaic image from the single-channel intensity image is modulated by MSFA, the spectral band distribution at different pixel positions on the intensity difference mosaic image plane also changes periodically. If the traditional global parameter sharing standard convolution operation is used to extract features from all pixels on the intensity difference mosaic image plane, it will introduce a large error.

[0037] Based on the above problems, in this application, as Figure 3 As shown, the intensity difference mask convolution demosaic module is configured as follows: after subtracting the multispectral mosaic image from the single-channel intensity image, an intensity difference mosaic image is obtained; using position mask convolution, pixels belonging to the same MSFA position on the intensity difference mosaic image are assigned the same convolution weight, and pixels belonging to different MSFA positions on the intensity difference mosaic image are assigned different convolution weights to obtain a convolution feature map.

[0038] This application employs a positional mask convolution method, which allows pixels belonging to the same MSFA relative spatial position (i.e., the same spectral band) in a multispectral mosaic image to share weights, and pixels belonging to different MSFA relative spatial positions (i.e., different spectral bands) in a multispectral mosaic image to be assigned different weights. This enables dynamic weight allocation based on pixel spatial position without increasing the computational load of the model, avoiding the problem of spatial-spectral information aliasing caused by directly using standard convolution operations to share global parameters of all spatial position pixels in a multispectral mosaic image, and effectively improving the accuracy of image reconstruction and restoration under different spectral bands.

[0039] In this application, as shown in formula (7), a single multispectral mosaic image is used. The single-channel intensity image obtained after local intensity averaging By subtracting the values, we obtain a mosaic image with varying intensity differences. Formula (7) is as follows:

[0040] in m , n These are the length and width of the multispectral mosaic image, respectively. l This indicates that the number of channels is 1.

[0041] Specifically, the location mask convolution method includes: using a general standard convolution method, using a different set of standard convolution weights to perform preliminary feature extraction with global parameter sharing on the intensity difference mosaic image. Each set of standard convolution weights is only responsible for the reconstruction of one MSFA relative pixel position feature on the intensity difference mosaic image, and the MSFA relative pixel position features handled by each of the a sets of standard convolution weights are different, resulting in a preliminary feature maps with errors in some pixel position information; using a binary location mask method, processing the a preliminary feature maps so that the i-th preliminary feature map only retains the accurate feature information belonging to the i-th relative pixel position of the MSFA, while the inaccurate feature information of other relative pixel positions becomes 0, resulting in a sparse convolution feature maps; adding the a sparse convolution feature maps pixel by pixel to obtain the convolution feature map; where the value of i ranges from 0 to a-1 natural numbers.

[0042] By combining general standard convolution with two-dimensional position masks, dynamic weight allocation based on pixel spatial location can be achieved without increasing the computational load of the model, thus reducing computational complexity.

[0043] Furthermore, a preliminary feature map is obtained using formula (1), as follows:

[0044] in, Preliminary feature map; Represents the convolutional layer function; Standard convolution weights; A mosaic image based on intensity difference; i The value of is a natural number from 0 to a-1.

[0045] The sparse convolution feature map is obtained using formula (2), which is as follows:

[0046] in, This is a sparse convolution feature map; This is the initial feature map; 'e' indicates pixel-by-pixel multiplication. , m i A binary location mask of the same size as the multispectral mosaic image, with a value of 1 or 0 for each pixel location. R Represents the set of real numbers. m , n These represent the length and width of the multispectral mosaic image, respectively.

[0047] The convolutional feature map is obtained using formula (8), which is as follows:

[0048] in, For convolutional feature maps; It is a sparse feature map; i The value of is a natural number from 0 to a-1.

[0049] Taking a 4×4 multispectral filter array (MSFA) as an example, consider that there are 16 different relative spatial positions on the 4×4 MSFA, i.e., distributed across 16 different spectral bands. Therefore, for intensity difference mosaic images... There are 16 types of pixels with different relative positions in the MSFA, requiring 16 different sets of convolution weights. Feature extraction is performed on pixels at different relative positions.

[0050] Specifically, according to the above formula (1), the general standard convolution uses 16 different sets of standard convolution weights. To each Preliminary feature extraction is performed directly using global parameter sharing. Each set of convolutional weights only applies to the intensity difference mosaic image. The first MSFA method is responsible for reconstructing the relative pixel position features. The feature reconstruction results obtained by the convolution weights for the remaining pixel positions are inaccurate and will be filtered out by subsequent binary position masking operations. Thus, a preliminary feature map is obtained with errors in the partial pixel position information. .

[0051] To remove the 16 preliminary feature maps obtained from the above-mentioned general standard convolution operation Inaccurate feature information at some pixel locations in the middle section is retained only. The feature information belonging to the i-th relative pixel position of MSFA is used to operate on the 16 feature maps obtained by applying a binary position mask according to the above formula (2). Among them, A binary location mask of the same size as the original multispectral mosaic image is used, where each pixel location has a value of 1 (belonging to the i-th relative pixel location on the MSFA) or 0 (other locations), and 'e' represents pixel-wise multiplication. In this way, each preliminary feature map... Only the accurate feature information belonging to the i-th relative pixel position of MSFA is retained, while the inaccurate feature information of other pixel positions is reduced to 0, thus obtaining 16 sparse convolution feature maps. According to formula (9), these 16 sparse convolutional feature maps are added pixel by pixel to finally obtain a convolutional feature map with complete and accurate spatial information. Formula (9) is as follows:

[0052] (II) Multi-level convolutional feature extraction module: In this application, the multi-level convolutional feature extraction module is configured to process the convolutional feature map to obtain multi-level convolutional fusion features.

[0053] Multi-level convolutional feature extraction module Includes multiple convolutional cascaded submodules As an example, the convolutional feature map The input consists of 8 consecutive convolutional cascaded sub-modules. The hierarchical convolutional feature extraction module consists of This yields multi-level convolutional fusion features.

[0054] In this application, formula (3) is used to obtain multi-level convolutional fusion features, as follows:

[0055] in, This is a multi-level convolutional feature extraction module; This represents a multi-level convolutional fusion feature. This is a convolutional feature map.

[0056] Continuous convolutional cascaded submodules Configured to repeatedly execute formulas (4) and (5) sequentially, the output result is... Formula (4) is as follows:

[0057] Formula (5) is as follows:

[0058] in, x For input tensors; x 1 is to obtain the tensor; Represents the convolutional layer function; Concatenate ( ) represents the concatenation function of vectors along the feature dimension.

[0059] Performing a standard convolution operation on the input tensor x yields x 1. Then convert the input tensor x With obtaining tensor x 1. Concatenate feature dimensions.

[0060] (III) Multi-dimensional attention feature fusion module: Because multispectral mosaic images contain both partially missing spatial and spectral information, traditional image feature extraction methods based solely on convolution operations can only capture local features of the input data, making it difficult to fully utilize the long-range correlations of the data. Attention networks based on the Transformer architecture can effectively capture autocorrelation and cross-correlation between data, but their computational complexity is high, especially when the data dimension N is high, exhibiting an O(n) computational complexity. 2 It increases, but ignores high-frequency information in the image itself.

[0061] Based on the above problems, this application combines the advantages of CNN and Transformer to design a multi-dimensional attention feature fusion module. The multi-dimensional attention feature fusion module is configured to extract high-frequency convolution features and cross-attention fusion features based on multi-level convolution fusion features to obtain multi-dimensional adaptive fusion features.

[0062] The multi-dimensional attention feature fusion module includes a multi-dimensional adaptive feature extraction component; as an example, the multi-dimensional attention feature fusion module includes two cascaded multi-dimensional adaptive feature extraction components.

[0063] like Figure 4 As shown, the multi-dimensional adaptive feature extraction component includes a convolutional high-frequency enhancement component, a cross-attention feature extraction component, and a feature fusion component based on the Transformer architecture.

[0064] In this application, the high-frequency convolutional enhancement component is configured to: extract local high-frequency convolutional features from a multispectral mosaic image; divide the input tensor into two equal parts along the feature dimension; then feed them into a convolutional layer and a pooling layer for processing to obtain the corresponding feature quantities; re-concatenate the two processed feature quantities along the feature dimension and perform convolution processing; then add the input tensor using a residual connection to obtain the high-frequency convolutional features.

[0065] In this application, as Figure 5 As shown, the cross-attention feature extraction component is configured to: adaptively extract adaptive features of the input tensor from the spatial dimension, spectral dimension and feature channel dimension, and fuse the adaptive features to obtain the cross-attention fused features.

[0066] Furthermore, adaptive features include spatial adaptive features, spectral adaptive features, and channel adaptive features.

[0067] The steps for obtaining spatial adaptive features include: dividing the input tensor into two paths, extracting salient features at different levels of the input tensor for each path using extraction operations, and generating a spatial adaptive mask for the input tensor; then multiplying the spatial adaptive mask with the input tensor to obtain the corrected spatial adaptive features; the extraction operations include sequential convolution and post-extraction processing; the post-extraction is sequential local average pooling and convolution, or the post-extraction processing is local max pooling.

[0068] The steps for obtaining spectral adaptive features include: dividing the input tensor into three paths, performing convolution and depthwise convolution on the first two paths respectively, and then performing autocorrelation on the first two paths in the spectral dimension to obtain a spectral correlation matrix mask; then multiplying the spectral correlation matrix mask with the input tensor in the spectral dimension to obtain the corrected spectral adaptive features.

[0069] The steps for obtaining channel adaptive features include: compressing the input tensor into a one-dimensional channel vector through global average pooling, obtaining the feature channel-dimensional adaptive mask using a nonlinear function; and then multiplying the feature channel-dimensional adaptive mask with the input tensor in the feature channel dimension to obtain the corrected channel adaptive features.

[0070] The spatial adaptive features, spectral adaptive features, and channel adaptive features are cascaded and fused together to obtain the cross-attention fusion features of the input tensor.

[0071] In this application, as Figure 6 As shown, the feature fusion component based on the Transformer architecture is configured to fuse high-frequency convolutional features and cross-attention fusion features to obtain multi-dimensional adaptive fusion features.

[0072] Multidimensional adaptive fusion features include local high-frequency information in the spatial dimension extracted by CNN and adaptive spatial, spectral and channel information extracted by cross-attention feature extraction components. This provides a more comprehensive and richer representation, which helps to better recover the missing information in multispectral mosaic images and improve the reconstruction accuracy of multispectral images.

[0073] Furthermore, the feature fusion component based on the Transformer architecture is configured to: use high-frequency convolutional features as query information, perform cross-correlation and feature extraction optimization on cross-attention fusion features, and then obtain multi-dimensional adaptive fusion features through a forward feedback structure.

[0074] Furthermore, the feature fusion component based on the Transformer architecture is configured to transpose the high-frequency convolutional features and the cross-attention fused features and then calculate the cross-correlation matrix in the channel dimension.

[0075] Specifically, the multi-dimensional attention feature fusion module The relevant formulas are as follows:

[0076]

[0077]

[0078] in, This represents a multi-dimensional adaptive feature extraction component; This represents a multi-level convolutional fusion feature. f 1 represents pass The features obtained later; f 2 indicates and f 1. Characteristics after addition; f RCRFG express f 2 through Features obtained later These are high-frequency convolutional features; Indicates the high-frequency enhancement component of convolution; x For input tensors; This is a cross-attention fusion feature; This represents the cross-attention feature extraction component; Multi-dimensional adaptive fusion features; This represents a feature fusion component based on the Transformer architecture; It is a spatial adaptive feature; For spatial adaptive feature extraction function; It is a spectral adaptive feature; For spectral adaptive feature extraction function; Channel-adaptive features; For feature channel dimension adaptive extraction function; This is a function for fusing multiple features.

[0079] (iv) CNN Residual Reconstruction Module In this application, the CNN residual reconstruction module is configured to: extract residual features from multi-dimensional adaptive fusion features, and process the residual features and single-channel intensity images using a nonlinear activation function to obtain a multispectral image.

[0080] Specifically, the multispectral image is obtained using formula (6), which is as follows:

[0081] in, Multispectral image; It is a non-linear activation function; Residual characteristics; This is a single-channel intensity image.

[0082] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below. Where specific conditions are not specified in the embodiments, conventional conditions or conditions recommended by the manufacturer shall apply.

[0083] Example This embodiment provides a multispectral image reconstruction system based on a 4×4 MSFA (i.e., 16 spectral bands) multispectral mosaic image restoration and reconstruction process. The specific steps are as follows: (1) Intensity difference mask convolution demosaic module A single multispectral mosaic image The single-channel intensity image obtained after local intensity averaging By subtracting the values, we obtain a mosaic image with varying intensity differences. :

[0084] in , These represent the length and width of the original multispectral image, respectively. 1 This indicates that the number of channels is 1.

[0085] The general standard convolution uses 16 different sets of standard convolution weights. To each Preliminary feature extraction is performed directly using global parameter sharing. Each set of convolutional weights only applies to the intensity difference mosaic image. The first MSFA method is responsible for reconstructing the relative pixel position features. The feature reconstruction results obtained by the convolution weights for the remaining pixel positions are inaccurate and will be filtered out by subsequent binary position masking operations. Thus, a preliminary feature map is obtained with errors in the partial pixel position information. :

[0086] To remove the 16 preliminary feature maps obtained from the above-mentioned general standard convolution operation Inaccurate feature information at some pixel locations in the middle section is retained only. The feature information belonging to the i-th relative pixel position of MSFA is used to operate on the resulting 16 feature maps using a binary position mask:

[0087] in, A binary location mask of the same size as the original multispectral mosaic image is used, where each pixel location has a value of 1 (belonging to the i-th relative pixel location on the MSFA) or 0 (other locations), and 'e' represents pixel-wise multiplication. In this way, each preliminary feature map... Only the accurate feature information belonging to the i-th relative pixel position of MSFA is retained, while the inaccurate feature information of other pixel positions is reduced to 0, thus obtaining 16 sparse convolution feature maps. By summing these 16 sparse convolutional feature maps pixel by pixel, a convolutional feature map with complete and accurate spatial information is finally obtained. :

[0088] (2) Multi-level convolutional feature extraction module Will Feed into 8 consecutive convolutional cascaded submodules Multi-level convolutional feature extraction module The process yields multi-level convolutional fusion features:

[0089] in For input tensor Perform standard convolution operations to obtain Then, concatenate the input tensor and the obtained tensor by their feature dimensions:

[0090] (3) Multi-dimensional attention feature fusion module Multi-dimensional attention feature fusion module Includes two cascaded multi-dimensional adaptive feature extraction components :

[0091] Multi-dimensional adaptive feature extraction component Includes high-frequency enhancement components for convolution. Cross-attention feature extraction component and feature fusion components based on Transformer architecture :

[0092] Among them, the high-frequency enhancement component of convolution Extracting local high-frequency convolutional features from a multispectral mosaic image will input a tensor. The feature vector is divided into two equal parts along the feature dimension, and then fed into convolutional and pooling layers for processing. The two processed feature vectors are then concatenated along the feature dimension and subjected to convolution. Finally, a residual connection is used to add the input tensor. Obtain the output of this module .

[0093] Cross-attention feature extraction component Adaptive extraction of input tensors from spatial dimension, spectral dimension and feature channel dimension Key features and their integration:

[0094] Among them, the spatial adaptive feature extraction function Input tensor The process is divided into two paths, which extract salient features at different levels of the tensor through convolution, local average pooling, convolution, local max pooling, feature summation, and residual connection operations, respectively, to generate the input tensor. The spatial adaptive mask is then combined with the input tensor. Multiplication yields the corrected spatial adaptive features. .

[0095] Spectral Adaptive Feature Extraction Function Input tensor The process is divided into three paths. After convolution and depthwise convolution, the first two paths perform autocorrelation on the spectral dimension to obtain a spectral correlation matrix mask, which is then compared with the input tensor. Multiplying by the spectral dimensions yields the corrected adaptive spectral features. .

[0096] Feature Channel Dimension Adaptive Extraction Function Input tensor The vector is compressed into a one-dimensional channel vector using global average pooling. An adaptive mask for the feature channel dimension is obtained using a nonlinear function, and then combined with the input tensor. Multiplying by the feature channel dimensions yields the corrected channel adaptive features. .

[0097] Finally, the spatial adaptive features will be used. Spectral adaptive features and channel adaptive features By cascading and adding the input tensors, we obtain the input tensor. Cross-attention fusion features .

[0098] The obtained high-frequency convolution features and cross-attention fusion features Feed into the feature fusion component based on the Transformer architecture To obtain multi-dimensional adaptive fusion features .

[0099] (4) CNN residual reconstruction module Residual feature extraction of multi-dimensional adaptive fusion features Using a nonlinear activation function For residual characteristics and single-channel intensity image Processing is performed to obtain a multispectral image. .

[0100]

[0101] Comparative Example 1 This comparative example provides a multispectral image reconstruction system. The difference between Comparative Example 1 and the embodiment is that: In this comparative example, a single multispectral mosaic image modulated by 4×4 (16 spectral bands) MSFA is used. It employs a deep convolutional network composed of multiple standard convolutional layers. A multispectral image with complete spatial information across 16 spectral channels was obtained through reconstruction. :

[0102] Comparative Example 2 This comparative example provides a multispectral image reconstruction system. The difference between Comparative Example 1 and the embodiment is that: In this comparative example, a single multispectral mosaic image modulated by 4×4 (16 spectral bands) MSFA is used. The initial feature map is obtained through a standard two-dimensional convolutional layer. :

[0103] Will Input is a deep convolutional network consisting of multiple standard 3D convolutional layers. To obtain deep 3D convolutional features :

[0104] The initial feature map is connected via residual connection. The depthwise 3D convolutional features obtained above The images are then summed and passed through a standard two-dimensional convolutional layer to obtain a fully reconstructed multispectral image with complete spatial information across 16 spectral channels. :

[0105] The quantitative evaluation results of the examples and comparative examples 1-2 are shown in Table 1.

[0106] Table 1 Quantitative Assessment Results

[0107] As can be seen from the comparison between the embodiments and comparative examples 1-2, comparative methods 1-2 all use the standard convolution operation with global parameter sharing for multispectral mosaic images whose spatial information is modulated by 4×4MSFA, and do not combine the Transformer architecture to extract long-range correlation features of the data, nor do they fully perform adaptive feature extraction and fusion of features in the spatial dimension, spectral dimension, and feature channel dimension, resulting in low reconstruction accuracy and easy generation of inaccurate information; while the reconstruction quality of the embodiments of this application is high, indicating that the multispectral image reconstruction system of the embodiments of this application can effectively improve the reconstruction accuracy of multispectral images.

[0108] In summary, the multispectral image reconstruction system provided in this application can achieve high-quality restoration and reconstruction of multispectral mosaic images with low computational complexity.

[0109] The embodiments described above are some, but not all, of the embodiments of this application. The detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A multispectral image reconstruction system, characterized in that, include: The module includes intensity difference mask convolution demosaicing module, multi-level convolution feature extraction module, multi-dimensional attention feature fusion module, and CNN residual reconstruction module; The intensity difference mask convolution demosaic module is configured to: subtract the multispectral mosaic image from the single-channel intensity image to obtain an intensity difference mosaic image; and use position mask convolution to assign the same convolution weight to pixels belonging to the same MSFA position in the intensity difference mosaic image, and assign different convolution weights to pixels belonging to different MSFA positions in the intensity difference mosaic image to obtain a convolution feature map. The multi-level convolutional feature extraction module is configured to process the convolutional feature map to obtain multi-level convolutional fusion features; The multi-dimensional attention feature fusion module is configured to: extract high-frequency convolution features and cross-attention fusion features based on the multi-level convolution fusion features to obtain multi-dimensional adaptive fusion features; The CNN residual reconstruction module is configured to: extract residual features from the multi-dimensional adaptive fusion features, and process the residual features and the single-channel intensity image using a nonlinear activation function to obtain a multispectral image.

2. The multispectral image reconstruction system according to claim 1, characterized in that, The location mask convolution method includes: Using a general standard convolution method, a different set of standard convolution weights are used to perform preliminary feature extraction with global parameter sharing on the intensity difference mosaic image. Each set of standard convolution weights is only responsible for the reconstruction of one MSFA relative pixel position feature on the intensity difference mosaic image, and the MSFA relative pixel position features that the a sets of standard convolution weights are responsible for are different, resulting in a preliminary feature map with errors in the position information of a partial pixels. A binary position mask is used to process the a preliminary feature maps, such that the i-th preliminary feature map retains only the accurate feature information of the i-th relative pixel position of the MSFA, while the inaccurate feature information of other relative pixel positions is reduced to 0, resulting in a sparse convolutional feature maps; the a sparse convolutional feature maps are then added pixel by pixel to obtain the convolutional feature map; where i is a natural number ranging from 0 to a-1.

3. The multispectral image reconstruction system according to claim 2, characterized in that, The preliminary feature map is obtained using formula (1), which is as follows: in, This is the preliminary feature map; Represents the convolutional layer function; The standard convolution weights; The intensity difference mosaic image; i The value of is a natural number from 0 to a-1.

4. The multispectral image reconstruction system according to claim 2, characterized in that, The sparse convolution feature map is obtained using formula (2), which is as follows: in, This refers to the sparse convolution feature map; This is the preliminary feature map; 'e' indicates pixel-by-pixel multiplication; , m i A binary location mask of the same size as the multispectral mosaic image, with a value of 1 or 0 for each pixel location. R Represents the set of real numbers. m , n These are the length and width of the multispectral mosaic image, respectively.

5. The multispectral image reconstruction system according to any one of claims 1 to 4, characterized in that, The multi-level convolutional fusion features are obtained using formula (3), which is as follows: in, This refers to the multi-level convolutional feature extraction module; This refers to the multi-level convolutional fusion feature; The convolutional feature map; The multi-level convolution feature extraction module Includes multiple convolutional cascaded submodules The continuous convolutional cascaded submodule Configured to repeatedly execute formulas (4) and (5) in sequence, the output result is... Formula (4) is as follows: Formula (5) is as follows: in, x For input tensors; x 1 is to obtain the tensor; Represents the convolutional layer function; Concatenate ( ) represents the concatenation function of vectors along the feature dimension.

6. The multispectral image reconstruction system according to any one of claims 1 to 4, characterized in that, The multi-dimensional attention feature fusion module includes a multi-dimensional adaptive feature extraction component; the multi-dimensional adaptive feature extraction component includes a convolutional high-frequency enhancement component, a cross-attention feature extraction component, and a feature fusion component based on the Transformer architecture; The high-frequency convolution enhancement component is configured to: extract local high-frequency convolution features from the multispectral mosaic image; divide the input tensor into two equal parts along the feature dimension; then feed them into a convolutional layer and a pooling layer for processing to obtain the corresponding feature values; re-concatenate the two processed feature values ​​along the feature dimension and perform convolution processing; then add the input tensor using a residual connection to obtain the high-frequency convolution features. The cross-attention feature extraction component is configured to: adaptively extract adaptive features of the input tensor from the spatial dimension, spectral dimension and feature channel dimension, and fuse the adaptive features to obtain the cross-attention fused features; The feature fusion component based on the Transformer architecture is configured to fuse the high-frequency convolutional features and the cross-attention fusion features to obtain the multi-dimensional adaptive fusion features.

7. The multispectral image reconstruction system according to claim 6, characterized in that, The adaptive features include spatial adaptive features, spectral adaptive features, and channel adaptive features; The steps for obtaining the spatial adaptive features include: dividing the input tensor into two paths, extracting salient features at different levels of the input tensor for each path using an extraction operation to generate a spatial adaptive mask for the input tensor; and then multiplying the spatial adaptive mask with the input tensor to obtain the corrected spatial adaptive features; the extraction operation includes sequential convolution and post-extraction processing; the post-extraction is sequential local average pooling and convolution, or the post-extraction processing is local max pooling; The steps for obtaining the spectral adaptive features include: dividing the input tensor into three paths, performing convolution and depthwise convolution on the first two paths respectively, and then performing autocorrelation on the first two paths to obtain a spectral correlation matrix mask; and then multiplying the spectral correlation matrix mask with the input tensor in the spectral dimension to obtain the corrected spectral adaptive features. The steps for obtaining the channel adaptive features include: compressing the input tensor into a one-dimensional channel vector through global average pooling, obtaining a feature channel-dimensional adaptive mask using a nonlinear function; and then multiplying the feature channel-dimensional adaptive mask with the input tensor in the feature channel dimension to obtain the corrected channel adaptive features.

8. The multispectral image reconstruction system according to claim 6, characterized in that, The feature fusion component based on the Transformer architecture is configured to: use the high-frequency convolutional features as query information, perform cross-correlation and feature extraction optimization on the cross-attention fusion features, and then obtain the multi-dimensional adaptive fusion features through a forward feedback structure.

9. The multispectral image reconstruction system according to claim 6, characterized in that, The feature fusion component based on the Transformer architecture is configured to: transpose the high-frequency convolutional features and the cross-attention fusion features and then calculate the cross-correlation matrix in the channel dimension.

10. The multispectral image reconstruction system according to any one of claims 1 to 4, characterized in that, The multispectral image is obtained using formula (6), which is as follows: in, The multispectral image; The nonlinear activation function is... The residual feature; This refers to the single-channel intensity image.

Images