Construction method and training method of metasurface hyperspectral reconstruction model based on cross-modal text enhancement and electronic device

Abstract

The application discloses a construction method and a training method of a metasurface hyperspectral reconstruction model based on cross-modal text enhancement and an electronic device, and relates to the field of metasurface hyperspectral reconstruction.The metasurface hyperspectral reconstruction model comprises an image encoder, a text encoder, a cross-modal alignment module, a cross-modal condition control module, an encoder level enhancement module and a hybrid space-spectrum decoder, realizes joint representation and collaborative reconstruction of a multispectral image collected by a metasurface and a text semantic prompt, effectively overcomes the semantic missing problem caused by the fact that an existing method only depends on the characteristics of an image, and provides a clear structure and a complete function framework for hyperspectral construction.

Description

Technical Field

[0001] This invention relates to the field of metasurface hyperspectral reconstruction, and in particular to a method for constructing, training, and an electronic device for a metasurface hyperspectral reconstruction model based on cross-modal text enhancement. Background Technology

[0002] Hyperspectral imaging technology, capable of simultaneously acquiring spatial and continuous, detailed spectral information of a target, has significant application value in fields such as material identification, biomedical detection, and environmental monitoring. Traditional hyperspectral systems rely on complex optical structures, resulting in large size and weight, which makes them difficult to meet the requirements of lightweight platforms. In recent years, miniature spectral imaging chips based on metasurfaces (such as real-time hyperspectral imaging chips) have achieved rapid and compact acquisition of multispectral images through micro- and nanostructure manipulation, significantly promoting the miniaturization and integration of hyperspectral imaging systems.

[0003] However, such metasurface chips typically output multispectral images with limited bands and low spectral resolution, still requiring subsequent computational reconstruction algorithms to map them to the target hyperspectral space. Existing reconstruction methods mostly employ general deep learning models, extracting features only from the image data itself, lacking explicit semantic modeling of the scene's spatial distribution characteristics and spectral variation patterns. This leads to deviations in the reconstruction results regarding the preservation of spatial details and consistency with spectral curves. Furthermore, existing fusion mechanisms often employ fixed weights or simple stitching strategies, making it difficult to achieve adaptive and selective fusion of cross-modal information, easily introducing information redundancy or semantic shifts, and affecting reconstruction accuracy.

[0004] While some methods attempt to introduce attention mechanisms to enhance feature representation, they often fail to distinguish the differentiated modeling needs of spatial and spectral dimensions, nor do they establish a dynamic relationship between external semantic cues and image features, resulting in limited joint spatial-spectral representation capabilities. Therefore, there is an urgent need for a hyperspectral reconstruction method that can combine spatial and spectral semantic priors and adaptively fuse features accordingly, in order to simultaneously ensure the spatial clarity and spectral authenticity of the reconstructed image. Summary of the Invention

[0005] To improve the spatial detail fidelity and spectral curve accuracy of hyperspectral reconstruction, this invention proposes a method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement, comprising: An image encoder is used to receive a multispectral image acquired by a metasurface and process the multispectral image through multiple cascaded convolutional layers to generate an image feature representation. A text encoder is used to receive spatial distribution cues, spectral change cues, or a combination of both corresponding to the multispectral image, and generate corresponding text semantic features; The cross-modal alignment module is used to calculate the cosine similarity between image feature representations and text semantic features to obtain a similarity score; The cross-modal conditional control module projects text semantic features to the same spatial-channel dimension as the image feature representation to obtain text projection features. Based on the text projection features, spatial guided features and spectral guided features are generated respectively through a spatial-spectral dual-path attention mechanism, and the two are weighted and fused using similarity scores to output cross-modal enhanced image features. The encoder-level enhancement module is used to perform multi-scale spatial spectrum modeling and cross-modal dynamic updating of the cross-modal enhanced image features, and output multiple level fusion features; Hybrid spatial spectrum decoder is used to construct a target hyperspectral feature map based on multi-level fused features.

[0006] Furthermore, the cross-modal condition control module includes: The text projection unit, consisting of multiple cascaded convolutional layers, is used to project the text semantic features to the same spatial-channel dimension as the image feature representation, thereby obtaining text projection features; A spatial attention unit is used to perform spatial dimension attention calculation on the image feature representation using the text projection features as a query, and generate spatial guidance features; A spectral attention unit is used to perform channel-dimensional attention calculations on the image feature representation using the text projection features as a query, thereby generating spectral guided features; The dynamic fusion unit performs a weighted summation of the spatial guidance features and the spectral guidance features based on the similarity score, and outputs the cross-modal enhanced image features.

[0007] Furthermore, the encoder-level enhancement module includes multiple sequentially cascaded transformation units, and a cross-modal update unit disposed between every two adjacent transformation units; The plurality of sequentially cascaded transformation units include, in order: Two basic transformation units are used to extract the local spatial structure of their input features and model the inter-channel spectral dependence. An enhancement transform unit is used to extract multi-scale spatial-spectral joint context information from its input features, and to construct a global context based on the information to generate attention weights, which are then used to adaptively modulate the input features of the enhancement transform unit. Four core transformation units, each of which is used to model the long-range dependencies between spectral channels of its input features and capture the structural consistency in the spatial dimension; The first transformation unit receives the cross-modal enhanced image features as input, while the remaining transformation units receive the output of the previous transformation unit after processing by the corresponding cross-modal update unit as input.

[0008] Furthermore, the cross-modal update unit is used to receive the output features of the previous transformation unit and the text semantic features, and align them; generate fusion weights based on the similarity scores, perform weighted fusion of the aligned output features and the aligned text semantic features based on the fusion weights, and use the fusion result as the input of the next transformation unit.

[0009] Furthermore, the multiple level fusion features output by the encoder level enhancement module include: The first-level fusion feature is the feature of the output of the second basic transformation unit after being processed by the cross-modal update unit set between it and the enhancement transformation unit; The second-level fusion feature is the feature of the output of the enhanced transform unit after being processed by the cross-modal update unit set between it and the first core transform unit; The third-level fusion feature is the feature of the output of the second core transformation unit after being processed by the cross-modal update unit set between it and the third core transformation unit; The fourth-level fusion feature is the output of the last core transformation unit.

[0010] Furthermore, the enhancement transformation unit specifically includes: The multi-scale convolution module is used to perform depthwise separable convolution operations of different sizes on its input features in parallel to capture spatial-spectral joint contextual information at the corresponding scale. The feature stitching module is used to stitch together the spatial-spectral joint context information at various scales along the channel dimension to generate a fusion feature with the number of channels superimposed while the spatial size remains unchanged. The global context extraction module is used to perform average pooling and max pooling on the fused features to obtain the first global statistical feature and the second global statistical feature. The attention weight generation module is used to fuse the first global statistical feature and the second global statistical feature to obtain the fused feature, and then perform multiple cascaded learnable linear transformations on the fused feature in sequence, followed by nonlinear normalization operation to generate attention weights with the same spatial-channel dimension as the input features of the multi-scale convolution module. The feature modulation module is used to multiply the attention weights element-wise with the input features of the multi-scale convolution module to obtain the modulation features, and add the modulation features to the input features of the multi-scale convolution module to obtain the output features of the enhancement transform unit.

[0011] Furthermore, the hybrid spatial spectrum decoder includes: The local feature processing module is used to enhance the local spatial structure and spectral channel correlation of the multiple layered fusion features through the feature enhancement layer, so as to obtain local enhanced features that correspond one-to-one with each layered fusion feature. The global feature modeling module is used to upsample the fused features at each level and the corresponding local enhancement features to a preset spatial resolution, spatially flatten each upsampled feature to obtain a corresponding feature sequence, and concatenate all feature sequences along the sequence dimension to form a global feature vector; linearly project the global feature vector to generate query features, key features, and value features respectively, and calculate attention weights based on the similarity between the query features and key features, and perform weighted aggregation of the value features through the attention weights to output the global fused features; The mapping module is used to map the global fused features into a target hyperspectral feature map through a convolutional layer.

[0012] This invention also proposes a training method for a metasurface hyperspectral reconstruction model based on cross-modal text enhancement as described above, comprising: Construct a text cue library, which includes spatial distribution cue describing the spatial distribution characteristics of multispectral images, and spectral change cue describing the spectral change characteristics; A multispectral image acquired from a metasurface, a corresponding text prompt for the multispectral image, and registered ground truth hyperspectral data are obtained to construct a triplet sample. The triplet sample is then preprocessed to form a training set. The text prompt is a spatial distribution prompt, a spectral change prompt, or a combination of both. The hyperspectral reconstruction model is trained end-to-end using the training set. In each training iteration, a multispectral image and its corresponding text prompt are input into the metasurface hyperspectral reconstruction model to obtain a target hyperspectral feature map. Channel adaptation is performed on the target hyperspectral feature map, and a joint loss function is calculated based on the adapted target hyperspectral feature map and the corresponding ground truth hyperspectral data to update the model parameters.

[0013] Furthermore, the joint loss function includes: Pixel-level mean square error loss is used to constrain the deviation between the target hyperspectral feature map after channel adaptation and the true hyperspectral data at the intensity values ​​of each spectral channel at each pixel. Spectral angle matching loss is used to constrain the directional consistency between the target hyperspectral feature map after channel adaptation and the spectral vector composed of spectral channel intensity values ​​at each pixel of the true hyperspectral data. The joint loss function is a weighted sum of the pixel-level mean square error loss and the spectral angle matching loss.

[0014] To address the aforementioned problems, another aspect of this invention provides an electronic device, including: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the training method for the metasurface hyperspectral reconstruction model based on cross-modal text enhancement described above.

[0015] To address the aforementioned problems, in another aspect of this invention, a non-transient machine-readable medium storing computer instructions is provided, the computer instructions being used to cause the computer to execute the training method for the metasurface hyperspectral reconstruction model based on cross-modal text enhancement described above.

[0016] To address the aforementioned problems, in another aspect of this invention, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the training method for the metasurface hyperspectral reconstruction model based on cross-modal text enhancement described above.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: (1) By constructing a model architecture that includes an image encoder, a text encoder, a cross-modal alignment module, a cross-modal condition control module, an encoder-level enhancement module, and a hybrid spatial spectrum decoder, this invention realizes the joint representation and collaborative reconstruction of multispectral images and text semantic prompts acquired from metasurfaces. This effectively overcomes the semantic loss problem caused by existing methods relying solely on the image's own features, and provides a clear and functional technical framework for hyperspectral reconstruction.

[0018] (2) This invention introduces a spatial-spectral dual-path attention mechanism in the cross-modal condition control module. Using text projection features as queries, it performs independent attention calculations on the spatial and channel dimensions of the image feature representation to generate spatial guidance features and spectral guidance features. Combined with dynamic weighted fusion of similarity scores, it realizes differentiated modeling of spatial structure distribution and spectral variation law, which significantly improves the spatial detail fidelity and spectral fidelity of the target hyperspectral feature map.

[0019] (3) This invention constructs a progressive, semantically guided feature optimization mechanism by cascading basic transformation units, enhancement transformation units, and core transformation units, and embedding cross-modal update units between adjacent units: the basic transformation unit first extracts the local spatial structure and models the local dependencies between spectral channels; the enhancement transformation unit extracts the spatial-spectral joint context through parallel multi-scale convolution, and adaptively modulates the input features by combining the attention weights generated by global pooling; the core transformation unit focuses on modeling the long-range dependencies between spectral channels and capturing the structural consistency in the spatial dimension. Crucially, each cross-modal update unit dynamically introduces textual semantic priors at each network layer: by aligning the output features of the previous transformation unit with the textual semantic features, and generating fusion weights based on similarity scores, adaptive weighted fusion of the aligned features is achieved. This mechanism enables the reconstruction process to progressively fuse image content related to the semantics of the textual prompts, effectively alleviating the information redundancy and semantic shift problems in multimodal fusion, and improving the robustness of feature representation.

[0020] (4) This invention constructs a text cue library containing spatial distribution cues and spectral change cues, and combines it with multispectral images collected from metasurfaces and their registered true hyperspectral data to form triplet samples, thereby achieving end-to-end semantic guidance training of the model. During the training process, the pixel-level mean square error loss and spectral angle matching loss are jointly optimized, which not only constrains the intensity accuracy of the reconstruction results in each spectral channel, but also ensures the directional consistency of the spectral vectors, thereby effectively improving the spectral fidelity and generalization ability of the model in complex scenarios. Attached Figure Description

[0021] Figure 1 This is a structural diagram of a metasurface hyperspectral reconstruction model based on cross-modal text enhancement, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0023] To improve the spatial detail fidelity and spectral accuracy of hyperspectral reconstruction, such as Figure 1 As shown, this invention proposes a method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement, comprising: An image encoder is used to receive a multispectral image acquired by a metasurface and process the multispectral image through multiple cascaded convolutional layers to generate an image feature representation. In one embodiment of the present invention, the image encoder employs a four-level cascaded convolutional layer structure for progressive downsampling and feature extraction of the input multispectral image. Each level of the convolutional layer group consists of multiple sequentially connected convolutional units, each including a 3×3 convolutional layer, a batch normalization (BN) layer, and a ReLU activation function. After four levels of downsampling, the encoder outputs a high-dimensional feature map, which is then compressed into a fixed-dimensional image feature representation (e.g., 256×31×12) through a 1×1 convolutional layer, serving as the input for subsequent modules.

[0024] A text encoder is used to receive spatial distribution cues, spectral change cues, or a combination of both corresponding to the multispectral image, and generate corresponding text semantic features; In one embodiment of the present invention, the spatial distribution cues are natural language statements describing the spatial layout, morphology, and distribution patterns of features or targets in a multispectral image, such as: "Vegetation areas are concentrated in blocks on the left side of the image, water bodies run through the middle in strips, and urban building clusters are densely distributed in the upper right corner." The spectral variation cues are natural language statements describing the response characteristics or trends of different regions or features in the spectral dimension, such as: "Vegetation areas have high reflectivity in the near-infrared band, water bodies have strong absorption in the visible light band and their reflectivity gradually decreases with increasing wavelength, and the spectral curve of bare soil is flat with no significant absorption valleys." Both can be used individually or combined into a single composite cues, such as: "Farmland is distributed in a grid pattern, and its spectrum has obvious steep slope characteristics in the red-edge band," to simultaneously guide the model in jointly modeling spatial structure and spectral patterns.

[0025] The cross-modal alignment module is used to calculate the cosine similarity between image feature representations and text semantic features to obtain a similarity score; In one specific implementation, the cross-modal alignment module treats the image feature representation and the text semantic features as two-dimensional matrices, calculates the product of the transpose of the image feature representation and the text semantic features, takes the trace of the product as the numerator, and takes the product of the Frobenius norms of the image feature representation and the text semantic features as the denominator, thereby obtaining a normalized similarity score.

[0026] The cross-modal conditional control module projects text semantic features to the same spatial-channel dimension as the image feature representation to obtain text projection features. Based on the text projection features, spatial guided features and spectral guided features are generated respectively through a spatial-spectral dual-path attention mechanism, and the two are weighted and fused using similarity scores to output cross-modal enhanced image features. This invention introduces a spatial-spectral dual-path attention mechanism in the cross-modal conditional control module. Using text projection features as queries, it performs independent attention calculations on the spatial and channel dimensions of the image feature representation to generate spatially guided features and spectrally guided features. Combined with dynamic weighted fusion of similarity scores, it achieves differentiated modeling of spatial structure distribution and spectral variation patterns, significantly improving the spatial detail fidelity and spectral fidelity of the target hyperspectral feature map.

[0027] The cross-modal condition control module includes: The text projection unit, consisting of multiple cascaded convolutional layers, is used to project the text semantic features to the same spatial-channel dimension as the image feature representation, thereby obtaining text projection features; A spatial attention unit is used to perform spatial dimension attention calculation on the image feature representation using the text projection features as a query, and generate spatial guidance features; In one embodiment of the present invention, the spatial attention unit operates as follows: Image feature representations are passed through two independent 1×1 convolutional layers to generate spatial key features and spatial value features; meanwhile, text projection features are passed through a 1×1 convolutional layer to generate spatial query features.

[0028] Subsequently, the spatial query features and spatial key features are kept unchanged along the channel dimension, and the spatial dimension is flattened into a sequence form. Then, the two are multiplied by matrix to obtain the spatial attention weight. This weight reflects the semantic correlation between text semantics and each spatial location of the image.

[0029] Using this set of attention weights, the spatial value features are weighted and combined to directly output spatial guidance features.

[0030] The spatial guidance feature maintains the same spatial size as the image feature representation, reflecting the guiding role of text semantics on different spatial regions in the image.

[0031] A spectral attention unit is used to perform channel-dimensional attention calculations on the image feature representation using the text projection features as a query, thereby generating spectral guided features; The operation of the spectral attention unit is as follows: First, the image feature representation is viewed as a feature map composed of multiple channels, where each channel corresponds to a spectral band. For each spatial location in the image feature representation, its response values ​​on all channels are combined into a vector, which characterizes the spectral properties of that location.

[0032] Next, the image feature representations are input into two non-shared linear transformation modules. One module generates the key features required for spectral attention calculation, and the other generates the corresponding value features. Both transformations maintain the spatial structure and only map the channel dimensions. Simultaneously, the text projection features are transformed into query features through an independent linear transformation; these query features have the same channel dimensions as the key and value features. Then, the query features and key features are flattened in space, and the transposes of the flattened query features and key features are multiplied by a matrix to obtain the channel attention weight matrix. This weight matrix reflects the correlation between text semantics and the various spectral bands of the image. Using the channel attention weight matrix, the value features are weighted and combined along the channel dimension to directly output spectral guided features. These spectral guided features maintain the spatial dimensions of the image feature representation, reflecting the modulation effect of textual semantics on the image's spectral response.

[0033] The dynamic fusion unit performs a weighted summation of the spatial guidance features and the spectral guidance features based on the similarity score, and outputs the cross-modal enhanced image features.

[0034] In this embodiment, the dynamic fusion unit uses the similarity score as the fusion weight of the spatial guided feature and the difference between the similarity score and 1 as the fusion weight of the spectral guided feature. The two are weighted and summed to output cross-modal enhanced image features.

[0035] The encoder-level enhancement module is used to perform multi-scale spatial spectrum modeling and cross-modal dynamic updating of the cross-modal enhanced image features, and output multiple level fusion features; The encoder-level enhancement module includes multiple sequentially cascaded transformation units, and a cross-modal update unit disposed between every two adjacent transformation units; The plurality of sequentially cascaded transformation units include, in order: Two basic transformation units are used to extract the local spatial structure of their input features and model the inter-channel spectral dependence. In one specific implementation, the basic transformation unit comprises a depthwise separable convolutional layer, a normalization layer, and a nonlinear activation function connected in sequence. The depthwise separable convolutional layer uses a 3×3 spatial convolutional kernel to capture the joint spatial-spectral response within the local neighborhood while maintaining channel independence. The normalization layer can be batch normalization or layer normalization. The nonlinear activation function can be ReLU, GELU, or Swish. This structure does not include multi-scale operations or attention mechanisms, aiming to achieve initial feature stabilization with low computational overhead.

[0036] An enhancement transform unit is used to extract multi-scale spatial-spectral joint context information from its input features, and to construct a global context based on the information to generate attention weights, which are then used to adaptively modulate the input features of the enhancement transform unit. The enhancement transformation unit specifically includes: The multi-scale convolution module is used to perform depthwise separable convolution operations of different sizes (such as 5×5 and 7×7) on its input features in parallel to capture spatial-spectral joint contextual information at the corresponding scales. The feature stitching module is used to stitch together the spatial-spectral joint context information at various scales along the channel dimension to generate a fusion feature with the number of channels superimposed while the spatial size remains unchanged. The global context extraction module is used to perform average pooling and max pooling on the fused features to obtain the first global statistical feature and the second global statistical feature. The attention weight generation module is used to fuse the first global statistical feature and the second global statistical feature to obtain the fused feature, and then perform multiple cascaded learnable linear transformations on the fused feature in sequence, followed by nonlinear normalization operation to generate attention weights with the same spatial-channel dimension as the input features of the multi-scale convolution module. In one embodiment of the present invention, the sequential execution of multiple cascaded learnable linear transformations on the fused features refers to: inputting the fused features into the first learnable linear transformation module (e.g., a 1×1 convolutional layer), using its output as the input to the second learnable linear transformation module, and so on, forming a serially connected multi-level structure. The output of each level of transformation serves as the input to the next level, and the output of the last learnable linear transformation serves as the object of the nonlinear normalization operation.

[0037] The feature modulation module is used to multiply the attention weights element-wise with the input features of the multi-scale convolution module to obtain the modulation features, and add the modulation features to the input features of the multi-scale convolution module to obtain the output features of the enhancement transform unit.

[0038] Four core transformation units, each of which is used to model the long-range dependencies between spectral channels of its input features and capture the structural consistency in the spatial dimension; The first transformation unit receives the cross-modal enhanced image features as input, while the remaining transformation units receive the output of the previous transformation unit after processing by the corresponding cross-modal update unit as input.

[0039] In one specific implementation, the core transformation unit may employ a spatial-spectral joint modeling structure. For example, it may include: The long-range spectral dependency modeling subunit is used to model the global correlation between different spectral channels along the spectral dimension (e.g., through a self-attention mechanism). Spatial structure consistency constraint subunits are used to maintain semantic consistency between adjacent pixels in the spatial dimension (e.g., through local window attention or graph convolution operations).

[0040] This implementation aims to enhance the model's ability to express higher-order spatial spectrum semantics. It should be noted that the above structure is merely an example, and the invention is not limited thereto.

[0041] The cross-modal update unit is used to receive the output features of the previous transformation unit and the text semantic features, and align them; generate fusion weights based on the similarity scores, perform weighted fusion of the aligned output features and the aligned text semantic features based on the fusion weights, and use the fusion result as the input of the next transformation unit.

[0042] In one embodiment of the present invention, the cross-modal update unit first adjusts the text semantic features and the output features of the previous transformation unit to the same number of channels through convolutional layers and fully connected layers, respectively. Then, it expands the adjusted text semantic features in the spatial dimension so that they have the same spatial size as the output features of the previous transformation unit, thereby obtaining two features with the same shape. Next, the two features are concatenated in the channel dimension to form a combined feature, and the combined feature is input into a small convolutional network (e.g., a network containing a 3×3 convolutional layer and a ReLU activation function), which generates a preliminary spatial attention map. Subsequently, the value at each position in the spatial attention map is multiplied by the similarity score output by the cross-modal alignment module to obtain the modulated attention map. Then, the value at each spatial position in the modulated attention map is independently processed by the Sigmoid function to compress all values ​​to between 0 and 1, forming the final fusion weight. Finally, the adjusted output features of the previous transform unit and the expanded text semantic features are weighted and fused using the fusion weight: specifically, at each spatial location, the corresponding part of the adjusted output feature of the previous transform unit is multiplied by the fusion weight, and the corresponding part of the expanded text semantic features is multiplied by "1 minus the fusion weight". The two are then added together to obtain the fusion result at that location. The entire fusion result is used as the input to the next transform unit.

[0043] The encoder-level enhancement module outputs multiple level fusion features, including: The first-level fusion feature is the feature of the output of the second basic transformation unit after being processed by the cross-modal update unit set between it and the enhancement transformation unit; The second-level fusion feature is the feature of the output of the enhanced transform unit after being processed by the cross-modal update unit set between it and the first core transform unit; The third-level fusion feature is the feature of the output of the second core transformation unit after being processed by the cross-modal update unit set between it and the third core transformation unit; The fourth-level fusion feature is the output of the last core transformation unit.

[0044] This invention constructs a progressive, semantically guided feature optimization mechanism by cascading basic transformation units, enhancement transformation units, and core transformation units, and embedding cross-modal update units between adjacent units. The basic transformation unit first extracts local spatial structure and models local dependencies between spectral channels; the enhancement transformation unit extracts spatial-spectral joint context through parallel multi-scale convolution and adaptively modulates the input features using attention weights generated by global pooling; the core transformation unit focuses on modeling long-range dependencies between spectral channels and capturing structural consistency in the spatial dimension. Crucially, each cross-modal update unit dynamically introduces textual semantic priors at each network layer: by aligning the output features of the previous transformation unit with textual semantic features and generating fusion weights based on similarity scores, adaptive weighted fusion of the aligned features is achieved. This mechanism enables the reconstruction process to progressively fuse image content related to textual prompts, effectively mitigating information redundancy and semantic shift issues in multimodal fusion and improving the robustness of feature representation.

[0045] Hybrid spatial spectrum decoder is used to construct a target hyperspectral feature map based on multi-level fused features.

[0046] The hybrid spatial spectrum decoder includes: The local feature processing module is used to enhance the local spatial structure and spectral channel correlation of the multiple layered fusion features through the feature enhancement layer, so as to obtain local enhanced features that correspond one-to-one with each layered fusion feature. In one implementation, the feature enhancement layer is specifically a CSPRepLayer, which consists of a Cross Stage Partial Connection structure and a Reparameterized Convolutional Module (RepVGG).

[0047] In another implementation, the feature enhancement layer may specifically include: The feature splitting unit is used to divide the input hierarchical fusion features into first sub-features and second sub-features along the spectral channel dimension; The parallel processing unit includes a first branch and a second branch, wherein the first branch is used to directly pass the first sub-feature, and the second branch is used to perform convolution transformation on the second sub-feature through the reparameterized convolution module. The feature aggregation unit is used to concatenate the output of the first branch and the output of the second branch along the spectral channel dimension to obtain local enhancement features.

[0048] The above structure helps to enhance the correlation between local spatial structure and spectral channels, and control computational complexity.

[0049] The global feature modeling module is used to upsample the fused features at each level and the corresponding local enhancement features to a preset spatial resolution, spatially flatten each upsampled feature to obtain a corresponding feature sequence, and concatenate all feature sequences along the sequence dimension to form a global feature vector; linearly project the global feature vector to generate query features, key features, and value features respectively, and calculate attention weights based on the similarity between the query features and key features, and perform weighted aggregation of the value features through the attention weights to output the global fused features; The mapping module is used to map the global fused features into a target hyperspectral feature map through a convolutional layer.

[0050] This invention constructs a model architecture that includes an image encoder, a text encoder, a cross-modal alignment module, a cross-modal conditional control module, an encoder-level enhancement module, and a hybrid spatial spectrum decoder. This architecture enables the joint representation and collaborative reconstruction of multispectral images acquired from metasurfaces and textual semantic cues. It effectively overcomes the semantic loss problem caused by existing methods that rely solely on the image's own features, and provides a clear and functionally complete technical framework for hyperspectral reconstruction.

[0051] This invention also proposes a training method for a metasurface hyperspectral reconstruction model based on cross-modal text enhancement as described above, comprising: Construct a text cue library, which includes spatial distribution cue describing the spatial distribution characteristics of multispectral images, and spectral change cue describing the spectral change characteristics; A multispectral image acquired from a metasurface, a corresponding text prompt for the multispectral image, and registered ground truth hyperspectral data are obtained to construct a triplet sample. The triplet sample is then preprocessed to form a training set. The text prompt is a spatial distribution prompt, a spectral change prompt, or a combination of both. In this embodiment, the preprocessing of the triplet samples includes: normalizing the multispectral image and the true hyperspectral data to make the pixel value distribution uniform to a preset range (such as [0,1]).

[0052] The hyperspectral reconstruction model is trained end-to-end using the training set. In each training iteration, a multispectral image and its corresponding text prompt are input into the metasurface hyperspectral reconstruction model to obtain a target hyperspectral feature map. Channel adaptation is performed on the target hyperspectral feature map, and a joint loss function is calculated based on the adapted target hyperspectral feature map and the corresponding ground truth hyperspectral data to update the model parameters.

[0053] The channel adaptation can be achieved through a 1×1 convolutional layer, which is used to adjust the channel scale and offset of the hyperspectral reconstruction model output so that it is aligned with the numerical distribution of the ground truth hyperspectral data.

[0054] The joint loss function includes: Pixel-level mean square error loss is used to constrain the deviation between the target hyperspectral feature map after channel adaptation and the true hyperspectral data at the intensity values ​​of each spectral channel at each pixel. Spectral angle matching loss is used to constrain the directional consistency between the target hyperspectral feature map after channel adaptation and the spectral vector composed of spectral channel intensity values ​​at each pixel of the true hyperspectral data. The joint loss function is a weighted sum of the pixel-level mean square error loss and the spectral angle matching loss.

[0055] In this embodiment, the weights of pixel-level mean square error loss and spectral angle matching loss are set to 100 and 10, respectively, in order to achieve a balance between spatial fidelity and spectral consistency.

[0056] In one specific implementation, the hyperspectral reconstruction model uses the Adam optimizer for parameter updates, with an initial learning rate set to... The training consisted of 11 epochs. The learning rate remained constant for the first 8 epochs, and from the 8th to the 11th epochs, a cosine annealing strategy was used to smoothly decay the learning rate. To balance convergence speed and final performance.

[0057] This invention constructs a text cue library containing spatial distribution and spectral change cues, and combines it with multispectral images acquired from metasurfaces and their registered ground truth hyperspectral data to form triplet samples, thereby achieving end-to-end semantic-guided training of the model. During training, the pixel-level mean square error loss and spectral angle matching loss are jointly optimized, which not only constrains the intensity accuracy of the reconstruction results in each spectral channel, but also ensures the directional consistency of the spectral vectors, thus effectively improving the spectral fidelity and generalization ability of the model in complex scenes.

[0058] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of this invention.

[0059] The present invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of the present invention.

[0060] This invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of this invention.

[0061] refer to Figure 2 The present invention will now describe a structural block diagram of an electronic device that can serve as a server or client in embodiments of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0062] like Figure 2 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0063] Multiple components in the electronic device are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information into the electronic device. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 408 may include, but is not limited to, disks and optical discs. Communication unit 409 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0064] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the methods and processes described above. For example, in some embodiments, the method embodiments of the present invention may be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0065] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0066] In the context of embodiments of the present invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0067] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of the components in a specific posture (as shown in the attached figures). If the specific posture changes, the directional indication will also change accordingly. Furthermore, descriptions involving "first," "second," or "a" in the present invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. In the description of the present invention, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly defined. In the present invention, unless otherwise explicitly specified and defined, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can be a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal communication of two components or the interaction between two components, unless otherwise explicitly defined. For those skilled in the art, the specific meanings of the above terms in this invention can be understood according to the specific circumstances. Furthermore, the technical solutions of the various embodiments of this invention can be combined with each other, but only on the basis that those skilled in the art can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement, characterized in that, The metasurface hyperspectral reconstruction model includes: An image encoder is used to receive a multispectral image acquired by a metasurface and process the multispectral image through multiple cascaded convolutional layers to generate an image feature representation. A text encoder is used to receive spatial distribution cues, spectral change cues, or a combination of both corresponding to the multispectral image, and generate corresponding text semantic features; The cross-modal alignment module is used to calculate the cosine similarity between image feature representations and text semantic features to obtain a similarity score; The cross-modal conditional control module projects text semantic features to the same spatial-channel dimension as the image feature representation to obtain text projection features. Based on the text projection features, spatial guided features and spectral guided features are generated respectively through a spatial-spectral dual-path attention mechanism, and the two are weighted and fused using similarity scores to output cross-modal enhanced image features. The cross-modal condition control module includes: The text projection unit, consisting of multiple cascaded convolutional layers, is used to project the text semantic features to the same spatial-channel dimension as the image feature representation, thereby obtaining text projection features; A spatial attention unit is used to perform spatial dimension attention calculation on the image feature representation using the text projection features as a query, and generate spatial guidance features; A spectral attention unit is used to perform channel-dimensional attention calculations on the image feature representation using the text projection features as a query, thereby generating spectral guided features; The dynamic fusion unit performs a weighted summation of the spatial guidance features and the spectral guidance features based on the similarity score, and outputs the cross-modal enhanced image features. The encoder-level enhancement module is used to perform multi-scale spatial spectrum modeling and cross-modal dynamic updating of the cross-modal enhanced image features, and output multiple level fusion features; Hybrid spatial spectrum decoder is used to construct a target hyperspectral feature map based on multi-level fused features.

2. The method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement according to claim 1, characterized in that, The encoder-level enhancement module includes multiple sequentially cascaded transformation units, and a cross-modal update unit disposed between every two adjacent transformation units; The plurality of sequentially cascaded transformation units include, in order: Two basic transformation units are used to extract the local spatial structure of their input features and model the inter-channel spectral dependence. An enhancement transform unit is used to extract multi-scale spatial-spectral joint context information from its input features, and to construct a global context based on the information to generate attention weights, which are then used to adaptively modulate the input features of the enhancement transform unit. Four core transformation units, each of which is used to model the long-range dependencies between spectral channels of its input features and capture the structural consistency in the spatial dimension; The first transformation unit receives the cross-modal enhanced image features as input, and the remaining transformation units receive the output of the previous transformation unit after being processed by the corresponding cross-modal update unit as input. The cross-modal update unit is used to receive the output features of the previous transformation unit and the text semantic features, and align them; generate fusion weights based on the similarity scores, perform weighted fusion of the aligned output features and the aligned text semantic features based on the fusion weights, and use the fusion result as the input of the next transformation unit.

3. The method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement according to claim 2, characterized in that, The encoder-level enhancement module outputs multiple level fusion features, including: The first-level fusion feature is the feature of the output of the second basic transformation unit after being processed by the cross-modal update unit set between it and the enhancement transformation unit; The second-level fusion feature is the feature of the output of the enhanced transform unit after being processed by the cross-modal update unit set between it and the first core transform unit; The third-level fusion feature is the feature of the output of the second core transformation unit after being processed by the cross-modal update unit set between it and the third core transformation unit; The fourth-level fusion feature is the output of the last core transformation unit.

4. The method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement according to claim 3, characterized in that, The enhancement transformation unit specifically includes: The multi-scale convolution module is used to perform depthwise separable convolution operations of different sizes on its input features in parallel to capture spatial-spectral joint contextual information at the corresponding scale. The feature stitching module is used to stitch together the spatial-spectral joint context information at various scales along the channel dimension to generate a fusion feature with the number of channels superimposed while the spatial size remains unchanged. The global context extraction module is used to perform average pooling and max pooling on the fused features to obtain the first global statistical feature and the second global statistical feature. The attention weight generation module is used to fuse the first global statistical feature and the second global statistical feature to obtain the fused feature, and then perform multiple cascaded learnable linear transformations on the fused feature in sequence, followed by nonlinear normalization operation to generate attention weights with the same spatial-channel dimension as the input features of the multi-scale convolution module. The feature modulation module is used to multiply the attention weights element-wise with the input features of the multi-scale convolution module to obtain the modulation features, and add the modulation features to the input features of the multi-scale convolution module to obtain the output features of the enhancement transform unit.

5. The method for constructing a metasurface hyperspectral reconstruction model based on cross-modal text enhancement according to claim 4, characterized in that, The hybrid spatial spectrum decoder includes: The local feature processing module is used to enhance the local spatial structure and spectral channel correlation of the multiple layered fusion features through the feature enhancement layer, so as to obtain local enhanced features that correspond one-to-one with each layered fusion feature. The global feature modeling module is used to upsample the fused features at each level and the corresponding local enhancement features to a preset spatial resolution, spatially flatten each upsampled feature to obtain a corresponding feature sequence, and concatenate all feature sequences along the sequence dimension to form a global feature vector; linearly project the global feature vector to generate query features, key features, and value features respectively, and calculate attention weights based on the similarity between the query features and key features, and perform weighted aggregation of the value features through the attention weights to output the global fused features; The mapping module is used to map the global fused features into a target hyperspectral feature map through a convolutional layer.

6. A training method for a metasurface hyperspectral reconstruction model based on cross-modal text enhancement, wherein the metasurface hyperspectral reconstruction model is constructed using the method described in any one of claims 1 to 5, characterized in that, include: Construct a text cue library, which includes spatial distribution cue describing the spatial distribution characteristics of multispectral images, and spectral change cue describing the spectral change characteristics; A multispectral image acquired from a metasurface, a corresponding text prompt for the multispectral image, and registered ground truth hyperspectral data are obtained to construct a triplet sample. The triplet sample is then preprocessed to form a training set. The text prompt is a spatial distribution prompt, a spectral change prompt, or a combination of both. The hyperspectral reconstruction model is trained end-to-end using the training set. In each training iteration, a multispectral image and its corresponding text prompt are input into the metasurface hyperspectral reconstruction model to obtain a target hyperspectral feature map. Channel adaptation is performed on the target hyperspectral feature map, and a joint loss function is calculated based on the adapted target hyperspectral feature map and the corresponding ground truth hyperspectral data to update the model parameters.

7. The training method for the metasurface hyperspectral reconstruction model based on cross-modal text enhancement according to claim 6, characterized in that, The joint loss function includes: Pixel-level mean square error loss is used to constrain the deviation between the target hyperspectral feature map after channel adaptation and the true hyperspectral data at the intensity values ​​of each spectral channel at each pixel. Spectral angle matching loss is used to constrain the directional consistency between the target hyperspectral feature map after channel adaptation and the spectral vector composed of spectral channel intensity values ​​at each pixel of the true hyperspectral data. The joint loss function is a weighted sum of the pixel-level mean square error loss and the spectral angle matching loss.

8. An electronic device, comprising: A processor, and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the training method for a metasurface hyperspectral reconstruction model based on cross-modal text enhancement according to claim 6.

Images