A method for constructing a metasurface snapshot hyperspectral reconstruction model, a training method and an electronic device

Abstract

This invention discloses a method for constructing, training, and an electronic device for a metasurface snapshot hyperspectral reconstruction model, relating to the field of metasurface snapshots. By constructing a metasurface snapshot hyperspectral reconstruction model comprising a segmentation module, a spectral stability evaluation module, an inter-spectral feature encoder, a conditional feature generation module, and a hyperspectral generation module, this invention achieves end-to-end reconstruction from low-channel multispectral snapshot images to a high-dimensional hyperspectral data cube. This model not only effectively suppresses interference from background and occlusion regions but also explicitly models spectral stability and inter-channel dependencies. Furthermore, through iterative autoregressive modeling based on conditional features and random noise, it generates multi-channel hyperspectral features and ultimately outputs a hyperspectral data cube. Thus, while maintaining system compactness, this invention significantly improves the spectral fidelity, physical consistency, and spatial-spectral joint accuracy of the reconstruction results.

Description

Technical Field

[0001] This invention relates to the field of metasurface snapshots, and more particularly to a method for constructing, training, and an electronic device for hyperspectral reconstruction of metasurface snapshot models. Background Technology

[0002] Traditional hyperspectral imaging systems typically rely on spectroscopic elements such as prisms, gratings, or filters, requiring scanning or multi-frame acquisition to obtain complete spectral information. This results in bulky and complex equipment that struggles to meet the real-time imaging needs of dynamic scenes. In contrast, metasurface-based snapshot imaging technology utilizes monolithically integrated optical devices to simultaneously capture spatial and partial spectral information in a single exposure, significantly improving system compactness and imaging speed. However, its output is limited to multispectral snapshot images with only a limited number of channels (e.g., 4–8 channels), far below the tens to hundreds of bands of hyperspectral resolution required for practical applications. Therefore, there is an urgent need to use deep learning-based spectral reconstruction techniques to map low-channel snapshots into high-dimensional hyperspectral data cubes. However, existing reconstruction methods generally suffer from multiple shortcomings: on the one hand, most models fail to effectively segment the target region in the input image, including background or occluded areas in the reconstruction, introducing noise interference; on the other hand, they lack an evaluation mechanism for the consistency of response between multispectral channels, failing to distinguish between statically stable and dynamically unstable regions, resulting in distorted spectral reconstruction at unstable pixels; simultaneously, existing network structures often ignore cross-channel dependencies between spectral channels, failing to explicitly model inter-spectral correlations, easily leading to channel crosstalk; furthermore, although some generative methods attempt to introduce autoregressive or conditionally guided strategies to improve detail fidelity, they do not integrate spatial structure priors and spectral stability constraints, and the training process is not optimized for small sample scenarios, making it difficult to achieve high-fidelity, physically consistent hyperspectral reconstruction with limited data. Therefore, there is an urgent need for a reconstruction framework that can jointly perceive the target region, evaluate spectral stability, model inter-spectral dependencies, and support conditionally driven iterative generation, in order to efficiently and accurately recover hyperspectral data cubes under the hardware constraints of metasurface snapshot imaging. Summary of the Invention

[0003] To address the challenge of existing technologies in metasurface snapshot hyperspectral reconstruction that struggle to simultaneously achieve spatial structure perception, spectral stability modeling, and inter-spectral dependency learning, this invention proposes a metasurface snapshot hyperspectral reconstruction model, comprising: The segmentation module is used to automatically segment multispectral snapshot images acquired by metasurfaces and generate target region masks; The spectral stability evaluation module is used to evaluate the response consistency of each effective pixel in each spectral channel of the multispectral snapshot image based on the effective pixels indicated by the target region mask, and generate structured features based on the evaluation results. An inter-spectral feature encoder is used to extract features from the structured features, model the cross-channel dependencies between spectral channels based on the extracted features, and output spatial-spectral fusion features. The conditional feature generation module is used to generate conditional features by fusing spatial structure information and spectral constraints based on the spatial-spectral fusion features and the structured features; The hyperspectral generation module is used to generate multi-channel hyperspectral features based on the conditional features and random noise through iterative autoregressive modeling, and to convert the multi-channel hyperspectral features into a hyperspectral data cube output.

[0004] Furthermore, the evaluation of the response consistency of each effective pixel across each spectral channel of the multispectral snapshot image, and the generation of structured features based on the evaluation results, specifically involves: Calculate the standard deviation of the spectral response of the effective pixels in each spectral channel; divide the target area mask into statically stable regions and dynamically unstable regions based on the standard deviation of the spectral response. The statically stable region is encoded as a first binary mask, and the dynamically unstable region is encoded as a second binary mask; The multispectral snapshot image, the first binary mask, and the second binary mask are stitched together along the channel dimension to obtain the multispectral content tensor; The target region mask is fused with the first binary mask and the second binary mask to generate a guiding mask for indicating statically stable regions and dynamically unstable regions.

[0005] Furthermore, the inter-spectral feature encoder includes: A deep convolutional layer is used to extract local spatial-spectral features from the multispectral content tensor and output a preliminary feature representation. The inter-spectral attention layer is used to split the preliminary feature representation into multiple channel feature matrices according to the spectral channel dimension, and calculate the inter-dependency weight between each spectral channel based on each channel feature matrix through an attention mechanism. Based on the weight, the feature vectors from each channel feature matrix at each spatial location are weighted and fused to generate spectral enhancement features. A multi-head attention layer is used to perform global context modeling of the spectral enhancement features across spatial locations, and output spatial-spectral fusion features.

[0006] Furthermore, the condition feature generation module includes: A layer encoder is used to extract spatial structural features from the spatial-spectral fusion features and output a structural feature representation. The encoder is used to adjust the spatial dimensions of the guide mask to match the spatial dimension of the structural feature representation, and to encode the adjusted guide mask into spectral constraint features through a convolution function and a nonlinear activation function. A cross-attention fusion unit is used to perform a cross-attention operation based on the structural feature representation and the spectral constraint features, and to perform residual fusion with the structural feature representation to generate conditional features.

[0007] Furthermore, the hyperspectral generation module specifically includes: An initialization unit is used to align the conditional features with random noise to generate initial generated features; An iterative generation unit is used to generate multi-channel hyperspectral features based on the initial generated features and the conditional features through multiple rounds of autoregressive processes involving upsampling and linear transformation.

[0008] Furthermore, the autoregressive process is used to divide the spectral channels of the multi-channel hyperspectral features to be generated into multiple channel groups according to the band order, and process them in rounds according to a preset channel group order to generate multi-channel hyperspectral features; wherein: In the first round, based on the initial generated features and the conditional features, a first channel group feature is generated; In the second and subsequent rounds, the channel group features output in the previous round are spatially upsampled, and the upsampled results are linearly transformed and added to the conditional features respectively. Then, the results are processed by the Softmax activation function to output the channel group features corresponding to the current round.

[0009] Furthermore, the step of converting the multi-channel hyperspectral features into a hyperspectral data cube output specifically involves: The multi-channel hyperspectral features are convolved to map the feature map of each spectral channel to a single-band intensity image at the corresponding wavelength; the single-band intensity images are stacked in ascending order of wavelength to form a hyperspectral data cube; each pixel in the single-band intensity image corresponds to a spectral channel intensity value.

[0010] This invention also proposes a training method for the metasurface snapshot hyperspectral reconstruction model as described above, comprising: Acquire multispectral snapshot images collected from metasurfaces and obtain ground truth hyperspectral data cubes registered with them. Construct paired samples and preprocess the paired samples to obtain training samples, forming a dataset. The metasurface snapshot hyperspectral reconstruction model is trained using a dataset and a joint loss function, and the parameters of the interspectral feature encoder, except for the interspectral attention layer, are frozen during the training process.

[0011] Furthermore, the joint loss function includes: Pixel-level L1 loss is used to constrain the absolute error between the hyperspectral data cube output by the metasurface snapshot hyperspectral reconstruction model and the ground hyperspectral data cube at each spectral channel intensity value of each pixel. Spectral angle constraint loss is used to constrain the directional consistency between the hyperspectral data cube and the true hyperspectral data cube at each pixel, based on the spectral curves formed by the intensity values ​​of each spectral channel. Cross-entropy loss is used to monitor the consistency between the channel group features generated by the hyperspectral generation module in each round of the autoregression process and the prediction of the corresponding spectral channels in the ground truth hyperspectral data cube. The distribution consistency constraint is used to constrain the consistency of the spectral channel intensity value distribution of the hyperspectral data cube and the true hyperspectral data cube across all pixels.

[0012] To address the aforementioned problems, another aspect of this invention provides an electronic device, including: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the training method described above.

[0013] To address the aforementioned problems, in another aspect of this invention, a non-transitory machine-readable medium storing computer instructions is provided, the computer instructions being used to cause the computer to execute the training method described above.

[0014] To address the aforementioned problems, in another aspect of this invention, a computer program product is provided, comprising a computer program / instructions, which, when executed by a processor, implement the training method described above.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: (1) This invention constructs a metasurface snapshot hyperspectral reconstruction model that includes a segmentation module, a spectral stability evaluation module, an inter-spectral feature encoder, a conditional feature generation module, and a hyperspectral generation module, thereby achieving end-to-end reconstruction from low-channel multispectral snapshot images to high-dimensional hyperspectral data cubes. This model not only effectively suppresses the interference of background and occlusion regions, but also explicitly models the spectral stability and inter-channel dependencies. Through iterative autoregressive modeling based on conditional features and random noise, it generates multi-channel hyperspectral features and finally outputs a hyperspectral data cube. Thus, this invention significantly improves the spectral fidelity, physical consistency, and spatial-spectral joint accuracy of the reconstruction results while maintaining the system's compactness.

[0016] (2) The present invention generates a target region mask by segmentation module and combines it with spectral stability evaluation module to quantify the response consistency of effective pixels in each spectral channel, dividing the target region mask into statically stable regions and dynamically unstable regions, forming a multispectral content tensor and a guiding mask; this mechanism enables subsequent modules to differentiate between statically stable regions and dynamically unstable regions, thereby improving the robustness of hyperspectral reconstruction in complex scenarios and the directional consistency of spectral curves, effectively solving the problem of spectral distortion of unstable pixels caused by the inability to distinguish regional stability in the prior art, and significantly improving the spectral fidelity of reconstruction results in dynamic or interference regions.

[0017] (3) In this invention, an inter-spectral attention layer is introduced into the inter-spectral feature encoder. The inter-channel feature matrix after the preliminary feature representation extracted by the deep convolutional layer is split into spectral channels is dynamically calculated, and the inter-channel information is weighted and fused at each spatial location to generate spectral enhancement features. This design explicitly models the nonlinear correlation between spectral channels, effectively alleviates the channel crosstalk problem caused by ignoring inter-spectral coupling in traditional methods, and enhances the continuity and physical rationality of the reconstructed spectrum.

[0018] (4) During the training process, the present invention freezes the parameters of other layers in the interspectral feature encoder except for the interspectral attention layer, and adopts a joint loss function that includes pixel-level norm loss, spectral angle constraint loss, cross-entropy loss and distribution consistency constraint term. This not only reduces the risk of overfitting in small sample scenarios, but also optimizes the model from multiple dimensions such as pixel accuracy, spectral direction, generation distribution and channel prediction consistency. This strategy significantly improves the generalization ability under limited training data and ensures that the reconstructed hyperspectral data cube is highly consistent with the true value in terms of detail fidelity and global statistical characteristics. Attached Figure Description

[0019] Figure 1 This is a structural diagram of a metasurface snapshot hyperspectral reconstruction model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0021] To address the challenges of existing techniques in hyperspectral reconstruction of metasurface snapshots that struggle to simultaneously achieve spatial structure perception, spectral stability modeling, and interspectral dependency learning, such as... Figure 1 As shown, this invention proposes a metasurface snapshot hyperspectral reconstruction model, comprising: The segmentation module is used to automatically segment multispectral snapshot images acquired by metasurfaces and generate target region masks; Specifically, the segmentation module can perform single-channel segmentation on each spectral channel of the multispectral snapshot image to generate corresponding channel region masks, and then fuse multiple channel region masks into a target region mask by means of union or weighted average.

[0022] In this embodiment, the automatic segmentation function separates the target region from the background region. The generated target region mask is used to limit the processing range of subsequent spectral reconstruction, thereby reducing the interference of background noise on the reconstruction results.

[0023] During the model inference phase, in order to maintain consistency with the data distribution during the training phase, the multispectral snapshot image can be preprocessed in the same way as during the training phase (for example, the intensity value of each pixel in each spectral channel is divided by 255 and normalized to the [0,1] interval) to ensure the consistency of the input data distribution. Subsequently, the preprocessed multispectral snapshot image is input to the segmentation module, which automatically segments and generates a target region mask.

[0024] The spectral stability evaluation module is used to evaluate the response consistency of each effective pixel in each spectral channel of the multispectral snapshot image based on the effective pixels indicated by the target region mask, and generate structured features based on the evaluation results. In this embodiment, the effective pixel refers to the pixel with a mask value of 1 in the target area mask.

[0025] The evaluation of the consistency of response of each effective pixel across the spectral channels of the multispectral snapshot image, and the generation of structured features based on the evaluation results, specifically involves: Calculate the standard deviation of the spectral response of the effective pixels in each spectral channel; divide the target area mask into statically stable regions and dynamically unstable regions based on the standard deviation of the spectral response. In one embodiment, the method of dividing the region according to the standard deviation of the spectral response is as follows: For each valid pixel in the target region mask, its intensity value in each spectral channel of the multispectral snapshot image is extracted, and the standard deviation of the spectral response of that pixel is calculated. If the standard deviation is less than or equal to a preset threshold (e.g., 0.1), the corresponding pixel is classified as a statically stable region; if it is greater than the preset threshold, it is classified as a dynamically unstable region. The preset threshold can be adjusted according to the spectral noise level of the metasurface imaging system and the dynamic characteristics of the scene.

[0026] The statically stable region is encoded as a first binary mask, and the dynamically unstable region is encoded as a second binary mask; The multispectral snapshot image, the first binary mask, and the second binary mask are stitched together along the channel dimension to obtain the multispectral content tensor; The target region mask is fused with the first binary mask and the second binary mask to generate a guiding mask for indicating statically stable regions and dynamically unstable regions.

[0027] This invention generates a target region mask through a segmentation module and quantifies the response consistency of effective pixels across spectral channels using a spectral stability evaluation module. The target region mask is divided into statically stable and dynamically unstable regions, forming a multispectral content tensor and a guiding mask. This mechanism enables subsequent modules to differentiate between statically stable and dynamically unstable regions, thereby improving the robustness of hyperspectral reconstruction in complex scenes and the directional consistency of spectral curves. It effectively solves the problem of spectral distortion of unstable pixels caused by the inability to distinguish regional stability in existing technologies, and significantly improves the spectral fidelity of reconstruction results in dynamic or interference regions.

[0028] An inter-spectral feature encoder is used to extract features from the structured features, model the cross-channel dependencies between spectral channels based on the extracted features, and output spatial-spectral fusion features. The interspectral feature encoder includes: A deep convolutional layer is used to extract local spatial-spectral features from the multispectral content tensor and output a preliminary feature representation. The inter-spectral attention layer is used to split the preliminary feature representation into multiple channel feature matrices according to the spectral channel dimension, and calculate the inter-dependency weight between each spectral channel based on each channel feature matrix through an attention mechanism. Based on the weight, the feature vectors from each channel feature matrix at each spatial location are weighted and fused to generate spectral enhancement features. In this embodiment, the preliminary feature represents a shape H×W×N×C, wherein: H represents the height of the feature map; W represents the width of the feature map; C represents the feature dimension of each spatial location in a single spectral channel; N represents the number of spectral channels; The interspectral attention layer first decomposes the preliminary feature representation into N channel feature matrices along the spectral channel dimension. Each The feature representation corresponding to the k-th spectral channel, .

[0029] For each spatial location From the first Channel feature matrix Extract the feature vector at this location That is, take In coordinates The C-dimensional feature vector at that location yields the spatial position. The corresponding feature vector sequence: .

[0030] Based on feature vector sequences: The spatial position of each spectral channel is calculated using an attention mechanism. The interdependence weights are used to obtain the spatial location. Each feature vector Corresponding interdependence weights and form this spatial location. The corresponding weight sequence: The spatial location is determined by this weight sequence. Weighted fusion of feature vectors: ; In the formula, Indicates spatial location The corresponding fusion results.

[0031] Finally, the fusion results of all spatial locations are reorganized into spectral enhancement features based on the spatial layout represented by the preliminary features.

[0032] In this embodiment, N is set to 4.

[0033] This invention introduces an inter-spectral attention layer into the inter-spectral feature encoder. It dynamically calculates the inter-dependency weights of the channel feature matrices after splitting the preliminary feature representation extracted by the deep convolutional layer into spectral channels, and weights and fuses cross-channel information at each spatial location to generate spectral enhancement features. This design explicitly models the nonlinear correlation between spectral channels, effectively alleviating the channel crosstalk problem caused by neglecting inter-spectral coupling in traditional methods, and enhancing the continuity and physical rationality of the reconstructed spectrum.

[0034] A multi-head attention layer is used to perform global context modeling of the spectral enhancement features across spatial locations, and output spatial-spectral fusion features.

[0035] In one embodiment, the metasurface snapshot hyperspectral reconstruction model further includes a reconstruction optimization module for refining the spatial-spectral fusion features output by the inter-spectral feature encoder to improve the visual quality and spectral fidelity of the reconstruction results.

[0036] The reconstruction optimization module adopts a lightweight convolutional network structure, specifically including: a 3×3 deep convolutional layer for extracting local spatial-spectral details; a normalization operation (e.g., batch normalization or layer normalization) is connected after the deep convolutional layer; at the same time, residual connections are introduced to add the input spatial-spectral fusion features to the normalized output of the deep convolutional layer, thereby enhancing high-frequency details while preserving the original feature information.

[0037] Through the above structure, the reconstruction optimization module can effectively suppress imaging blur and artifacts introduced by the metasurface optical system, achieve local detail enhancement and degradation compensation of spatial-spectral fusion features, and thus improve the spatial clarity and spectral consistency of the final hyperspectral reconstruction result.

[0038] The conditional feature generation module is used to generate conditional features by fusing spatial structure information and spectral constraints based on the spatial-spectral fusion features and the structured features; The conditional feature generation module includes: A layer encoder is used to extract spatial structural features from the spatial-spectral fusion features and output a structural feature representation. The encoder is used to adjust the spatial dimensions of the guide mask to match the spatial dimension of the structural feature representation, and to encode the adjusted guide mask into spectral constraint features through a convolution function and a nonlinear activation function. A cross-attention fusion unit is used to perform a cross-attention operation based on the structural feature representation and the spectral constraint features, and to perform residual fusion with the structural feature representation to generate conditional features.

[0039] In this embodiment, the cross-attention fusion unit is implemented as follows: Using the structural feature representation as the query (Query, Q) and the spectral constraint features as the key (Key, K) and value (Value, V) respectively, the cross-modal dependency between them is calculated through a scaled dot product attention mechanism to obtain the attention output; subsequently, the attention output is residually connected with the structural feature representation to generate the final conditional features.

[0040] The hyperspectral generation module is used to generate multi-channel hyperspectral features based on the conditional features and random noise through iterative autoregressive modeling, and to convert the multi-channel hyperspectral features into a hyperspectral data cube output.

[0041] The hyperspectral generation module specifically includes: An initialization unit is used to align the conditional features with random noise to generate initial generated features; An iterative generation unit is used to generate multi-channel hyperspectral features based on the initial generated features and the conditional features through multiple rounds of autoregressive processes involving upsampling and linear transformation.

[0042] The autoregressive process is used to divide the spectral channels of the multi-channel hyperspectral features to be generated into multiple channel groups according to the band order, and process them in rounds according to the preset channel group order to generate multi-channel hyperspectral features; wherein: In the first round, based on the initial generated features and the conditional features, a first channel group feature is generated; In the second and subsequent rounds, the channel group features output in the previous round are spatially upsampled, and the upsampled results are linearly transformed and added to the conditional features respectively. Then, the results are processed by the Softmax activation function to output the channel group features corresponding to the current round.

[0043] In one embodiment, the multi-channel hyperspectral feature to be generated includes 18 spectral channels, which are arranged in ascending order of their center wavelengths and divided into three channel groups: Channels 1 through 6 constitute the first channel group; Channels 7 through 12 constitute the second channel group; Channels 13 through 18 constitute the third channel group.

[0044] The autoregressive process is processed in rounds according to the above channel group sequence: The first round generates the first channel group features, namely the features of the 1st to 6th spectral channels, based on the initial generated features and the conditional features; In the second round, the first channel group features output from the previous round are spatially upsampled, and the upsampled results are linearly transformed and added to the conditional features respectively. Then, after processing by the Softmax activation function, the second channel group features, namely the features of the 7th to 12th spectral channels, are output. In the third round, the features of the second channel group output from the previous round are spatially upsampled, and the upsampled results are added to the conditional features after linear transformation. Then, the results are processed by the Softmax activation function to output the features of the third channel group, namely the features of the 13th to 18th spectral channels.

[0045] The channel group features generated in each round are sequentially spliced ​​together along the channel dimension to form a complete 18-channel hyperspectral feature.

[0046] The process of converting the multi-channel hyperspectral features into a hyperspectral data cube output is as follows: The multi-channel hyperspectral features are convolved to map the feature map of each spectral channel to a single-band intensity image at the corresponding wavelength; the single-band intensity images are stacked in ascending order of wavelength to form a hyperspectral data cube; each pixel in the single-band intensity image corresponds to a spectral channel intensity value.

[0047] This invention achieves end-to-end reconstruction from low-channel multispectral snapshot images to high-dimensional hyperspectral data cubes by constructing a metasurface snapshot hyperspectral reconstruction model that includes a segmentation module, a spectral stability assessment module, an inter-spectral feature encoder, a conditional feature generation module, and a hyperspectral generation module. This model not only effectively suppresses interference from background and occlusion regions but also explicitly models spectral stability and inter-channel dependencies. Through iterative autoregressive modeling based on conditional features and random noise, it generates multi-channel hyperspectral features and finally outputs a hyperspectral data cube. Thus, this invention significantly improves the spectral fidelity, physical consistency, and spatial-spectral joint accuracy of the reconstruction results while maintaining system compactness.

[0048] This invention also proposes a training method for the metasurface snapshot hyperspectral reconstruction model as described above, comprising: Acquire multispectral snapshot images collected from metasurfaces and obtain ground truth hyperspectral data cubes registered with them. Construct paired samples and preprocess the paired samples to obtain training samples, forming a dataset. In one embodiment, the multispectral snapshot image acquired by the metasurface is a 4-channel multispectral image obtained by an off-axis focusing metasurface mirror composed of multi-resonant metaatoms. Its spectrum covers the visible light band of 500–650 nm and includes four center wavelengths of 513 nm, 549 nm, 593 nm and 633 nm, which matches the natural spectral dispersive characteristics of the metasurface mirror. The ground truth hyperspectral data cube is an 18-channel hyperspectral image that has been spectrally calibrated and geometrically registered. Its center wavelength covers 480–650 nm, and the wavelength interval between adjacent channels is about 10 nm. It is used to construct paired samples of “4-channel input–18-channel output”. This ground truth hyperspectral data cube is used as a reconstruction target during model training to supervise the output of the metasurface snapshot hyperspectral reconstruction model.

[0049] In this embodiment, under the condition of limited labeled data, the dataset contains 18 to 20 sets of training samples.

[0050] In one embodiment of the present invention, the preprocessing of the paired samples includes: A normalization operation is performed on the multispectral snapshot image acquired by the metasurface. Specifically, the intensity value of each pixel in each spectral channel is divided by a pre-set constant (which is equal to the maximum pixel value supported by the image data format, for example, 255 for an 8-bit image), thereby scaling all intensity values ​​uniformly to the [0, 1] interval. Gaussian blur is then applied to the normalized image to simulate the imaging degradation characteristics of the metasurface optical system. The constant remains fixed during the model training and inference phases to ensure the consistency of the input data distribution.

[0051] The metasurface snapshot hyperspectral reconstruction model is trained using a dataset and a joint loss function, and the parameters of the interspectral feature encoder, except for the interspectral attention layer, are frozen during the training process.

[0052] The training process also includes completing the mask of the target region corresponding to the current training sample, specifically including: Select another multispectral snapshot image from the dataset that belongs to the same scene as the current training sample, and generate its corresponding target region mask as a contrast mask; Based on set difference operation, the difference between the contrast mask and the target region mask corresponding to the current training sample is calculated to obtain the dynamically added region mask; The dynamically added region mask is fused with the target region mask of the multispectral snapshot image corresponding to the current training sample to obtain the completed target region mask.

[0053] During the training phase, for the multispectral snapshot image corresponding to the current training sample, another multispectral snapshot image belonging to the same scene is selected from the dataset, and their respective target region masks are generated. The difference between the target region mask of the other frame and the target region mask corresponding to the current training sample is calculated by set difference operation to obtain a dynamically added region mask. This region corresponds to dynamic targets or temporary occlusions that may be missed in the segmentation results of the current training sample. The dynamically added region mask is fused with the target region mask corresponding to the current training sample to obtain the completed target region mask, thereby improving the model's adaptability to complex scenes and reconstruction robustness.

[0054] The joint loss function includes: Pixel-level L1 loss is used to constrain the absolute error between the hyperspectral data cube output by the metasurface snapshot hyperspectral reconstruction model and the ground hyperspectral data cube at each spectral channel intensity value of each pixel. Spectral angle constraint loss is used to constrain the directional consistency between the hyperspectral data cube and the true hyperspectral data cube at each pixel, based on the spectral curves formed by the intensity values ​​of each spectral channel. Cross-entropy loss is used to monitor the consistency between the channel group features generated by the hyperspectral generation module in each round of the autoregression process and the prediction of the corresponding spectral channels in the ground truth hyperspectral data cube. The distribution consistency constraint is used to constrain the consistency of the spectral channel intensity value distribution of the hyperspectral data cube and the true hyperspectral data cube across all pixels.

[0055] During training, this invention freezes the parameters of all layers in the interspectral feature encoder except for the interspectral attention layer, and employs a joint loss function that includes pixel-level norm loss, spectral angle constraint loss, cross-entropy loss, and distribution consistency constraint term. This reduces the risk of overfitting in small sample scenarios and collaboratively optimizes the model from multiple dimensions such as pixel accuracy, spectral direction, generation distribution, and channel prediction consistency. This strategy significantly improves the generalization ability under limited training data and ensures that the reconstructed hyperspectral data cube is highly consistent with the ground truth in terms of detail fidelity and global statistical properties.

[0056] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of this invention.

[0057] The present invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of the present invention.

[0058] This invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the method of this invention.

[0059] refer to Figure 2 The present invention will now describe a structural block diagram of an electronic device that can serve as a server or client in embodiments of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0060] like Figure 2 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0061] Multiple components in the electronic device are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information into the electronic device. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 408 may include, but is not limited to, disks and optical discs. Communication unit 409 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0062] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the methods and processes described above. For example, in some embodiments, the method embodiments of the present invention may be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0063] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0064] In the context of embodiments of the present invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0065] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of the components in a specific posture (as shown in the attached figures). If the specific posture changes, the directional indication will also change accordingly. Furthermore, descriptions involving "first," "second," or "a" in the present invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. In the description of the present invention, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly defined. In the present invention, unless otherwise explicitly specified and defined, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can be a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal communication of two components or the interaction between two components, unless otherwise explicitly defined. For those skilled in the art, the specific meanings of the above terms in this invention can be understood according to the specific circumstances. Furthermore, the technical solutions of the various embodiments of this invention can be combined with each other, but only on the basis that those skilled in the art can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A method for constructing a metasurface snapshot hyperspectral reconstruction model, characterized in that, The model includes: The segmentation module is used to automatically segment multispectral snapshot images acquired by metasurfaces and generate target region masks; The spectral stability evaluation module is used to evaluate the response consistency of each effective pixel in each spectral channel of the multispectral snapshot image based on the effective pixels indicated by the target region mask, and generate structured features based on the evaluation results. The evaluation of the consistency of response of each effective pixel across the spectral channels of the multispectral snapshot image, and the generation of structured features based on the evaluation results, specifically involves: Calculate the standard deviation of the spectral response of the effective pixels in each spectral channel; divide the target area mask into statically stable regions and dynamically unstable regions based on the standard deviation of the spectral response. The statically stable region is encoded as a first binary mask, and the dynamically unstable region is encoded as a second binary mask; The multispectral snapshot image, the first binary mask, and the second binary mask are stitched together along the channel dimension to obtain the multispectral content tensor; The target region mask is fused with the first binary mask and the second binary mask to generate a guiding mask for indicating statically stable regions and dynamically unstable regions; An interspectral feature encoder is used to extract features from the multispectral content tensor, model the cross-channel dependencies between spectral channels based on the extracted features, and output spatial-spectral fusion features. The conditional feature generation module is used to generate conditional features by fusing spatial structure information with the guiding mask based on the spatial-spectral fusion features and the structured features; The hyperspectral generation module is used to generate multi-channel hyperspectral features based on the conditional features and random noise through iterative autoregressive modeling, and to convert the multi-channel hyperspectral features into a hyperspectral data cube output.

2. The method of claim 1, wherein, The interspectral feature encoder includes: A deep convolutional layer is used to extract local spatial-spectral features from the multispectral content tensor and output a preliminary feature representation. The inter-spectral attention layer is used to split the preliminary feature representation into multiple channel feature matrices according to the spectral channel dimension, and calculate the inter-dependency weight between each spectral channel based on each channel feature matrix through an attention mechanism. Based on the weight, the feature vectors from each channel feature matrix at each spatial location are weighted and fused to generate spectral enhancement features. A multi-head attention layer is used to perform global context modeling of the spectral enhancement features across spatial locations, and output spatial-spectral fusion features.

3. The method of claim 2, wherein, The conditional feature generation module includes: A layer encoder is used to extract spatial structural features from the spatial-spectral fusion features and output a structural feature representation. The encoder is used to adjust the spatial dimensions of the guide mask to match the spatial dimension of the structural feature representation, and to encode the adjusted guide mask into spectral constraint features through a convolution function and a nonlinear activation function. A cross-attention fusion unit is used to perform a cross-attention operation based on the structural feature representation and the spectral constraint features, and to perform residual fusion with the structural feature representation to generate conditional features.

4. The method of claim 3, wherein, The hyperspectral generation module specifically includes: An initialization unit is used to align the conditional features with random noise to generate initial generated features; An iterative generation unit is used to generate multi-channel hyperspectral features based on the initial generated features and the conditional features through multiple rounds of autoregressive processes involving upsampling and linear transformation.

5. The method for constructing a metasurface snapshot hyperspectral reconstruction model according to claim 4, characterized in that, The autoregressive process is used to divide the spectral channels of the multi-channel hyperspectral features to be generated into multiple channel groups according to the band order, and process them in rounds according to the preset channel group order to generate multi-channel hyperspectral features; wherein: In the first round, based on the initial generated features and the conditional features, a first channel group feature is generated; In the second and subsequent rounds, the channel group features output in the previous round are spatially upsampled, and the upsampled results are linearly transformed and added to the conditional features respectively. Then, the results are processed by the Softmax activation function to output the channel group features corresponding to the current round.

6. The method of claim 1, wherein, The process of converting the multi-channel hyperspectral features into a hyperspectral data cube output is as follows: The multi-channel hyperspectral features are convolved to map the feature map of each spectral channel to a single-band intensity image at the corresponding wavelength; the single-band intensity images are stacked in ascending order of wavelength to form a hyperspectral data cube.

7. A training method for a metasurface snapshot hyperspectral reconstruction model, wherein the metasurface snapshot hyperspectral reconstruction model is constructed using the method described in any one of claims 1 to 6, characterized in that, include: Acquire multispectral snapshot images collected from metasurfaces and obtain ground truth hyperspectral data cubes registered with them. Construct paired samples and preprocess the paired samples to obtain training samples, forming a dataset. The metasurface snapshot hyperspectral reconstruction model is trained using a dataset and a joint loss function, and the parameters of the interspectral feature encoder, except for the interspectral attention layer, are frozen during training.

8. The training method for a metasurface snapshot hyperspectral reconstruction model according to claim 7, characterized in that, The joint loss function includes: Pixel-level L1 loss is used to constrain the absolute error between the hyperspectral data cube output by the metasurface snapshot hyperspectral reconstruction model and the ground hyperspectral data cube at each spectral channel intensity value of each pixel. Spectral angle constraint loss is used to constrain the directional consistency between the hyperspectral data cube and the true hyperspectral data cube at each pixel, based on the spectral curves formed by the intensity values ​​of each spectral channel. Cross-entropy loss is used to monitor the consistency between the channel group features generated by the hyperspectral generation module in each round of the autoregression process and the prediction of the corresponding spectral channels in the ground truth hyperspectral data cube. The distribution consistency constraint is used to constrain the consistency of the spectral channel intensity value distribution of the hyperspectral data cube and the true hyperspectral data cube across all pixels.

9. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the training method as described in claim 7.

Images