Single-exposure compressive imaging reconstruction method and system fusing pre-trained language model

Abstract

This invention specifically relates to a single-exposure compressed imaging reconstruction method and system that integrates pre-trained language models, solving the problem that existing technologies can only decode a single imaging modality. It employs a model framework of "physical constraints-semantic guidance-sparse controlled diffusion," uniformly mapping cross-modal compressed data to align the data stream. On one hand, it utilizes a lightweight mask-gated constraint feature extraction module to extract high-fidelity underlying physical features; on the other hand, it cascades a pre-trained multimodal large language model to generate a global semantic representation. In the controlled diffusion large model denoising and reconstruction stage, an asymmetric sparse controlled DiT architecture is used to solve the problem of easily losing high-frequency details, and computational redundancy is significantly reduced by decreasing the number of controllable modules and layers. Through this hardware-software co-design, 3D positional encoding mechanism, and text prompt word design, a single model can adaptively reconstruct high-precision images with multiple imaging modalities and compression rates.

Description

Technical Field

[0001] This invention belongs to the field of single-exposure compression imaging reconstruction technology, specifically relating to a single-exposure compression imaging reconstruction method and system that integrates a pre-trained language model. Background Technology

[0002] In the current field of image processing and generation, artificial intelligence technology based on diffusion models is developing rapidly. This technology, with its powerful ability to extract prior knowledge, can generate high-quality scene images from simple text prompts. Meanwhile, in modern scientific exploration and industrial applications, computational imaging technologies, such as video single-exposure compression imaging and hyperspectral single-exposure compression imaging, are playing a crucial role. The core of these technologies lies in the collaborative process of optical encoding at the front end of the hardware system and software decoding algorithms at the back end, reconstructing high-quality high-dimensional original images or videos from reduced, compressed, or blurred two-dimensional physical measurements, thus possessing significant advantages such as low bandwidth, low storage, and low power consumption.

[0003] However, existing single-exposure compressed imaging reconstruction algorithms generally suffer from the limitation of "task silos," meaning they can only decode a single imaging modality. Because different imaging models cannot share parameters and data, their generalization ability is extremely poor, and system development and training costs are high. Facing different imaging tasks, varying compression ratios, or novel hardware mask configurations, significant resources are often required to recollect data and train dedicated networks from scratch. This barrier severely hinders the transfer and reuse of cross-modal prior knowledge, limiting the rapid deployment of new computational imaging systems. Therefore, how to break down the barriers between different computational imaging tasks, effectively transfer and apply the powerful prior capabilities of diffusion models in image generation to complex computational imaging reconstruction tasks, and solve the mapping problem from complex physical measurements to high-fidelity images, is a pressing technical focus in the industry.

[0004] Despite some progress in existing technologies, the following significant drawbacks still exist in single-exposure compression imaging reconstruction tasks:

[0005] 1. High randomness in generation and lack of physical realism: Traditional diffusion models generate results with high randomness, which makes it difficult to meet the strict requirements of computational imaging tasks for "result realism" and "physical measurement consistency". They often pursue visual appeal while ignoring physical fidelity.

[0006] 2. The models are isolated and lack versatility: Existing single-exposure compressed imaging reconstruction methods typically design and train independent, dedicated models for different imaging tasks (such as compressed video, hyperspectral imaging, single-pixel imaging, etc.). This makes it difficult to transfer prior knowledge between different tasks, and the development and optimization process is cumbersome.

[0007] 3. Feature extraction loses a lot of local details: To reduce computational complexity, existing diffusion models often use traditional variational autoencoders (VAEs) to map images to the latent space. Traditional VAEs typically perform 8x spatial downsampling. This excessive downsampling leads to the loss of a large amount of key details such as image edges and textures, resulting in overly smooth reconstruction results.

[0008] 4. Lack of precise semantic guidance mechanism: Existing condition-guided reconstruction relies on manually designed prompts or simple condition injection, which cannot provide rich and detailed semantic guidance for complex and specific observation scenarios, thus limiting the reconstruction accuracy in complex scenarios.

[0009] 5. Insufficient generalization ability and flexibility: Traditional models often need to be retrained or fine-tuned when faced with different modulation mask distributions, different compression ratios or different types of sensors (grayscale / color), making it difficult to use them out of the box. Summary of the Invention

[0010] To address the technical problems in the background art, the purpose of this invention is to provide a single-exposure compressed imaging reconstruction method and system that integrates a pre-trained language model.

[0011] To achieve the aforementioned objectives, the present invention employs the following technical solution: a single-exposure compressed imaging reconstruction system integrating a pre-trained language model, comprising:

[0012] The multimodal data alignment and physical observation module uses a unified physical dimension mapping mechanism to convert color video, grayscale video, and hyperspectral images into a standard pseudo-color video sequence. The physical observation model then outputs the standard pseudo-color video sequence as aligned original two-dimensional compressed measurement values.

[0013] The feature extraction module for mask gating constraints uses a lightweight spatiotemporal encoder to extract features from the original two-dimensional compressed measurement values ​​and the corresponding three-dimensional modulation mask. The lightweight spatiotemporal encoder introduces a hierarchical physical mask and uses the temporal 1D convolution features as dynamic gating weights for spatial 2D convolution to extract the underlying physical fidelity constraint features.

[0014] The fast decoding module for mask-gated constraints uses a lightweight spatiotemporal decoder to decode the underlying physical fidelity constraint features output by the feature extraction module of the mask-gated constraints, generating an initial reconstruction result containing the basic physical contours. The lightweight spatiotemporal decoder is symmetrical with the lightweight spatiotemporal encoder.

[0015] The multimodal large language model fusion module uses a pre-trained multimodal large language model to output the initial reconstruction result as text features containing high-dimensional semantic understanding of the scene. The text prompts and text features of the specific modality introduced are embedded in the channel dimension after passing through the text encoder and then concatenated and spliced ​​to form a hybrid semantic feature with modality awareness.

[0016] The dual-condition guided sparse controlled diffusion Transformer module uses the controllable diffusion large model to output the extracted feature results as the denoising prediction results. The controllable diffusion large model uses the mixed semantic features as the global semantic guidance condition of the diffusion Transformer backbone network and the underlying physical fidelity constraint features as the structural guidance condition of the sparse controlled bypass of the diffusion Transformer backbone network. The outputs of the two are added together to obtain the denoising prediction results.

[0017] A further preferred embodiment is the mathematical expression of the physical observation model, as follows:

[0018] ;

[0019] In the formula, This is a two-dimensional compression measurement value. ; To unify the temporal or spectral depth dimension after mapping; For element-wise multiplication; This is system noise; For the first The three-dimensional modulation mask corresponding to each slice; The first high-dimensional target signal to be reconstructed One slice;

[0020] When converting to color video, directly... As input;

[0021] When dealing with grayscale video, a single-channel grayscale frame is copied three times along the channel dimension and then used as input. ;

[0022] For hyperspectral imaging, three consecutive adjacent spectral bands are treated as a single color video frame, combined and mapped, and then used as input. .

[0023] In a further preferred embodiment, the lightweight spatiotemporal encoder consists of six layers of lightweight spatiotemporal coding modules connected in series, with each layer of lightweight spatiotemporal coding modules consisting of one spatial... 2D convolution and time It consists of 1D convolutions, and an adaptive cross-dimensional gating mechanism for masking is embedded inside each lightweight spatiotemporal coding module;

[0024] The 3D position encoding module converts discrete pixel coordinates and the compression depth after mapping Mapped to a continuous high-frequency sinusoidal space, the expression is: ;

[0025] The specific operation of the lightweight spatiotemporal encoder is as follows:

[0026] Initial 3D mask As the feature extraction network deepens, specific lightweight masked pooling or convolutions are used. Layer-by-layer downsampling generates hierarchical mask features aligned with the feature resolution of the current layer. ;

[0027] In the lightweight spatiotemporal encoder In the layer, the hidden features output from the previous layer are first processed. Physical mask features of the current layer Element-wise multiplication is performed to simulate the physical process of optical modulation in the hardware, and then 1D convolution is used to extract the temporal / spectral dynamic features that are explicitly constrained by the mask along the depth dimension. :

[0028] ;

[0029] Generate cross-dimensional dynamic gating weights based on physical constraint features :

[0030] ;

[0031] Dynamic gating activation and residual fusion of spatial features extracted by 2D convolution using controlled weights:

[0032] ;

[0033] In the formula, For time 1D convolutional networks; For element-wise multiplication; For global average pooling; It is a two-layer fully connected network; This is the Sigmoid activation function, used to normalize the weights to a value between 0 and 1. It is the tensor product; For space 2D convolutional networks; For the first The spatiotemporal features of the fused temporal and spatial information output by the lightweight spatiotemporal coding module;

[0034] A convolution with a spatial sliding stride of 2 is applied after the third layer of lightweight spatiotemporal coding module;

[0035] Finally, output the underlying physical fidelity constraint features. This serves as a structural guiding condition for a large-scale controlled diffusion model.

[0036] A further preferred embodiment is that the expression of the multimodal large language model is:

[0037] ;

[0038] The expression for the fusion process of specific modality text prompts and text feature outputs is as follows:

[0039] ;

[0040] In the formula, Text features that incorporate high-dimensional semantic understanding of the scene; For multimodal large language models; This is the initial reconstruction result containing the basic physical contours; For hybrid semantic representations with modality awareness capabilities; To perform cascading and splicing at the channel dimension; For text encoders; Text prompts for a specific modality.

[0041] In a further preferred embodiment, the backbone network of the controllable diffusion large model consists of several cascaded DiT modules. One controllable module is set every four DiT modules, and one controllable module is set at the last DiT module. All controllable modules constitute a controller, and the number of layers of each controllable module is 1 / 3 of that of one DiT module.

[0042] The specific operation of the controllable diffusion large-scale model is as follows:

[0043] The total number of layers in the backbone network of the controllable diffusion large model is The index of the DiT module currently being processed is The input hidden state features of the current processing layer are ; in the In each denoising iteration step;

[0044] T01. Dense Modulation of Global Semantic Conditions: Hybrid Semantic Representation Adaptive layer normalization is used to globally scale and translate each layer of the backbone network of the controllable diffusion large model:

[0045] ;

[0046] In the formula, and For the first Layer by The affine transformation parameters obtained from the regression; Features after global scaling and translation adjustment; Normalization for adaptive layers; For element-wise multiplication;

[0047] T02, Sparse Local Correction of Underlying Physical Conditions: Control Gating Indicator Function as follows:

[0048] ;

[0049] Underlying physical conditions The computation is performed only when the indicator function is activated, using a lightweight and controllable module, and local feature correction increments are generated through zero convolution. :

[0050] ;

[0051] In the formula, For modulo operation; It is a lightweight and controllable module; It is a zero-convolutional network;

[0052] T03, Sparse Residual Injection and Backbone Forward Propagation: Modifying Local Features Incrementally Add the residuals to the output of the backbone network:

[0053] ;

[0054] In the formula, This forms the backbone network of a large-scale controllable diffusion model. The output hidden state features of the current processing layer;

[0055] go through After forward propagation of the stacked layers, the final output is the denoised prediction result for the current time step. .

[0056] A further preferred embodiment is the overall loss function of the controllable diffusion large-scale model, as follows:

[0057]

[0058] In the formula, The mean square error between the predicted value and the actual signal; This is the expected value; To use the feature extraction module Extracting feature representations of the real high-dimensional target signal X. ; For network parameters The DiT module at that time; In order to spread forward, to Add step size The feature representation obtained from Gaussian noise; It is the square of the L2 norm.

[0059] On the other hand, this invention discloses a single-exposure compressed imaging reconstruction method that integrates a pre-trained language model, comprising:

[0060] S01. Based on the physical dimension unified mapping mechanism, all modal data are converted into standard pseudo-color video sequences. The physical observation model is used to output the pseudo-color video sequences and the corresponding three-dimensional modulation mask as the original two-dimensional compressed measurement values.

[0061] S02. Use a lightweight spatiotemporal encoder to extract features from the original two-dimensional compressed measurement values ​​and the corresponding three-dimensional modulation mask to obtain the underlying physical fidelity constraint features, which serve as structural guiding conditions for the controllable diffusion large model.

[0062] S03. Use a lightweight spatiotemporal decoder to decode the output of the lightweight spatiotemporal encoder to generate an initial reconstruction result containing the basic physical contours.

[0063] S04. Input the initial reconstruction results into the pre-trained multimodal large language model to obtain text features containing high-dimensional semantic understanding of the scene. Introduce text prompts of specific modalities. After the text prompts and text features are processed by the text encoder, the embedded features are concatenated and spliced ​​in the channel dimension to form a hybrid semantic representation with modality awareness, which serves as the global semantic guidance condition for the controllable diffusion large model.

[0064] S05. Construct and train a controllable diffusion large model. The controllable diffusion large model connects several DiT modules in series as the backbone network. Every four DiT modules and the last DiT module are set as a sparse controlled bypass. Hybrid semantic features are used as global semantic guidance conditions for the backbone network. The underlying physical fidelity constraint features are used as structural guidance conditions for the sparse controlled bypass. The output of the backbone network is added to the local feature correction increment obtained by sparse computation to obtain the denoising prediction result.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. This invention constructs a single and universal network architecture that can be simultaneously compatible with various single-exposure compression imaging systems such as video and hyperspectral imaging with different compression rates. This architecture unifies the data and model parameters for multiple imaging tasks, greatly simplifying the development process of new imaging systems.

[0067] 2. This invention unifies the cross-modal data mapping mechanism, transforming color / grayscale videos and hyperspectral images into standard pseudo-color video sequences. Combined with a unified mathematical observation model, it achieves compatibility of a single network with multiple modalities. In the feature extraction stage, three-dimensional positional encoding and hierarchical physical masking are introduced, and temporal 1D convolutional features are used as dynamic gating weights for spatial 2D convolution to accurately simulate the optical modulation process at the hardware end, thereby extracting high-fidelity underlying physical features.

[0068] 3. In the field of compression reconstruction, this invention uses DiT to replace the traditional U-Net. By utilizing its self-attention mechanism with the same spatial resolution, it perfectly preserves high-frequency physical features such as spectral absorption peaks and fast-moving edges. At the same time, it uses high-level semantic conditions and lightweight control modules to achieve highly efficient local feature correction.

[0069] 4. This invention combines artificial and multimodal large language model dual prompt word design, with dynamic adjustment of three-dimensional position encoding, giving the model a powerful zero-sample generalization ability; when faced with a new imaging modality or an unknown compression rate, the system can directly and adaptively output high-precision reconstruction results without fine-tuning network parameters. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the structure of the single-exposure compression imaging reconstruction system of the present invention;

[0071] Figure 2 This is a schematic diagram of the structure of the feature extraction module and the fast decoding module of the present invention;

[0072] Figure 3 This is a schematic diagram of the network structure of the controllable diffusion large model of the present invention;

[0073] Figure 4 This is a schematic flowchart of the single-exposure compression imaging reconstruction method of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0075] like Figure 1As shown, this invention addresses the limitation of existing single-exposure compressed imaging reconstruction algorithms, which often suffer from "task silos"—meaning they can only decode a single imaging modality. It proposes a large-scale single-exposure compressed imaging reconstruction framework based on "physical constraints, semantic guidance, and sparse controlled diffusion." First, cross-modal compressed data is uniformly mapped to align the data stream. Then, a lightweight mask-gated constraint feature extraction module extracts high-fidelity underlying physical features, while a cascaded pre-trained multimodal large language model (MLLM) generates global semantic representations. In the controlled diffusion large-scale model denoising and reconstruction stage, the framework innovatively employs an asymmetric sparse controlled diffusion Transformer (DiT) architecture, reducing the problem of losing high-frequency details in traditional U-Net denoising architectures. Furthermore, by reducing the number of controllable modules and layers, computational redundancy is significantly reduced. Through this hardware-software co-design, 3D positional encoding mechanism, and textual prompt word design, a single model can adaptively reconstruct high-precision images with varying imaging modalities and compression rates.

[0076] like Figure 1-3 As shown, this invention discloses a single-exposure compressed imaging reconstruction system that integrates a pre-trained language model, comprising:

[0077] The multimodal data alignment and physical observation module uses a unified physical dimension mapping mechanism to convert color video, grayscale video, and hyperspectral images into a standard pseudo-color video sequence. The physical observation model then outputs the standard pseudo-color video sequence as aligned original two-dimensional compressed measurement values.

[0078] The feature extraction module for mask gating constraints uses a lightweight spatiotemporal encoder to extract features from the original two-dimensional compressed measurement values ​​and the corresponding three-dimensional modulation mask. The lightweight spatiotemporal encoder introduces a hierarchical physical mask and uses the temporal 1D convolution features as dynamic gating weights for spatial 2D convolution to extract the underlying physical fidelity constraint features.

[0079] The fast decoding module for mask-gated constraints uses a lightweight spatiotemporal decoder to decode the underlying physical fidelity constraint features output by the feature extraction module of the mask-gated constraints, generating an initial reconstruction result containing the basic physical contours. The lightweight spatiotemporal decoder is symmetrical with the lightweight spatiotemporal encoder.

[0080] The multimodal large language model fusion module uses a pre-trained multimodal large language model to output the initial reconstruction result as text features containing high-dimensional semantic understanding of the scene. The text prompts and text features of the specific modality introduced are embedded in the channel dimension after passing through the text encoder and then concatenated and spliced ​​to form a hybrid semantic feature with modality awareness.

[0081] The dual-condition guided sparse controlled diffusion Transformer module uses the controllable diffusion large model to output the extracted feature results as the denoising prediction results. The controllable diffusion large model uses the mixed semantic features as the global semantic guidance condition of the diffusion Transformer backbone network and the underlying physical fidelity constraint features as the structural guidance condition of the sparse controlled bypass of the diffusion Transformer backbone network. The outputs of the two are added together to obtain the denoising prediction results.

[0082] A further implementation involves acquiring the raw two-dimensional compressed measurement values ​​obtained by the computational imaging system. and the corresponding three-dimensional modulation mask To break down data barriers between different imaging modalities, a unified physical dimension mapping mechanism was designed at the input end to convert all modalities into a standard pseudo-color video sequence format.

[0083] For color videos, directly use as enter;

[0084] For grayscale video, a single-channel grayscale frame is copied three times along the channel dimension to achieve format uniformity. ;

[0085] For hyperspectral imaging (HSI), three consecutive adjacent spectral bands are treated as a single color video frame and combined for mapping. ;

[0086] It is the space of real numbers; The number of frames in the video sequence; This refers to the resolution of the space, specifically the height; The resolution of the space, specifically the width; This represents the total number of spectral bands in hyperspectral imaging. The input is a single-channel grayscale video sequence; This is a grayscale video sequence that conforms to a uniform format after channel copying; For input single-channel hyperspectral image sequences; It is a pseudo-color hyperspectral video sequence after spectral band grouping and mapping.

[0087] Then, a unified physical observation model is established, the mathematical expression of which is as follows:

[0088] ;

[0089] In the formula, These are the raw two-dimensional compressed measurement values ​​that the imaging system needs to acquire. ; To unify the temporal or spectral depth dimension after mapping; For element-wise multiplication; This is system noise; For the first The three-dimensional modulation mask corresponding to each slice ; The first high-dimensional target signal to be reconstructed Each slice. The aligned data serves as the unified input for subsequent modules.

[0090] Further implementation methods, such as Figure 2 As shown, in order to efficiently extract features from different modal inputs, a mask-gated constraint feature extraction module was designed, which utilizes a lightweight spatiotemporal encoder to extract features from two-dimensional compressed measurements. and 3D modulation mask Fast feature extraction is performed. The lightweight spatiotemporal encoder consists of 6 layers of lightweight spatiotemporal coding modules connected in series. Each layer of lightweight spatiotemporal coding module consists of one spatial... 2D convolution and time It consists of 1D convolutions, and embeds a mask adaptive cross-dimensional gating mechanism inside each lightweight spatiotemporal coding module.

[0091] The feature extraction module for mask-gated constraints is front-end equipped with a 3D position encoding module, which is then injected. To adapt to different compression ratios and resolutions, the implementation process is as follows:

[0092] First, the measured value and the mask are multiplied to obtain the input fuzzy 3D signal. Discrete pixel space coordinates Mapped compression depth index (Time or Spectral Dimension) Mapped to A continuous interval is used to eliminate the effects of different resolutions and compression ratios:

[0093] ;

[0094] Then, multi-scale frequency coding is performed on the normalized coordinates. For any coordinate component... Its encoding function Defined as:

[0095] ;

[0096] in, This is a hyperparameter that determines the bandwidth of the position coding.

[0097] Finally, the three-dimensional encoding vectors are concatenated to obtain... and using a linear projection layer ( Align its dimensions to the input fuzzy 3D signal. The number of channels is determined, and finally, element-by-element addition is performed:

[0098] ;

[0099] ;

[0100] In the formula, Represents normalized three-dimensional continuous coordinates; The number of frequency layers determines the level of precision in position perception. This indicates that the encoded vectors of the three dimensions are concatenated; This indicates a 1×1 convolutional or fully connected layer used for aligning the number of channels, ensuring that positional features can be integrated with the blurred 3D signal. Add them directly. The output after injecting location information serves as the input for the subsequent feature extraction module.

[0101] The specific operation of the lightweight spatiotemporal encoder is as follows:

[0102] (1) Hierarchical transfer and alignment of mask features: To introduce accurate physical priors at each layer, enabling the feature extraction network to adapt to different imaging modalities. Initial 3D mask. As the feature extraction network deepens, specific lightweight masked pooling or convolutions are used. Layer-by-layer downsampling generates hierarchical mask features aligned with the feature resolution of the current layer. .

[0103] (2) Full-layer cross-dimensional gating operation based on physical mask constraints: in the lightweight spatiotemporal encoder In the layer, the hidden features output from the previous layer are first processed. Physical mask features of the current layer Element-wise multiplication is performed to simulate the physical process of optical modulation in the hardware, and then 1D convolution is used to extract the temporal / spectral dynamic features that are explicitly constrained by the mask along the depth dimension. :

[0104] ;

[0105] Based on this physical constraint characteristic (temporal / spectral dynamic characteristics explicitly constrained by the mask) Generate cross-dimensional dynamic gating weights :

[0106] ;

[0107] This dynamically controlled weight is used to perform dynamic gating activation and residual fusion on the spatial features extracted by 2D convolution:

[0108] ;

[0109] In the formula, For time 1D convolutional networks; For element-wise multiplication; For global average pooling; It is a two-layer fully connected network; Use the Sigmoid activation function; It is the tensor product; For space 2D convolutional networks; For the first The spatiotemporal features of the fused time and space information output by the lightweight spatiotemporal coding module.

[0110] This operation adaptively redistributes feature responses between space and depth at each feature level of the feature extraction network, greatly enhancing the expressive power of mask features under different modalities.

[0111] (3) Physical condition generation: After the lightweight spatiotemporal coding module in the third layer, a convolution with a spatial sliding stride of 2 is added to reduce the complexity of the feature extraction module and the input resolution of the large controllable diffusion model. At the same time, excessive downsampling is not performed to retain more realistic details.

[0112] Finally, the final layer output of the fast encoding module for mask-gated constraints is the underlying physical fidelity constraint feature. This serves as a structural guiding condition for a large-scale controlled diffusion model.

[0113] In a further implementation, the lightweight spatiotemporal decoder of the mask-gated constraint fast decoding module is symmetrical to the lightweight spatiotemporal encoder. Similarly, mask constraint gating operations are introduced in each layer of the lightweight spatiotemporal decoding module. After the third layer of the lightweight spatiotemporal decoding module, a deconvolution with a spatial interaction stride of 2 is used to quickly restore the output features of the mask-gated constraint feature extraction module to the original spatial size, generating an initial reconstruction result containing the basic physical contours. .

[0114] A further implementation involves using the initial reconstruction results containing the basic physical contours. Inputting these features into a pre-trained multimodal large language model, such as Qwen2.5-VL or CogVLM, automatically extracts text features containing high-dimensional semantic understanding of the scene. :

[0115] ;

[0116] In the formula, Text features that incorporate high-dimensional semantic understanding of the scene; For multimodal large language models; This is the initial reconstruction result containing the basic physical contours;

[0117] Then introduce modal-specific text prompts. Examples of text prompts, such as "reconstructing grayscale videos of fast motion" or "restoring hyperspectral absorption peak features," will include specific modalities. Text features with high-dimensional semantic understanding of the scene The language features are embedded separately after passing through a text encoder, and then concatenated and spliced ​​along the channel dimension to form a hybrid semantic representation with modality awareness. :

[0118] ;

[0119] In the formula, This is a hybrid semantic representation with modality awareness, which serves as a global semantic guidance condition for the DiT network. To perform cascading and splicing at the channel dimension; For text encoders, CLIP text encoders can be used; Text prompts for a specific modality.

[0120] In a further implementation, the backbone network of the controllable diffusion large-scale model consists of several cascaded DiT modules. A controllable module is set every four DiT modules, and a controllable module is set corresponding to the last DiT module. All controllable modules constitute a controller, and the number of layers in each controllable module is one-third that of a single DiT module. DiT can use different imaging modalities and train quickly. This is similar to a "low-rank adaptive" approach, which significantly reduces additional computational cost and avoids the constraints of underlying physical fidelity features. Overfitting leads to degraded generated textures.

[0121] The specific operation of the controllable diffusion large-scale model is as follows:

[0122] The total number of layers in the backbone network of the controllable diffusion large model is (That is, the total number of DiT modules is) The index of the DiT module currently being processed is... The input hidden state features of the current processing layer are Input of the first-level DiT module , For the first The input latent features of the denoising iteration step. In each denoising iteration step:

[0123] T01, Dense Modulation of Global Semantic Conditions (Executed at Each Layer): Hybrid Semantic Representation Adaptive layer normalization is used to globally scale and translate each layer of the backbone network of the controllable diffusion large model:

[0124] ;

[0125] In the formula, and For the first Layer by The affine transformation parameters obtained from the regression; Features after global scaling and translation adjustment; Normalization for adaptive layers; For element-wise multiplication;

[0126] T02. Sparse Local Correction of Underlying Physical Conditions (Executed Only in Specific Layers): To implement the sparse bypass mechanism for inserting controllable modules every four DiT modules and at the end, a control gating indicator function is defined. as follows:

[0127] ;

[0128] Underlying physical fidelity constraint features Only when the indicator function is activated (i.e., in 1 / 4 of specific layer nodes) is computation performed through a lightweight controllable module, and local feature correction increments are generated through zero convolution. :

[0129] ;

[0130] In the formula, For modulo operation; It is a lightweight and controllable module; It is a zero-convolutional network;

[0131] T03, Sparse Residual Injection and Backbone Forward Propagation: Modifying the incremental local features obtained from sparse computation. Add the residuals to the output of the backbone network (this value is 0 for non-control layers):

[0132] ;

[0133] In the formula, This forms the backbone network of a large-scale controllable diffusion model. The output hidden state features of the current processing layer;

[0134] go through After forward propagation of the stacked layers, the final output is the denoised prediction result for the current time step. .

[0135] Further implementation methods, such as Figure 3 As shown, during the training phase, the feature extraction module (which shares the network structure and weights with the feature extraction module under masked gating constraints) extracts the feature representation of the real high-dimensional target signal X as the supervision target. During the forward diffusion process, towards Add step size Gaussian noise is obtained Denoising network acquisition and Drive controllable DiT output denoising prediction results The overall loss function for training uses the mean squared error between the predicted values ​​and the true signal feature representations.

[0136]

[0137] In the formula, The mean square error between the predicted value and the actual signal; This is the expected value; To use the feature extraction module Extracting feature representations of the real high-dimensional target signal X. ; For network parameters The DiT module at that time; In order to spread forward, to Add step size The feature representation obtained from Gaussian noise; It is the square of the L2 norm.

[0138] Deoptimize DiT parameters using the AdamW optimizer And control the bypass parameters to make the model accurately aligned with the original physical signal.

[0139] During the inference phase, based on specific imaging text cues and location encoding mechanisms, after denoising by multiple diffusion models, the final reconstructed spectrum or video sequence is adaptively output. Then, a fast decoding module is used to obtain the final reconstruction result, without the need to readjust the model for new imaging modalities or compression ratios.

[0140] This invention is the first to employ Diffusion Transformer (DiT) and design a lightweight control mechanism in single-exposure compression reconstruction to overcome the high-frequency information loss bottleneck of traditional U-Net. Diffusion Transformer abandons the downsampling-upsampling structure of U-Net, and DiT adopts a self-attention mechanism with equal spatial resolution to maintain a constant token resolution in the network depth, thus preserving real high-frequency physical information such as spectral absorption peaks and fast-moving edges to the greatest extent.

[0141] On the other hand, such as Figure 4 As shown, this invention discloses a single-exposure compressed imaging reconstruction method that integrates a pre-trained language model, comprising:

[0142] S01. Based on the physical dimension unified mapping mechanism, all modal data are converted into standard pseudo-color video sequences. The physical observation model is used to output the pseudo-color video sequences and the corresponding three-dimensional modulation mask as the original two-dimensional compressed measurement values.

[0143] S02. Use a lightweight spatiotemporal encoder to extract features from the original two-dimensional compressed measurement values ​​and the corresponding three-dimensional modulation mask to obtain the underlying physical fidelity constraint features, which serve as structural guiding conditions for the controllable diffusion large model.

[0144] S03. Use a lightweight spatiotemporal decoder to decode the output of the lightweight spatiotemporal encoder to generate an initial reconstruction result containing the basic physical contours.

[0145] S04. Input the initial reconstruction results into the pre-trained multimodal large language model to obtain text features containing high-dimensional semantic understanding of the scene. Introduce text prompts of specific modalities. After the text prompts and text features are processed by the text encoder, the embedded features are concatenated and spliced ​​in the channel dimension to form a hybrid semantic representation with modality awareness, which serves as the global semantic guidance condition for the controllable diffusion large model.

[0146] S05. Construct and train a controllable diffusion large model. The controllable diffusion large model connects several DiT modules in series as the backbone network. Every 4 DiT modules and the last DiT module, a controllable module is set as a sparse controlled bypass. Hybrid semantic features are used as global semantic guidance conditions for the backbone network. The underlying physical fidelity constraint features are used as structural guidance conditions for the sparse controlled bypass. The output of the backbone network is added to the local feature correction increment obtained by sparse computation to obtain the denoising prediction result.

[0147] The single-exposure compressed imaging reconstruction method and system disclosed in this invention, which integrates a pre-trained language model, breaks through the task barriers of traditional single-exposure compressed imaging models and achieves the unification of multiple single-exposure compressed imaging (SCI) tasks. Through the dual mechanism of "physical constraints of measurement values ​​+ semantic guidance of multimodal large language models", it ensures the physical authenticity of the reconstruction results while guaranteeing extremely high visual quality. It has strong versatility and can be widely adapted to different imaging physical conditions without repeated training.

[0148] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, alterations, substitutions, or variations made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A single-exposure compressed imaging reconstruction system that fuses pre-trained language models, characterized by: include: The feature extraction module for mask-gated constraints uses a lightweight spatiotemporal encoder to extract features from two-dimensional compressed measurements and corresponding three-dimensional modulation masks in order to extract underlying physical fidelity constraint features. The fast decoding module for mask-gated constraints uses a lightweight spatiotemporal decoder to decode the underlying physical fidelity constraint features and generate an initial reconstruction result containing the basic physical contours. The lightweight spatiotemporal decoder is symmetrical with the lightweight spatiotemporal encoder. The multimodal large language model fusion module uses a multimodal large language model to output the initial reconstruction results as text features containing high-dimensional semantic understanding of the scene. After the text encoder passes the text prompts and text features of a specific modality, the embedded features are concatenated and spliced ​​in the channel dimension to form a hybrid semantic feature with modality awareness. The dual-condition guided sparse controlled diffusion Transformer module uses the controllable diffusion large model to output the extracted feature results as the denoising prediction results. The controllable diffusion large model uses the mixed semantic features as the global semantic guidance condition of the diffusion Transformer backbone network and the underlying physical fidelity constraint features as the structural guidance condition of the sparse controlled bypass of the diffusion Transformer backbone network. The outputs of the two are added together to obtain the denoising prediction results.

2. The single-exposure compressive imaging reconstruction system of fusing pre-trained language models of claim 1, wherein: It also includes a multimodal data alignment and physical observation module, which converts all modal data into a standard pseudo-color video sequence based on a unified physical dimension mapping mechanism, and uses a physical observation model to output the standard pseudo-color video sequence as aligned two-dimensional compressed measurement values. The mathematical expression for the physical observation model is as follows: ; In the formula, This is a two-dimensional compression measurement value. ; To unify the temporal or spectral depth dimension after mapping; For element-wise multiplication; This is system noise; For the first The three-dimensional modulation mask corresponding to each slice; The first high-dimensional target signal to be reconstructed One slice; When converting to color video, directly... As input; When dealing with grayscale video, a single-channel grayscale frame is copied three times along the channel dimension and then used as input. ; For hyperspectral imaging, three consecutive adjacent spectral bands are treated as a single color video frame, combined and mapped, and then used as input. .

3. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 1, characterized in that: The lightweight spatiotemporal encoder consists of six layers of lightweight spatiotemporal coding modules connected in series, with each layer of lightweight spatiotemporal coding modules consisting of one spatial... 2D convolution and time It consists of 1D convolutions, and embeds a mask adaptive cross-dimensional gating mechanism inside each lightweight spatiotemporal coding module.

4. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 3, characterized in that: The lightweight spatiotemporal encoder is equipped with a three-dimensional position encoding module at its front end. This three-dimensional position encoding module encodes discrete pixel coordinates. and the compression depth after mapping It is mapped to a continuous high-frequency sinusoidal space.

5. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 4, characterized in that: The specific operation of the lightweight spatiotemporal encoder is as follows: Initial 3D mask As the feature extraction network deepens, specific lightweight masked pooling or convolutions are used. Layer-by-layer downsampling generates hierarchical mask features aligned with the feature resolution of the current layer. ; In the lightweight spatiotemporal encoder In the layer, the hidden features output from the previous layer are first processed. Physical mask features of the current layer Element-wise multiplication is performed to simulate the physical process of optical modulation in the hardware, and then 1D convolution is used to extract the temporal / spectral dynamic features that are explicitly constrained by the mask along the depth dimension. : ; Generate cross-dimensional dynamic gating weights based on physical constraint features : ; Dynamic gating activation and residual fusion of spatial features extracted by 2D convolution using controlled weights: ; In the formula, For time 1D convolutional networks; For element-wise multiplication; For global average pooling; It is a two-layer fully connected network; Use the Sigmoid activation function; It is the tensor product; For space 2D convolutional networks; For the first The spatiotemporal features of the fused temporal and spatial information output by the lightweight spatiotemporal coding module; A convolution with a spatial sliding stride of 2 is applied after the third layer of lightweight spatiotemporal coding module; Finally, output the underlying physical fidelity constraint features. This serves as a structural guiding condition for a large-scale controlled diffusion model.

6. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 1, characterized in that: The expression for the multimodal large language model is: ; The expression for the fusion process of specific modality text prompts and text feature outputs is as follows: ; In the formula, Text features that incorporate high-dimensional semantic understanding of the scene; For multimodal large language models; This is the initial reconstruction result containing the basic physical contours; For hybrid semantic representations with modality awareness capabilities; To perform cascading and splicing at the channel dimension; For text encoders; Text prompts for a specific modality.

7. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 1, characterized in that: The backbone network of the controllable diffusion large model consists of several cascaded DiT modules. One controllable module is set every four DiT modules, and one controllable module is set at the last DiT module. All controllable modules constitute a controller. The number of layers of each controllable module is 1 / 3 of that of a DiT module.

8. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 7, characterized in that: The specific operation of the controllable diffusion large-scale model is as follows: The total number of layers in the backbone network of the controllable diffusion large model is The index of the DiT module currently being processed is The input hidden state features of the current processing layer are ; in the In each denoising iteration step; T01. Dense Modulation of Global Semantic Conditions: Hybrid Semantic Representation Adaptive layer normalization is used to globally scale and translate each layer of the backbone network of the controllable diffusion large model: ; In the formula, and For the first Layers are composed of hybrid semantic representations The affine transformation parameters obtained from the regression; Features after global scaling and translation adjustment; Normalization for adaptive layers; For element-wise multiplication; T02, Sparse Local Correction of Underlying Physical Conditions: Control Gating Indicator Function as follows: ; Underlying physical conditions The computation is performed only when the indicator function is activated, using a lightweight and controllable module, and local feature correction increments are generated through zero convolution. : ; In the formula, For modulo operation; It is a lightweight and controllable module; It is a zero-convolutional network; T03, Sparse Residual Injection and Backbone Forward Propagation: Modifying Local Features Incrementally Add the residuals to the output of the backbone network: ; In the formula, This forms the backbone network of a large-scale controllable diffusion model. The output hidden state features of the current processing layer; go through After forward propagation of the stacked layers, the final output is the denoised prediction result for the current time step. .

9. The single-exposure compressed imaging reconstruction system based on a fused pre-trained language model according to claim 8, characterized in that: The overall loss function of the controllable diffusion large-scale model is as follows: ; In the formula, The mean square error between the predicted value and the actual signal; This is the expected value; To use the feature extraction module Extracting feature representations of the real high-dimensional target signal X. ; For network parameters The DiT module at that time; In order to spread forward, to Add step size The feature representation obtained from Gaussian noise; It is the square of the L2 norm.

10. A single-exposure compressed imaging reconstruction method integrating a pre-trained language model, characterized in that: Includes the following steps: S01. Based on the physical dimension unified mapping mechanism, all modal data are converted into standard pseudo-color video sequences. The physical observation model is used to output the pseudo-color video sequences and the corresponding three-dimensional modulation mask as the original two-dimensional compressed measurement values. S02. Use a lightweight spatiotemporal encoder to extract features from the original two-dimensional compressed measurement values ​​and the corresponding three-dimensional modulation mask to obtain the underlying physical fidelity constraint features. S03. Use a lightweight spatiotemporal decoder to decode the output of the lightweight spatiotemporal encoder to generate an initial reconstruction result containing the basic physical contours. S04. Input the initial reconstruction results into the multimodal large language model to obtain text features containing high-dimensional semantic understanding of the scene. Introduce text prompts of specific modalities. After the text prompts and text features are processed by the text encoder, the embedded features are concatenated and spliced ​​in the channel dimension to form a hybrid semantic representation with modality awareness. S05. Construct and train a controllable diffusion large model. Connect several DiT modules in series as the backbone network. Set one controllable module as a sparse controlled bypass every four DiT modules and at the last DiT module. Use the hybrid semantic features as the global semantic guidance condition of the backbone network. Use the underlying physical fidelity constraint features as the structural guidance condition of the sparse controlled bypass. Add the backbone network output to the local feature correction increment obtained by sparse computation to obtain the denoising prediction result.

Images