Image compression sensing method of characterizing domain sampling and mixing transformer

Abstract

The application discloses an image compression sensing method of domain sampling and mixed transformer, comprising the following steps: a sampling recovery stage: learning the deep representation of the input image through multi-layer progressive 5*5 and 7*7 convolution and nonlinear activation function, and fully interacting the fragmented window information, and then generating the fine reconstruction iteration stage containing low and high level aggregation features at the same time through the jump connection; a fine reconstruction iteration stage: using a deep gradient descent module to perform optimization update of the reconstruction feature, and the deep gradient descent module expands the calculation of the update fidelity term to the neural network. Through the innovative network architecture and training strategy, the application realizes excellent image reconstruction performance, especially at a low sampling rate, and the method exhibits significant advantages and positive gain effects in detail preservation, texture reconstruction, robustness and universality compared with existing methods, thereby providing strong technical support for research and application in the field of image reconstruction.

Description

Technical Field

[0001] This invention relates to the technical field of image compressed sensing, and more particularly to an image compressed sensing method using representation domain sampling and hybrid Transformer. Background Technology

[0002] Compressive sensing (CS) is a signal processing theory that breaks through the limitations of the traditional Shannon-Nyquist sampling theorem. It can recover sparse or compressible signals from sampled data at significantly lower sampling rates than traditional ones, thereby greatly reducing sampling costs and data storage requirements. Therefore, CS theory has been widely applied in many fields, including snapshot compressed imaging, medical magnetic resonance imaging (MRI), hyperspectral compressed imaging, and laser scanning imaging.

[0003] The core issues of CS methods mainly include sampling and reconstruction strategies. Various sampling matrices, such as random matrices, binary matrices, and structure matrices, have been proposed to address the sampling strategy problem. Regarding reconstruction strategies, convex optimization algorithms, greedy algorithms, and iterative thresholding algorithms are widely used. However, these methods have some drawbacks, such as slow convergence speed and limited ability to model complex signals. These algorithms not only constrain the consistency between the reconstructed and measured values ​​but also typically encourage sparsity of solutions; prior terms often involve sparse operators related to predefined transform bases (e.g., L1 regularization of ISTA, Discrete Cosine Transform (DCT)). These methods perform well in convergence and mathematical analysis but face high computational complexity and poor adaptability. In recent years, the superior capabilities of deep neural networks have spurred the development of various deep compressed sensing (DCS) algorithms. One class of methods employs a purely data-driven CS architecture, directly training the model to learn latent inverse mappings from large amounts of data. Although these methods can automatically solve CS problems, their black-box nature, coupled with a large number of parameters and computational trial and error, makes them highly inefficient. In contrast, the Deep Unfolded Networks (DUNs) strategy advocates for the complementarity of neural networks and optimization algorithms, replacing the manual calculation of optimization / prior terms with learnable parameters of neural networks (CNN / Transformer). Therefore, this DCS method balances reconstruction quality and mathematical interpretability.

[0004] However, current DCS methods generally suffer from three key challenges: Cross-domain feature alignment loss: Compression is performed in the low-level pixel domain, but reconstruction is based on the high-level feature domain, and frequent cross-domain gradient descent leads to significant feature alignment loss. Insufficient multi-scale information fusion: Feature representations fail to fully integrate multi-scale information; they often use traditional proximal projection terms and fail to leverage the fitting and learning capabilities of prior terms. Model volume expansion: DCS models gain only a small improvement in quality at the cost of significant volume expansion, which is highly susceptible to limitations imposed by edge devices (such as laser scanners, MRI machines, etc.).

[0005] Therefore, overcoming these challenges and designing an efficient, lightweight, and high-reconstruction-quality compressed sensing method has become an important research direction. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] In view of the problems existing in the above-mentioned image compressed sensing methods of representation domain sampling and hybrid Transformer, this invention is proposed.

[0008] Therefore, the purpose of this invention is to provide an image compressed sensing method that combines representation domain sampling and hybrid Transformer, and it innovatively designs a representation compressed sensing (RCS) sub-model. Unlike other direct sampling CS methods, RCS advocates first extracting compact feature representations with high information density and rich semantics in the representation space, thereby completing a more efficient signal compression and reconstruction process in the representation domain.

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: an image compressed sensing method using representation domain sampling and hybrid Transformer, comprising the following steps:

[0010] Sampling and restoration stage: The input image is learned through multi-layer progressive 5×5 and 7×7 convolutions and non-linear activation functions. The system provides a deep representation and allows for full interaction of fragmented window information. Then, it generates aggregated features that simultaneously contain low and high levels through skip connections. And based on this, the downsampled Y is obtained;

[0011] Fine-grained reconstruction iteration phase: The deep gradient descent module is used to perform optimized updates of the reconstructed features. The deep gradient descent module expands the calculation of the update fidelity term into the neural network.

[0012]

[0013] Among them, the nonlinear network f res (·) Improve the ability to extract residual information from observations, and and These serve as projection mappings for the observation domain and the depth feature domain, respectively. This is recorded as an intermediate result of the recovery process; As a pseudo-inverse mapping of sample S;

[0014] The near-end projection is replaced by a three-scale sparse denoising sub-network; then, the gradient descent term is finely supplemented and fused through the Transformer module, which includes a cross attention module, a window local attention module, and a FeedForwardNetwork module.

[0015] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer as described in this invention, wherein: in sampling, Divided into non-overlapping blocks of size P×P (P value experimentally set to 32), and processed by unbiased linear convolution. Sampling to obtain compressed value Where r is the sampling rate, the image dimensions are H and W, the convolution kernel size Φ is P×P, the stride is P, and the sampling subnetwork S(·) is represented as:

[0016]

[0017] The encoder is a feature encoder; a deconvolution with parameters shared with the sampling kernel Φ is applied to the compressed result Y. Obtain initial characterization estimation results

[0018] The compressed reconstruction representation is transformed back to the original image domain, and the initial reconstruction subnetwork is then established. Represented as:

[0019]

[0020] Reconstructing subnetworks is the initial pseudo-inverse of the sampling subnetwork S(·); Decoder is the feature decoder.

[0021] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer as described in this invention, the depth gradient descent module first calibrates the depth reconstruction features at each stage using observation domain residual information to ensure optimized features. Consistency with the observed value Y.

[0022] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer described in this invention, the three-scale sparse denoising method performs three downsampling operations on the features using a 2×2 convolution with a stride of 2 and GELU (a specific activation function), keeping the number of channels constant, and successively reducing the feature map size to [a smaller value]. Feature representations at different scales are generated. Then, low-resolution features are upsampled by ×2 using linear interpolation and concatenated with features of the corresponding resolution through residual connections. Finally, feature information is aggregated with a larger window using a 5×5 depthwise convolution.

[0023] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer described in this invention, auxiliary features are introduced during the three-scale sparse denoising process. The residual block combines the optimized features with the original X. (k) Merge along the channel dimension and output via 3×3 convolution.

[0024] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer described in this invention, wherein: the specific input Q of the cross-attention module comes from K and V, and is projected onto... The new component is processed through a Softmax operation (a probabilistic activation function) to generate a transposed attention map of cross-CS level features. Cross-attention module The calculation is defined as follows:

[0025]

[0026] Conv 1×1 This is denoted as linear convolution; in the specific iterations of the network, the pseudo-inverse of the measured features is... Cross-attention is introduced as a component of the Q1 query; while the K1 and V1 key values ​​both come from the reconstruction result components of the current stage.

[0027]

[0028] Calculation through dual-domain feature cross Treating it as a potential error term of the PGD (proximal gradient descent) fidelity constraint term, and then re-inputting it as the K2 and V2 components into the second-stage GCA, along with the current... As the new Q2, a second cross-attention calculation is performed. This refers to the refined supplementary items in the first stage of PGD after modeling the global information, used to guide... Further updates:

[0029]

[0030] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer described in this invention, the FeedForwardNetwork module consists of LayerNorm layers, linear layers, and DConv layers.3×3 Composed of (depth-group convolution).

[0031] As a preferred embodiment of the image compressed sensing method using representation domain sampling and hybrid Transformer as described in this invention, the window local attention module utilizes a sliding window of depthwise convolution to generate a dynamic local attention map A′, and aggregates local features through channel-wise convolution, specifically including:

[0032]

[0033] Among them, f σ It is the GeLU activation function, used to produce a smoother attention distribution and enhance the input. The local information representation capability is obtained, and W' is the projection matrix of the corresponding component; then, the output of the window local attention module is residually connected with the original features, and further processed through a customized feedforward network:

[0034]

[0035] In the loss function, RHT-Net defines the error between image pairs as... And a hybrid loss function is used for end-to-end optimization:

[0036]

[0037] Where N is the number of training sample pairs, and Θ represents all trainable parameters; the ratios of λ′ and λ″ are both set to 0.5 to balance the robustness and convergence speed of the model.

[0038] The beneficial effects of this invention are:

[0039] 1. High Reconstruction Performance: The method of this invention (RHT-Net) achieves superior reconstruction quality compared to competing methods at various sampling rates. Especially at low sampling rates, the model of this invention maintains excellent performance, resulting in reconstructed images with higher clarity and accuracy.

[0040] 2. Detail Preservation and Texture Reconstruction: Compared with other methods, the model of this invention excels in handling detail and texture features. Through the powerful feature extraction and global modeling capabilities of the Representation Domain Sampling (RCS) module and the refined reconstruction module, the RHT-Net model can better recover detailed features in images, providing clearer edge and texture information.

[0041] 3. Robustness and Universality: The RHT-Net model proposed in this invention has been comprehensively tested on multiple benchmark datasets and achieves better reconstruction quality at different sampling rates. Even after Gaussian noise is added to the image, it still outperforms other methods. This indicates that the method has strong robustness and universality, and is suitable for various sampling rates and image scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0043] Figure 1 : Overall design framework diagram of the network model (RHT-Net) of this invention;

[0044] Figure 2 : A schematic diagram of the characterization compressed sensing module (RCS) included in this invention;

[0045] Figure 3 : A schematic diagram of the deep and fine reconstruction stage included in this invention;

[0046] Figure 4 : A quantitative comparison chart of the model proposed in this invention with other advanced methods in publicly available data;

[0047] Figure 5 Visualization of the model proposed in this invention compared with other advanced methods in publicly available data;

[0048] Figure 6 Comparison chart of computational overhead and performance of the model proposed in this invention;

[0049] Figure 7 : A schematic diagram illustrating the robustness of the proposed model against noise decay. Detailed Implementation

[0050] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0053] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0054] Reference Figure 1 - Figure 7 This paper provides an image compressed sensing method using representation domain sampling and hybrid Transformer, comprising the following steps:

[0055] Sampling and restoration stage: The input image is learned through multi-layer progressive 5×5 and 7×7 convolutions and non-linear activation functions. The system provides a deep representation and allows for full interaction of fragmented window information. Then, it generates aggregated features that simultaneously contain low and high levels through skip connections. And based on this, the downsampled Y is obtained;

[0056] Fine-grained reconstruction iteration phase: The deep gradient descent module is used to perform optimized updates of the reconstructed features. The deep gradient descent module expands the calculation of the update fidelity term into the neural network.

[0057]

[0058] Among them, the nonlinear network f res (·) Improve the ability to extract residual information from observations, and and These serve as projection mappings for the observation domain and the depth feature domain, respectively. This is recorded as an intermediate result of the recovery process; As a pseudo-inverse mapping of sample S;

[0059] Near-end projection is replaced by a three-scale sparse denoising subnetwork.

[0060] Then, the gradient descent term is finely supplemented and fused through the Transformer module, which includes the cross attention module, the window local attention module, and the FeedForwardNetwork module.

[0061] In the sampling, Divided into non-overlapping blocks of size P×P (P value experimentally set to 32), and processed by unbiased linear convolution. Sampling to obtain compressed value Where r is the sampling rate, the image dimensions are H and W, the convolution kernel size Φ is P×P, the stride is P, and the sampling subnetwork S(·) is represented as:

[0062]

[0063] The encoder is a feature encoder; a deconvolution with parameters shared with the sampling kernel Φ is applied to the compressed result Y. Obtain initial characterization estimation results

[0064] The compressed reconstruction representation is transformed back to the original image domain, and the initial reconstruction subnetwork is then established. Represented as:

[0065]

[0066] The decoder is a feature decoder; the reconstructed subnetwork Let S be the initial pseudo-inverse of the sampling subnetwork S(·).

[0067] Furthermore, the depth gradient descent module first calibrates the depth reconstruction features at each stage using residual information from the observation domain, ensuring optimized features. Consistency with the observed value Y.

[0068] The three-scale sparse denoising method involves downsampling the features three times using a 2×2 convolution with a stride of 2 and GELU activation, while keeping the number of channels constant, and successively reducing the feature map size to [a smaller value]. Feature representations at different scales are generated. Then, low-resolution features are upsampled by ×2 using linear interpolation and concatenated with features of the corresponding resolution through residual connections. Finally, feature information is aggregated with a larger window using a 5×5 depthwise convolution.

[0069] Specifically, auxiliary features are introduced during the three-scale sparse denoising process. The residual block combines the optimized features with the original X. (k) Merge along the channel dimension and output via 3×3 convolution.

[0070] Furthermore, the specific input Q of the cross-attention module comes from K and V, and is projected onto... The new component is processed through a Softmax operation (a probabilistic activation function) to generate a transposed attention map of cross-CS level features. Cross-attention module The calculation is defined as follows:

[0071]

[0072] Conv 1×1 This is denoted as linear convolution; in the specific iterations of the network, the pseudo-inverse of the measured features is... Cross-attention is introduced as a component of the Q1 query; while the K1 and V1 key values ​​both come from the reconstruction result components of the current stage.

[0073]

[0074] Calculation through dual-domain feature cross Treating it as a potential error term of the PGD (proximal gradient descent) fidelity constraint term, and then re-inputting it as the K2 and V2 components into the second-stage GCA, along with the current... As the new Q2, a second cross-attention calculation is performed. This refers to the refined supplementary items in the first stage of PGD after modeling the global information, used to guide... Further updates:

[0075]

[0076] The FeedForwardNetwork module consists of LayerNorm, linear layers, and DConv. 3×3 Composed of (depth-group convolution); the window local attention module utilizes a sliding window of depth-group convolution to generate a dynamic local attention map A′, and aggregates local features through channel-wise convolution, specifically including:

[0077]

[0078] Among them, f σ It is the GeLU activation function, used to produce a smoother attention distribution and enhance the input. The local information representation capability is obtained, and W' is the projection matrix of the corresponding component; then, the output of the window local attention module is residually connected with the original features, and further processed through a customized feedforward network:

[0079]

[0080] In the loss function, RHT-Net defines the error between image pairs as... And a hybrid loss function is used for end-to-end optimization:

[0081]

[0082] Where N is the number of training sample pairs, and Θ represents all trainable parameters; the ratios of λ′ and λ″ are both set to 0.5 to balance the robustness and convergence speed of the model.

[0083] Example:

[0084] Model Input: In this embodiment of the invention, the BSD500 dataset, containing 400 images, is used as the training set. The training data is obtained by randomly cropping to 96×96 pixels and randomly flipping them, resulting in 80,000 sub-images. To improve computational efficiency and model robustness, the images are converted to the YCbCr color space, and only the Y channel is used for training and testing. The test datasets include Set5, Set11, BSDS100, and Urban100. The test images are center-cropped and resized to 128×128 pixels.

[0085] Training details: The Adam optimizer was used to update the model parameters, with momentum and weight decay set to 0.9 and 0.999, respectively. The initial learning rate was set to 1e-3 and decayed at epochs 60, 90, 120, 150, and 180.

[0086] Training method: The model is trained over 200 epochs using training data with a batch size of 6. The learning rate decay strategy is to multiply the learning rate by 0.3 at the 60th, 90th, 120th, 150th, and 180th epochs.

[0087] Test details: During the test phase, test images with different sampling rates were reconstructed, and the reconstruction quality was evaluated using PSNR and SSIM.

[0088] Experimental platform: The experimental platform uses the PyTorch framework to implement the model and trains it on an Nvidia RTX 3090 GPU.

[0089] High reconstruction performance: The method of this invention (RHT-Net) achieves superior reconstruction quality compared to competing methods at various sampling rates. Especially at low sampling rates, the model of this invention maintains excellent performance, resulting in reconstructed images with higher clarity and accuracy.

[0090] Detail Preservation and Texture Reconstruction: Compared to other methods, the model of this invention excels in handling detail and texture features. Through the powerful feature extraction and global modeling capabilities of the Representation Domain Sampling (RCS) module and the refined reconstruction module, the RHT-Net model is able to better recover detailed features in images, providing clearer edge and texture information.

[0091] Robustness and versatility: The RHT-Net model proposed in this invention has been comprehensively tested on multiple benchmark datasets and achieves better reconstruction quality at different sampling rates. Even after Gaussian noise is added to the image, it still outperforms other methods. This demonstrates that the proposed method has strong robustness and versatility, and is suitable for various sampling rates and image scenarios.

[0092] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An image compressed sensing method using representation domain sampling and hybrid Transformer, characterized in that, Includes the following steps: Sampling recovery phase: through multiple progressive layers and Convolution and nonlinear activation functions learn the input image The system provides a deep representation and allows for full interaction of fragmented window information. Then, it generates aggregated features that simultaneously contain low and high levels through skip connections. And on this basis, downsampling is obtained. ; Fine-grained reconstruction iteration phase: The deep gradient descent module is used to perform optimized updates of the reconstructed features. The deep gradient descent module expands the calculation of the update fidelity term into the neural network. ； Among them, nonlinear networks To improve the ability to extract residual information from observations, and and These serve as projection mappings for the observation domain and the depth feature domain, respectively. This is recorded as an intermediate result of the recovery process; As a sample The pseudo-inverse mapping; Near-end projection is replaced by a three-scale sparse denoising subnetwork. Then, the gradient descent term is finely supplemented and fused through the Transformer module, which includes the cross attention module, the window local attention module, and the FeedForwardNetwork module.

2. The image compressed sensing method using representation domain sampling and hybrid Transformer as described in claim 1, characterized in that: During sampling, Divided into sizes of Non-overlapping blocks are processed by unbiased linear convolution. Sampling to obtain compressed value ,in That is, the sampling rate, and the image dimensions are... convolution kernel The size is stride is Sampling subnetwork Represented as: ； Represented as a feature encoder; Characterization compression results Applications and sampling kernels Deconvolution with shared parameters Obtain initial characterization estimation results ; The compressed reconstruction representation is transformed back to the original image domain, and the initial reconstruction subnetwork is then established. Represented as: ； Reconstructing subnetworks For sampling subnetwork The initial pseudo-inverse; This is a feature decoder.

3. The image compressed sensing method using representation domain sampling and hybrid Transformer as described in claim 1, characterized in that: The depth gradient descent module first calibrates the depth reconstruction features at each stage using residual information from the observation domain, ensuring optimized features. Compared with observed values Consistency.

4. The image compressed sensing method using representation domain sampling and hybrid Transformer according to claim 1, characterized in that: The three-scale sparse denoising is achieved through a step size of 2. Convolution and GELU perform three downsampling operations on the features, keeping the number of channels constant, and successively reducing the feature map size to [a smaller value]. , , This generates feature representations at different scales, and then uses linear interpolation to process the low-resolution features. Upsampling is then performed and concatenated with features of the corresponding resolution via residual connections. The depthwise convolutions aggregate feature information into a larger window.

5. The image compressed sensing method using representation domain sampling and hybrid Transformer according to claim 4, characterized in that: Introducing auxiliary features during three-scale sparse denoising The residual block combines the optimized features with the original features. Merge along the channel dimension, and through Convolution output.

6. The image compressed sensing method using representation domain sampling and hybrid Transformer according to claim 1, characterized in that: The specific input of the cross-attention module From They were projected onto New portion, after The operation generates a transposed attention map of cross-level features. Cross-attention module The calculation is defined as follows: This is denoted as linear convolution; in the specific iterations of the network, the pseudo-inverse of the measured features is... As The query component introduces cross-attention calculation; and and The key values ​​are all derived from the reconstruction result components of the current stage. : Calculation through dual-domain feature cross This is considered as a potential error term of the PGD fidelity constraint term, and then used as... and The component is re-input into the second stage of GCA, and compared with the current... As a new Perform a second cross-attention calculation, at this time This refers to the refined supplementary items in the first stage of PGD after modeling the global information, used to guide... Further updates will follow, subsequently via Perform feedforward feature fusion: 。 7. The image compressed sensing method using representation domain sampling and hybrid Transformer according to claim 6, characterized in that: The FeedForwardNetwork module Composed of LayerNorm layers, linear layers and composition.

8. The image compressed sensing method using representation domain sampling and hybrid Transformer according to claim 7, characterized in that: The window local attention module utilizes a sliding window of depthwise convolution to generate dynamic local attention maps. And it aggregates local features through channel-wise convolution, specifically including: in, It is the GeLU activation function, used to produce a smoother attention distribution and enhance the input. The local information representation capability is then utilized. Next, the output of the window local attention module is residually connected to the original features, and further processed through a customized feedforward network. In the loss function, RHT-Net defines the error between image pairs as... And end-to-end optimization is performed using a hybrid loss function: in, It is the number of training sample pairs. Represents all trainable parameters; and The ratios were all set to 0.5 to balance the robustness and convergence speed of the model.

Images