Compressed sensing iterative reconstruction method combined with image hierarchical features

Abstract

The invention provides a compressed sensing iterative reconstruction method combining image level features. Relates to the technical field of graphic image processing. The image reconstruction quality is gradually improved by alternately using a gradient descent operation and a feature optimization operation. In each iteration stage, firstly, a multi-scale feature enrichment module MFEM is constructed, denoising of different scales is carried out on a reconstructed image in the previous stage, and the reconstructed image is enhanced into a multi-channel feature map. And then updating fine-grained information of the feature map by adopting gradient descent operation. Secondly, in order to enhance the perception capability of the network for high-resolution images and reduce the calculation complexity, a secondary spatial information capture module SSCM is constructed, and image level features are modeled through a lightweight CNN-Transform hybrid architecture and are deeply optimized. And finally, non-linear enhancement is performed on the feature information optimized by the SSCM through a post-processing module PPM, and a reconstructed image of the stage is obtained. After multiple iterations, a final reconstructed image can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of graphic image processing technology, and in particular to a compressed sensing iterative reconstruction method combining image level features. Background Art

[0002] The traditional ICS reconstruction algorithm solves the sparse regularization problem through iterative optimization to obtain the reconstructed signal. Among them, the alternating direction multiplier method decomposes a large optimization problem into several small sub-problems, solves each sub-problem separately through alternating iterations, and introduces a multiplier in each iteration to update the estimated value of the solution, thereby obtaining the solution to the original problem; the approximate message passing method is based on a probabilistic graphical model, and integrates multiple denoising methods in the iterative process to ensure that the signal disturbance at each iteration is close to the Gaussian white noise to be removed, thereby restoring the original signal; the iterative shrinkage threshold algorithm transforms the original problem into a first-order approximation problem, and continuously updates the estimated value of the solution vector through gradient descent operations and soft threshold operations, thereby gradually approaching the optimal solution to the original problem. However, these methods rely heavily on manually designed parameters, and the image reconstruction quality is poor.

[0003] Thanks to the powerful learning ability of CNN and Transformer, some scholars have introduced them into the ICS field to improve the quality of image reconstruction. CNN-based ICS reconstruction algorithms are mainly divided into two categories: deep feedforward algorithm and deep unfolding algorithm. The former is not restricted by any artificially designed parameters and reconstructs images through adaptive learning. ReconNet combines CNN with ICS for the first time, uses a fixed random Gaussian matrix to linearly sample image blocks, and introduces a BM3D denoiser to optimize the reconstructed image. Shi et al. proposed the CSNet series of networks, which uses a full-image reconstruction strategy to suppress the block effect caused by block sampling, and constrains the sampling matrix to improve signal transmission efficiency. DUSANet proposed by Tian Jinpeng et al. learns the high-frequency texture information of the image through the self-attention mechanism, and uses residual connections to effectively solve the gradient vanishing problem caused by the increase in network depth. These algorithms directly use CNN to map the measurement value to an image, which has poor interpretability. To address this problem, scholars proposed a deep unfolding algorithm to map the traditional iterative optimization algorithm into a deep network, thereby improving the image reconstruction quality and improving the interpretability of the algorithm. Zhang et al. proposed AMPNet by expanding the iterative denoising process of the AMP algorithm, using sampling matrices and CNN to fit image differences to characterize image prior information. During the reconstruction process, the network also integrates a de-artifacting module to eliminate the block effect of the reconstructed image. ISTANet+ implements the end-to-end ISTA algorithm through a deep network, effectively solving the neighbor mapping problem associated with nonlinear sparse transformation. On this basis, ISTANet++ introduces more prior knowledge to the ICS reconstruction network, handles the reconstruction task under multiple sampling rates through a dynamic expansion strategy, and adopts a cross-block strategy to reduce the block effect and improve the image reconstruction quality. OPINENet+ introduces sampling matrix orthogonal constraints and binary constraints to regulate the regularization term of the optimization problem, and optimizes image blocks through adaptive sampling and joint reconstruction. Chen Wenjun et al. proposed FSOINet, which uses the proximal gradient descent method to map the image to the feature domain and establishes an iteratively optimized information flow channel to achieve image reconstruction. The above algorithms significantly improve the image reconstruction quality compared with traditional iterative optimization algorithms, but are still constrained by the CNN architecture to extract local information of the image, and it is difficult to capture the long-distance dependencies of high-resolution images.

[0004] Transformer was originally designed for natural language processing (NLP) to process long-distance dependencies of sequence data. Its internal self-attention is considered to be a non-local filtering operation suitable for image processing. Inspired by the powerful performance of Transformer in NLP, ViT pioneered the extension of this architecture to the image field. Swin Transformer designed a multi-head self-attention with a shift mechanism, which effectively reduced the computational cost. In recent years, Transformer has also been applied to the ICS field. TransCS uses an iterative Transformer backbone in the reconstruction module to model the long-distance dependencies of images, and introduces an auxiliary CNN to capture the local features of images. However, it uses both the Transformer encoder and decoder in the deep reconstruction network, resulting in a significant increase in the number of network parameters and image reconstruction time. CSformer uses a multi-path feedforward method to map the measurement values ​​to the initialization backbone, CNN backbone, and Transformer backbone, aiming to explore the local features and global representation of the image, and simultaneously calculates the fine-grained and long-distance dependencies of the image through a progressive strategy, effectively improving the performance of the reconstruction algorithm, but the complex network structure also further increases the image reconstruction time.

[0005] The prior art has the following defects:

[0006] (1) In the task of high-resolution image reconstruction, the CNN-based ICS reconstruction algorithm is limited by the local receptive field of the convolution kernel, making it difficult to capture long-distance image dependencies, which limits the network's ability to interact with spatial information on a global scale, resulting in the overall effect of the reconstructed image failing to meet the ideal visual quality and information integrity standards.

[0007] (2) The ICS reconstruction algorithm using Transformer can model long-distance image dependencies and allow each element in the image (including pixels, features, etc.) to interact with other elements in the global scope, thereby improving the quality of the reconstructed image. However, the introduction of the Transformer architecture has led to an exponential increase in the number of network parameters and image reconstruction time, which has put forward higher requirements on hardware resources and may become a bottleneck in practical applications, affecting user experience and system response speed.

[0008] To address the above shortcomings, the present invention proposes a Combining Image Hierarchical-Feature Network (CHFNet), which solves the problem that the introduction of Transformer causes the number of network parameters and image reconstruction time to increase exponentially, while achieving the purpose of high-quality reconstruction of high-resolution images and reducing the reconstruction time. Summary of the invention

[0009] The purpose of the present invention is to solve the above-mentioned problem and to provide a compressed sensing iterative reconstruction method combining image level features.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] According to one aspect of the present invention, a compressed sensing iterative reconstruction method for joint image-level features is provided, including introducing prior information of the image in the spatial dimension, and constructing a CHFNet overall structure, characterized in that the constructed CHFNet overall structure includes three parts: a sampling subnetwork, an initial reconstruction subnetwork, and a hybrid reconstruction subnetwork:

[0012] The sampling subnetwork samples the original image through a learnable sampling matrix Sampling block by block to get the measured value , as follows:

[0013] The image is sampled using a learnable sampling matrix Perform linear sampling and transform the sampling matrix Recombination The size is , the step length is The convolution kernel without bias , and optimize the matrix parameters through training data so that the measured values ​​retain more non-redundant information. Therefore, the sampling subnetwork can be expressed as:

[0014]

[0015] in: Represents a sampling operation, represents the convolution operation, represents the compressed sensing measurement value;

[0016] The initial reconstruction subnetwork performs a linear transformation on the measured values ​​to obtain the initial restored image. , as follows:

[0017] By transposing the sampling matrix To complete the linear transformation from the measured value to the initial restored image. Similar to the sampling subnetwork, Reorganized into indivual Convolution kernel size , and then the initial restored image is obtained through the "PixelShuffle" operation , this process is expressed as:

[0018]

[0019] in: Represents an initial refactoring operation.

[0020] The hybrid reconstruction subnetwork designs an iterative strategy to gradually improve the image reconstruction quality by alternating between gradient descent operations and feature optimization operations. The specific operations of each iterative stage are as follows:

[0021] Firstly, a multi-scale feature enrichment module (MFEM) is constructed to denoise the reconstructed image at different scales and enhance it into a multi-channel feature map. Then, a gradient descent operation is used to update the fine-grained information of the feature map. Taking the iteration stage as an example, the information flow process can be expressed as:

[0022]

[0023]

[0024] Among them: of Indicates The reconstructed image obtained in the iterative stage, represents a multi-scale feature enrichment operation, Represents a multi-channel feature map; represents the gradient descent operation, Adaptive step size for gradient descent operation, implemented through two convolutional layers; It is a feature map containing high-grained information.

[0025] Secondly, in order to enhance the network's perception of high-resolution images and reduce computational complexity, a secondary spatial information capture module SSCM is designed to model image-level features through a lightweight CNN-Transformer hybrid architecture and perform deep optimization. Finally, the feature information optimized by the secondary spatial information capture module SSCM is nonlinearly enhanced through the post-processing module PPM, and the reconstructed image of this stage is obtained while retaining the key information required for reconstruction. SSCM and PPM are called feature optimization operations, and their mathematical expressions are as follows: ,Mode As shown:

[0026]

[0027]

[0028] in: Respectively represent the secondary spatial information capture operation and post-processing operation, Indicates the output information of SSCM. Represents the reconstructed image at this stage.

[0029] go through After iterations, the final reconstructed image can be obtained. .

[0030] Furthermore, the multi-scale feature enrichment module MFEM can integrate the prior information of different scales, effectively remove the noise of the reconstructed image in the previous stage, and generate image features rich in multi-scale details. MFEM adopts a double-layer convolution structure in each scale branch. The first layer outputs 2C-dimensional features, and the second layer outputs C-dimensional features. The convolution kernel sizes are respectively , , and After obtaining the output features of each scale branch, they are fused through the "Concat" operation and the convolution operation to obtain a high signal-to-noise ratio multi-channel feature map while simplifying the computational complexity of subsequent operations. The information flow process of MFEM can be expressed as:

[0031]

[0032] in: represents a multi-scale convolution operation, Represents the convolution kernel in the feature fusion stage.

[0033] Furthermore, the secondary spatial information capture module SSCM is modeled and optimized through the CNN-Transformer hybrid architecture. The local branch cascades two standard ResBlock blocks, and the model performance is improved by introducing jump connections. A lightweight hybrid attention block CBAM is introduced in the regional branch, and before executing the CBAM block, the input features are preprocessed by convolution operations to aggregate key information and promote the interaction between different channel features. Taking the first layer of CNN-Transformer hybrid architecture as an example, the feature optimization process of the local branch and the regional branch is expressed as ,Mode :

[0034]

[0035]

[0036]

[0037] Among them: ResBlock operation. The input information for this operation. is a two-layer convolution kernel. is a nonlinear activation function, of Indicates CBAM operation, Represents the two layers of pre-processed convolutional kernels for region branching.

[0038] Furthermore, a spatial hierarchical Transformer is designed in the global branch of SSCM. The spatial hierarchical Transformer first uses a Transformer based on small window AWSA to model the global features of the image, then cascades a Transformer using large window AWSA, and promotes information interaction between different windows through shifting technology, thereby obtaining global features different from those of a single-layer Transformer. The global branch feature optimization process The specific operation can be defined as:

[0039]

[0040]

[0041]

[0042]

[0043] Among them: , represents the Transformer operation based on small window AWSA, , Represents the Transformer operation based on large window AWSA, Respectively represent the output information of the two corresponding Transformer operations, They represent AWSA operations with small windows and large windows respectively. It is the input information of the spatial level Transformer, and in the first layer of the CNN-Transformer hybrid architecture, it is the input information of the SSCM , is a standard multi-layer perceptron. It is the LayerNorm normalization operation;

[0044] After obtaining the output of the global branch, the output information of the other two branches is fused by element-by-element addition, and then input into the standard MLP for processing, thereby obtaining the optimized output features of the single-layer CNN-Transformer hybrid architecture. Taking the first layer of hybrid architecture in the iteration stage as an example, the feature flow process can be described as follows:

[0045]

[0046] in: It is an element-by-element addition operation.

[0047] Furthermore, a post-processing module PPM is provided. The post-processing module PPM cascades a standard ResBlock block and two layers of convolution to spatially transform, filter and fuse the feature information, thereby ensuring that the reconstructed image is as close as possible to the original image in terms of structure and details. Specifically, it is expressed as follows:

[0048]

[0049] in: is the activation function, Represents two convolutional layers in cascade, the latter is without bias Convolution kernel.

[0050] Furthermore, to achieve end-to-end training, the following loss function is designed:

[0051]

[0052] Among them: the first term is the pixel loss, which is used to measure the difference between the original image and the reconstructed image. The second term is the sampling matrix constraint loss, which aims to regulate the correlation between rows in the matrix and ensure that there is less redundancy in the measurement values. is the identity matrix, is the number of measurements, is the regularization parameter and is set to 0.01 in the experiment.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) Iteration strategy and hybrid architecture parameter optimization: The number of iteration stages in the hybrid reconstruction subnetwork and the number of CNN-Transformer hybrid architecture layers have a significant impact on the reconstruction performance. The optimal combination of these parameters is one of the technical protection points of the present invention, ensuring that the network can achieve the best reconstruction effect in different application scenarios.

[0055] (2) Collaborative mechanism of gradient descent and feature optimization operations: In the present invention, the introduction position and collaborative mechanism of gradient descent and feature optimization operations are the key to improving the quality of image reconstruction. Reasonable design of the interaction between the two to maximize the synergistic effect is another important technical point that the present invention intends to protect.

[0056] (3) Design of the multi-scale feature enrichment module: The multi-scale feature enrichment module effectively removes the noise of the image reconstructed in the previous stage through different convolution operations, and generates image features rich in multi-scale details. The design of parameters such as the convolution kernel size and the number of output channels of the module branch and the integration method of the overall network architecture have an important impact on the quality of the reconstructed image, and is therefore one of the technical points to be protected by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is the anchor point self-attention structure of the present invention;

[0058] Figure 2 It is the overall network structure of the present invention;

[0059] Figure 3 is the multi-scale feature enrichment module MFEM of the present invention;

[0060] Figure 4 It is the secondary spatial information capture module SSCM of the present invention;

[0061] Figure 5 It is the spatial hierarchical Transformer structure of the present invention;

[0062] Figure 6 It is a comparison of visual effects of reconstructing images at low sampling rates using different algorithms of the present invention;

[0063] Figure 7 It is the reconstruction performance of the present invention with different numbers of iteration stages and layers of hybrid architecture at a sampling rate of 0.10;

[0064] Figure 8 It is the change curve of Loss with training epoch under different sampling rates of the present invention;

[0065] Fig. 9 It is the hierarchical feature map of SSCM in different iteration stages at a sampling rate of 0.25 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0067] The present invention provides a compressed sensing iterative reconstruction method for joint image-level features. In the prior art Transformer, the self-attention mechanism has good scalability for modeling long-distance image dependencies. By comparing the similarity of image tokens, the feature vector of each pixel contains the feature information of its context pixel. This operation converts the token into a query matrix in the spatial domain through linear mapping. , key matrix Sum Matrix , and calculate the weighted sum of all tokens using the following formula:

[0068]

[0069] in: ; is the input token of self-attention, is the transformation matrix; is the number of tokens, which is also the size of the feature map ; is the dimension of each token. In addition, represents the attention map, i.e. .

[0070] The time complexity and space complexity of self-attention are , , grows quadratically with the increase of image resolution, which can easily become the computational bottleneck of Transformer. To solve this problem, Anchored self-attention (ASA) is proposed. Figure 1 As shown in Figure 1, ASA reduces the dimension of the input token through average pooling and linear mapping to generate a low-dimensional anchor matrix. . This matrix aggregates the feature map information and reduces the spatiotemporal complexity of self-attention by acting as an intermediate link in calculating similarity. The specific mathematical expression of the ASA operation is:

[0071]

[0072]

[0073]

[0074]

[0075] in: Indicates the step length is The average pooling operation, Represent the attention mapping between anchor-value pairs and query-anchor pairs respectively. By introducing the anchor matrix, the time complexity of self-attention is reduced to , the space complexity is reduced to .

[0076] In the ICS field, the use of prior information such as Gaussian denoising and non-local similarity can significantly improve the quality of image reconstruction. Therefore, the present invention introduces prior information of the image in the spatial dimension and constructs the overall structure of CHFNet as follows: Figure 2 As shown in Figure 1, it mainly consists of three parts: sampling subnetwork, initial reconstruction subnetwork and hybrid reconstruction subnetwork. The sampling subnetwork reconstructs the original image through a learnable sampling matrix. Sampling block by block to get the measured value The initial reconstruction subnetwork performs a linear transformation on the measured value to obtain the initial restored image . Inspired by the ISTA algorithm, the present invention designs an iterative strategy in the hybrid reconstruction subnetwork, which gradually improves the image reconstruction quality by alternating between gradient descent operations and feature optimization operations. In each iterative stage, a multi-scale feature enrichment module (Multi-scale Feature Enrichment Module, MFEM) is first constructed to denoise the reconstructed image of the previous stage at different scales and enhance it into a multi-channel feature map. Subsequently, the gradient descent operation is used to update the fine-grained information of the feature map. Secondly, in order to enhance the network's perception of high-resolution images and reduce computational complexity, a secondary spatial information capture module (Secondary Spatial-information Capture Module, SSCM) is constructed to model image-level features through a lightweight CNN-Transformer hybrid architecture and deeply optimize it. Finally, the feature information optimized by SSCM is nonlinearly enhanced through the post-processing module (Post-processing Module, PPM), and the reconstructed image of this stage is obtained. After After iterations, the final reconstructed image can be obtained. .

[0077] Sampling subnetwork

[0078] In traditional ICS sampling, the original image First divided into Non-overlapping image patches of size , and vectorize them into , followed by a fixed sampling matrix Linear sampling is performed on each image block vector to obtain the measurement value of the entire image. hour, Similar to CSNet, this paper uses a learnable sampling matrix to classify images. Perform linear sampling. Specifically, Recombination The size is , the step length is The convolution kernel without bias , and optimize the matrix parameters through training data so that the measured values ​​retain more non-redundant information. Therefore, the sampling subnetwork can be expressed as:

[0079]

[0080] in: Represents a sampling operation, represents the convolution operation, represents the compressed sensing measurement value.

[0081] Initial reconstruction subnetwork

[0082] The present invention uses the transposition of the sampling matrix To complete the linear transformation from the measured value to the initial restored image. Similar to the sampling subnetwork, Reorganized into indivual Convolution kernel size , and then the initial restored image is obtained through the "PixelShuffle" operation This process is expressed as:

[0083]

[0084] in: Represents an initial refactoring operation.

[0085] Hybrid Reconstruction Sub-Network

[0086] In the gradient descent operation, reconstruction algorithms such as TransCS update the reconstructed image of the previous stage, but this method limits the perception ability of the network. Therefore, the present invention uses an adaptive step size to perform a gradient descent operation on the feature map of the reconstructed image of the previous stage, so that the feature optimization operation can process image information with higher granularity. Specifically, the present invention denoises the reconstructed image of the previous stage through MFEM and obtains a multi-channel feature map, and then uses a gradient descent operation to update the fine-grained information of the feature map. Taking the iteration stage as an example, the information flow process can be expressed as:

[0087]

[0088]

[0089] Among them: of Indicates The reconstructed image obtained in the iterative stage, represents a multi-scale feature enrichment operation, Represents a multi-channel feature map; represents the gradient descent operation, The adaptive step size of the gradient descent operation is implemented by two convolutional layers.

[0090] After completing the gradient descent operation, the network obtains a feature map containing high-grained information To further improve the image reconstruction quality, The information is input to the cascaded SSCM and PPM for feature optimization. Specifically, SSCM uses a lightweight CNN-Transformer hybrid architecture as the backbone network. By modeling and optimizing the hierarchical features of the image in the spatial dimension, the network can express more complete feature information, thereby enhancing the network's perception of high-resolution images. Subsequently, the output information of SSCM is sent to PPM for screening and fusion, and the reconstructed image of this stage is obtained while retaining the key information required for reconstruction. The mathematical expression of the feature optimization operation is as follows: ,Mode As shown:

[0091]

[0092]

[0093] in: Respectively represent the secondary spatial information capture operation and post-processing operation, Indicates the output information of SSCM. Represents the reconstructed image at this stage.

[0094] Multi-scale feature enrichment module

[0095] Convolutional neural networks can learn prior information such as potential noise distribution and perceptual quality from training data, but fixed-size convolution kernels can usually only extract information at a specific scale. In order to enable the gradient descent operation to update multi-channel feature maps with high signal-to-noise ratio, the present invention proposes a multi-scale feature enrichment module. MFEM can integrate prior information of different scales, effectively remove the noise of the reconstructed image in the previous stage, and generate image features rich in multi-scale details. Figure 3 As shown in the figure, MFEM uses a double-layer convolution structure in each scale branch. The first layer outputs 2C-dimensional features, and the second layer outputs C-dimensional features. The convolution kernel sizes are , , and After obtaining the output features of each scale branch, they are fused through the "Concat" operation and the convolution operation, which not only obtains a high signal-to-noise ratio multi-channel feature map but also simplifies the computational complexity of subsequent operations. The information flow process of MFEM can be expressed as:

[0096]

[0097] in: represents a multi-scale convolution operation, Represents the convolution kernel in the feature fusion stage.

[0098] Secondary space information capture module

[0099] Images contain rich local, regional and global features in the spatial dimension. These hierarchical features can effectively enhance the network's perception of high-resolution images. In view of this, the present invention proposes a secondary spatial information capture module based on a lightweight CNN-Transformer hybrid architecture. Figure 4 As shown, SSCM updates the features with a gradient descent operation As input, then through The layer-mixed architecture models image-level features and performs deep optimization. Convolution operations and large skip connections are used to fuse the features before and after optimization, so that the reconstruction network can more completely represent the image content.

[0100] The CNN-Transformer hybrid architecture designed in this paper aims to model and optimize image-level features. First, for local features that span dozens of pixels and mainly use edges as information, most networks usually use convolution kernels with small receptive fields (e.g., ) for modeling. In order to enhance the feature extraction capability, the local branch cascades two standard residual convolution blocks (ResBlock) and introduces skip connections to improve the model performance. Secondly, for the area range involving hundreds of pixels, a convolution kernel with a large receptive field can be used for modeling, but the feature expression capability of this design is not ideal and results in a large number of parameters. To solve this problem, the present invention introduces the reference The lightweight hybrid attention block CBAM combines the advantages of channel attention and spatial attention, and can adaptively select important objects and regions for learning. In order to further enhance the feature perception ability of the CBAM block to important regions, before executing the CBAM block, the input features are preprocessed by convolution operation to aggregate key information and promote the interaction between different channel features. Taking the first layer of CNN-Transformer hybrid architecture as an example, the feature optimization process of the local branch and the regional branch is expressed as ,Mode :

[0101]

[0102]

[0103]

[0104] Among them: ResBlock operation. The input information for this operation. is a two-layer convolution kernel. is a nonlinear activation function. of Indicates CBAM operation, Represents the two layers of pre-processed convolutional kernels for region branching.

[0105] In order to capture the global features of the image, it is necessary to design a network structure that can model long-distance dependencies. CNN introduces powerful inductive biases for modeling image relationships, including inherent properties such as local correlation and weight sharing, which enables it to adaptively learn scalable feature representations from images, but these characteristics also limit its ability to handle long-distance dependencies in images. The introduction of Transformer has brought new breakthroughs in image processing. It models the long-distance dependencies of images through self-attention, thereby more effectively capturing the global features of the image. However, with the increase of image resolution, the spatiotemporal complexity of self-attention grows quadratically, which greatly limits the application of Transformer in processing high-resolution images. Therefore, the present invention proposes an anchored-window self-attention (AWSA) that divides the image into AWSA treats each image window as a token, making its number from Reduce to , further reducing the computational complexity of self-attention feature modeling.

[0106] The AWSA-based Transformer designed by the present invention can effectively capture the global features of the image, but like conventional Transformers, they still face the challenge of monotony in the expression of image information during the modeling process, which limits the further improvement of the learning ability of the architecture. Therefore, the present invention designs a spatial hierarchical Transformer in the global branch to capture multivariate nonlinear global features through a cross-window strategy. Figure 5 As shown in the figure, the architecture first uses a Transformer based on small window AWSA to model the global features of the image, then cascades a Transformer using large window AWSA, and promotes information interaction between different windows through shifting technology, thereby obtaining global features different from those of a single-layer Transformer. It is worth noting that compared with conventional Transformers, its spatiotemporal complexity still increases linearly with the increase of image resolution. Therefore, the global branch feature optimization process The specific operation can be defined as:

[0107]

[0108]

[0109]

[0110]

[0111] Among them: , represents the Transformer operation based on small window AWSA, , Represents the Transformer operation based on large window AWSA, Respectively represent the output information of the two corresponding Transformer operations, They represent AWSA operations with small windows and large windows respectively. It is the input information of the spatial level Transformer, and in the first layer of the CNN-Transformer hybrid architecture, it is the input information of the SSCM , is a standard multi-layer perceptron. It is the LayerNorm normalization operation.

[0112] After obtaining the output of the global branch, the output information of the other two branches is fused by element-by-element addition, and then input into the standard MLP for processing, thereby obtaining the optimized output features of the single-layer CNN-Transformer hybrid architecture. Taking the first layer of hybrid architecture in the iteration stage as an example, the feature flow process can be described as follows:

[0113]

[0114] in: It is an element-by-element addition operation.

[0115] Post-processing module

[0116] Feature information deeply optimized by SSCM It still has multi-channel characteristics. In order to retain the key information required for reconstruction and obtain the reconstructed image at this stage , the present invention designs a post-processing module. PPM cascades a standard ResBlock block and two layers of convolution to transform, filter and fuse the feature information in space, so as to ensure that the reconstructed image is as close to the original image as possible in terms of structure and details. Specifically expressed as:

[0117]

[0118] in: is the activation function, Represents two convolutional layers in cascade, the latter is without bias Convolution kernel.

[0119] Loss Function

[0120] The algorithm proposed in this invention is based on the original image As input, the measurement value is first generated by the sampling subnetwork , and then the final reconstructed image is obtained through the reconstruction sub-network To achieve end-to-end training, the following loss function is designed:

[0121]

[0122] Among them: The first term is the pixel loss, which is used to measure the difference between the original image and the reconstructed image. The second term is the sampling matrix constraint loss, which aims to regulate the correlation between rows in the matrix and ensure that there is less redundancy in the measurement values. is the identity matrix, is the number of measurements, is the regularization parameter and is set to 0.01 in the experiment.

[0123] Comparison of reconstruction performance with existing algorithms

[0124] This paper selects six cutting-edge ICS reconstruction algorithms as comparison algorithms, including CSNet+, AMPNet, ISTANet++, OPINENet+, TransCS and CSformer. Among them, CSNet+ is a deep feedforward algorithm, while AMPNet, ISTANet++ and OPINENet+ are deep unfolding algorithms, and TransCS and CSformer are ICS reconstruction algorithms using Transformer. In order to more comprehensively evaluate the performance of the algorithm, the Set11, BSD68 and Urban100 data sets are tested at five sampling rates of 0.01, 0.04, 0.10, 0.25 and 0.50. The best reconstruction performance is marked in bold, and the suboptimal is marked with an underline.

[0125] Table 1 compares the image reconstruction performance of different algorithms on the Set11 dataset. When the sampling rate is 0.10, AMPNet simulates the denoising process of the iterative algorithm, and the image reconstruction performance is comparable to that of the reconstruction algorithm using Transformer. However, as the sampling rate decreases, the algorithm using Transformer shows image reconstruction performance far exceeding that of the CNN algorithm due to its excellent global feature modeling ability. At a sampling rate of 0.01, CSformer improves 1.38 dB and 0.0494 in PSNR and SSIM compared with AMPNet, respectively. The algorithm proposed in the present invention efficiently integrates the advantages of CNN and Transformer, models and optimizes image-level features, and makes the reconstruction performance of CHFNet at low sampling rates better than the comparison algorithm. Among them, at sampling rates of 0.04 and 0.10, PSNR / SSIM are improved by 0.12 dB / 0.0063 and 0.29dB / 0.0045 respectively compared with the suboptimal algorithm.

[0126] surface PSNR(dB) / SSIM comparison of reconstructed images using different sampling rates on the Set11 dataset

[0127]

[0128] In order to further verify the generalization performance of the proposed algorithm, the present invention is compared with other algorithms on BSD68 and Urban100. As shown in Table 2, the average PSNR and SSIM of the reconstructed images of the proposed algorithm are better than those of other algorithms, and are improved by 0.11 dB / 0.0085 and 0.63 dB / 0.0076 respectively compared with CSformer. At a low sampling rate of 0.10, the proposed algorithm improves the PSNR value by 0.50 dB on the BSD68 dataset compared with AMPNet, the best performing CNN algorithm; on the Urban100 dataset, compared with TransCS, the best performing algorithm among other algorithms, the PSNR and SSIM of the reconstructed image are improved by 0.83 dB and 0.0155 respectively. The Urban100 dataset contains high-resolution complex urban images, which makes it difficult for the CNN-based ICS reconstruction algorithm to achieve high-quality image reconstruction; however, the regular textures in these images provide more modeling opportunities for the Transformer that uses self-attention for similarity comparison, thereby achieving a more significant improvement in reconstruction performance. At a sampling rate of 0.01, CHFNet improves PSNR / SSIM by 0.88 dB / 0.0480 and 1.18 dB / 0.0874 compared to OPINENet+ and TransCS, respectively.

[0129] surface PSNR(dB) / SSIM comparison of reconstructed images by different algorithms at different sampling rates on different data sets

[0130]

[0131] Comparison of computational complexity with existing algorithms

[0132] In order to evaluate the computational complexity of the ICS reconstruction algorithm using Transformer, the present invention compares the network parameter amount and average reconstruction time of TransCS, CSformer and the algorithm of the present invention at sampling rates of 0.10 and 0.50, and all data are tested in the same environment. As shown in Table 3, CSformer uses a large number of feature channels in the reconstruction network, resulting in an exponential increase in its network parameter amount and average reconstruction time compared to TransCS. In contrast, the algorithm proposed in the present invention greatly reduces the network parameter amount by optimizing the spatiotemporal complexity of self-attention in Transformer and the number of feature channels of the hybrid reconstruction subnetwork. At a sampling rate of 0.10, the parameter amount of CHFNet is only 9.15% of that of CSformer, and the average reconstruction time on the three data sets is reduced by 2.7447s, 3.5510s and 4.7750s respectively.

[0133] surface Comparison of the complexity of different algorithms

[0134]

[0135] The present invention conducts an ablation study on different strategies for each iteration stage at a sampling rate of 0.50. As shown in Table 4, SSCM provides more performance improvement for the proposed algorithm than MFEM. On the three test sets, SSCM improves the PSNR of the reconstructed image by 0.82 dB, 1.37 dB and 0.88 dB, respectively, while the PSNR improvement of MFEM is 0.28 dB, 0.27 dB and 0.17 dB, respectively. In SSCM, the global branch shows superior performance in high-resolution image reconstruction due to its powerful ability to capture long-distance dependencies. Compared with the regional branch, it brings 0.25 dB and 0.15 dB PSNR improvement to BSD68 and Urban100, respectively. In summary, the various strategies proposed in the present invention have brought certain performance improvements to ICS reconstruction.

[0136] surface Ablation experiments of different strategies at 0.50 sampling rate

[0137]

[0138] As the sampling rate decreases, the ICS reconstruction algorithm faces greater challenges in capturing image information, which will increase the difficulty of image reconstruction. Figure 6 The visual effects of reconstructing images at low sampling rates by different algorithms are shown, with the sampling rates from top to bottom being 0.01, 0.10, and 0.25, respectively. It is obvious that at the extremely low sampling rate of 0.01, obvious block artifacts appear in the reconstructed images of CSNet+ and TransCS; while CSformer and the algorithm proposed in the present invention capture the long-distance dependencies of the image more effectively, and the reconstructed image quality is also fuller. In the Barbara image at a sampling rate of 0.10, TransCS and CSformer fail to accurately restore the texture of the scarf, and the detail information is seriously lost; while CHFNet successfully reconstructs an image containing rich information by deeply modeling the local and regional features of the image. In addition, in the reconstruction results at higher sampling rates, although the images reconstructed by TransCS and CSformer are better than the CNN algorithm in terms of overall effect, they are not ideal in terms of similarity details; while the algorithm of the present invention restores the similarity structure of the image in a highly fine-grained form, so that the reconstructed image has less noise and clearer texture.

[0139] Since each iteration stage of the hybrid reconstruction subnetwork corresponds to one iteration of the traditional algorithm, and the number of layers of the CNN-Transformer hybrid architecture of the SSCM in each stage is also variable, it is expected that and hybrid architecture layers As the image size increases, the image reconstruction performance will also improve. Figure 7 The PSNR of image reconstruction on the Set11 dataset of CHFNet trained with different numbers of iteration stages and hybrid architecture layers at a sampling rate of 0.10 is given. When the number of iteration stages is the same, the PSNR of image reconstruction increases by about 0.15dB for each additional layer of hybrid architecture; when the number of hybrid architecture layers remains unchanged, the reconstruction performance of the network also improves with the increase in the number of iteration stages, which indicates that more iteration stages and hybrid architecture layers can bring higher reconstruction performance. However, it should be noted that more iteration stages and hybrid architecture layers will also increase the spatiotemporal complexity of the network model. Considering the performance and complexity of the model comprehensively, the selection of , as the default parameters for the experiment.

[0140] At low sampling rates, the measurement information captured by network sampling is significantly reduced, resulting in greater challenges for the ICS reconstruction task when restoring the original image using limited measurement values, which in turn has an adverse effect on the convergence of the algorithm. Specifically, during the training process of the network, the convergence speed of the Loss curve will significantly slow down as the sampling rate decreases, and the convergence value is more difficult to reach the convergence level at a high sampling rate. The algorithm proposed in this invention belongs to the category of deep learning, so this invention is Figure 8 The variation curve of the loss of CHFNet in the training epoch at different sampling rates is given, where each epoch is iterated 2800 times during training. The experimental results show that, thanks to the better initial learning rate parameters, the algorithm of the present invention shows a faster loss decrease rate in the early stage of training. With the increase of the number of training epochs, the loss curve gradually stabilizes and shows good convergence. It is worth noting that the loss curve only oscillates slightly during the 3rd to 25th epochs at a sampling rate of 0.50, but the loss value always remains below 2.50. In addition, a significant phenomenon is that whenever the sampling rate is halved, the loss convergence value is about twice that of the previous sampling rate, which further verifies the influence of the sampling rate on the reconstruction task and the convergence of the algorithm.

[0141] In order to further explore the differences in feature modeling between local, regional and global branches in SSCM, the present invention visualizes the hierarchical features captured by the three branches of SSCM at different iteration stages at a sampling rate of 0.25. Fig. 9 As shown in the figure, the local branch has relatively weak image modeling ability in shallow iterative optimization. One possible reason for this is that the receptive field of the pure CNN architecture is small, which limits the breadth of image modeling. The regional branch efficiently extracts the key features of the image by using the CBAM block, thereby capturing a wider range of pixel relationships, especially the convex edge parts of the image. In contrast, the global branch benefits from the Transformer's ability to model long-distance dependencies, making its modeling range in shallow iterative optimization significantly better than the previous two branches. In addition, through the similarity comparison of self-attention, the global branch enables the network to learn richer texture details, allowing it to show higher fine-grained gains in deep iterative optimization.

[0142] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0143] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. The compressed sensing iterative reconstruction method combined with image-level features introduces the prior information of the image in the spatial dimension and constructs the overall structure of CHFNet, which is characterized by: The overall structure of the constructed CHFNet consists of three parts: sampling subnetwork, initial reconstruction subnetwork and hybrid reconstruction subnetwork: The sampling subnetwork samples the original image through a learnable sampling matrix Sampling block by block to get the measured value , as follows: The image is sampled using a learnable sampling matrix Perform linear sampling and transform the sampling matrix Recombination The size is , the step length is The convolution kernel without bias , and optimize the matrix parameters through training data so that the measured values ​​retain more non-redundant information; therefore, the sampling subnetwork can be expressed as: in: Represents a sampling operation, represents the convolution operation, represents the compressed sensing measurement value; The initial reconstruction subnetwork performs a linear transformation on the measured values ​​to obtain the initial restored image. , as follows: By transposing the sampling matrix To complete the linear transformation from the measured value to the initial restored image; similar to the sampling subnetwork, Reorganized into indivual Convolution kernel size , and then the initial restored image is obtained through the "PixelShuffle" operation , this process is expressed as: in: Represents the initial reconstruction operation; The hybrid reconstruction subnetwork designs an iterative strategy to gradually improve the image reconstruction quality by alternating between gradient descent operations and feature optimization operations; the specific operations of each iterative stage are as follows: Firstly, a multi-scale feature enrichment module (MFEM) is constructed to denoise the reconstructed image at different scales and enhance it into a multi-channel feature map. Then, a gradient descent operation is used to update the fine-grained information of the feature map. Taking the iteration stage as an example, the information flow process can be expressed as: Among them: of Indicates The reconstructed image obtained in the iterative stage, represents a multi-scale feature enrichment operation, Represents a multi-channel feature map; represents the gradient descent operation, Adaptive step size for gradient descent operation, implemented through two convolutional layers; It is a feature map containing high-grained information; Secondly, in order to enhance the network's perception of high-resolution images and reduce computational complexity, a secondary spatial information capture module SSCM is designed. The image-level features are modeled through a lightweight CNN-Transformer hybrid architecture and deeply optimized. Finally, the feature information optimized by the secondary spatial information capture module SSCM is nonlinearly enhanced through the post-processing module PPM. The reconstructed image of this stage is obtained while retaining the key information required for reconstruction. SSCM and PPM are called feature optimization operations, and their mathematical expressions are as follows: ,Mode As shown: in: Respectively represent the secondary spatial information capture operation and post-processing operation, Indicates the output information of SSCM. Indicates the reconstructed image at this stage; go through After iterations, the final reconstructed image can be obtained. .

2. The method for iterative reconstruction of compressed sensing based on joint image-level features according to claim 1, characterized in that: The multi-scale feature enrichment module MFEM can integrate the prior information of different scales, effectively remove the noise of the reconstructed image in the previous stage, and generate image features rich in multi-scale details. MFEM adopts a double-layer convolution structure in each scale branch. The first layer outputs 2C-dimensional features, and the second layer outputs C-dimensional features. The convolution kernel sizes are , , and After obtaining the output features of each scale branch, they are fused through the "Concat" operation and the convolution operation to obtain a high signal-to-noise ratio multi-channel feature map while simplifying the computational complexity of subsequent operations. The information flow process of MFEM can be expressed as: in: represents a multi-scale convolution operation, Represents the convolution kernel in the feature fusion stage.

3. The compressed sensing iterative reconstruction method of joint image level features according to claim 1, characterized in that: In the CNN-Transformer hybrid architecture of the SSCM module, local features are extracted by cascading two standard ResBlock blocks; and a lightweight hybrid attention block CBAM is introduced to model regional features. Before executing the CBAM block, the input features are preprocessed using convolution operations to aggregate key information and promote the interaction between features of different channels; taking the first-layer CNN-Transformer hybrid architecture as an example, the feature optimization processes of the local branch and the regional branch are expressed as ,Mode : Among them: ResBlock operation. The input information for this operation. is a two-layer convolution kernel. is a nonlinear activation function, of Indicates CBAM operation, Represents the two layers of pre-processed convolutional kernels for region branching.

4. The compressed sensing iterative reconstruction method of joint image level features according to claim 1, characterized in that: In the branch of SSCM module modeling global features, a spatial hierarchical Transformer is designed. This spatial hierarchical Transformer first uses a Transformer based on small window AWSA to model the global features of the image, then cascades a Transformer using large window AWSA, and promotes information interaction between different windows through shifting technology, thereby obtaining global features different from those of a single-layer Transformer. The global branch feature optimization process The specific operation can be defined as: Among them: , represents the Transformer operation based on small window AWSA, , Represents the Transformer operation based on large window AWSA, Respectively represent the output information of the two corresponding Transformer operations, They represent AWSA operations with small windows and large windows respectively. It is the input information of the spatial level Transformer, and in the first layer of the CNN-Transformer hybrid architecture, it is the input information of the SSCM , is a standard multi-layer perceptron. It is the LayerNorm normalization operation; After obtaining the output of the global branch, the output information of the other two branches is fused by element-by-element addition, and then input into the standard MLP for processing, thereby obtaining the optimized output features of the single-layer CNN-Transformer hybrid architecture. Taking the first layer of hybrid architecture in the iteration stage as an example, the feature flow process can be described as follows: in: It is an element-by-element addition operation.

5. The compressed sensing iterative reconstruction method of joint image level features according to claim 1, characterized in that: There is also a post-processing module PPM, which cascades a standard ResBlock block and two layers of convolution to spatially transform, filter and fuse the feature information, so as to ensure that the reconstructed image is as close to the original image as possible in terms of structure and details. It is specifically expressed as follows: in: is the activation function, Represents two convolutional layers in cascade, the latter is without bias Convolution kernel.

6. The compressed sensing iterative reconstruction method of joint image level features according to claim 1, characterized in that: To achieve end-to-end training, the following loss function is designed: Among them: the first term is the pixel loss, which is used to measure the difference between the original image and the reconstructed image. The second term is the sampling matrix constraint loss, which aims to regulate the correlation between rows in the matrix and ensure that there is less redundancy in the measurement values. is the identity matrix, is the number of measurements, is the regularization parameter and is set to 0.01 in the experiment.

Images