Adaptive image super-resolution reconstruction method based on compressed sensing

Abstract

The present invention relates to the field of image processing technology, and in particular to an adaptive image super-resolution reconstruction method based on compressed sensing. The present invention designs a trainable compressed sensing sampling module for performing sparse observation compression on an input image. The module supports end-to-end joint training and has data-driven optimization capabilities. Subsequently, the content structure in the compressed image is modeled through an image prior analysis module, and the image complexity index is calculated as a reference for subsequent path selection. The present invention combines compressed sensing theory with an image structure complexity perception mechanism, and realizes a more efficient and quality-controlled super-resolution image reconstruction process through an adaptive path selection strategy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an adaptive image super-resolution reconstruction method based on compressed sensing driving. Background Art

[0002] Image super-resolution (SR) aims to reconstruct high-resolution images from low-resolution ones and is widely used in remote sensing, medical imaging, video surveillance, and other scenarios. Deep learning-based methods have made significant progress in recent years. However, most methods suffer from high computational complexity and low inference efficiency, making them difficult to deploy in real-world applications.

[0003] In the development of image super-resolution reconstruction, compressed sensing theory can achieve efficient representation and reconstruction of images through sparsity, providing new possibilities for input preprocessing and representation enhancement. For example, the Chinese patent publication number is CN112330578A, the publication date is February 5, 2021, and the patent name is "An image reconstruction method and system combining compressed sensing and cleaning algorithm". It discloses that according to the compressed sensing principle, a suitable sparse domain is found for the sparse brightness temperature distribution image and an objective function is established, and a compressed sensing reconstructed image is obtained through an iterative algorithm; the compressed sensing reconstructed image is Fourier transformed, and the frequency domain visibility function value of the sampling baseline position is obtained; the original image sampling value is subtracted from the frequency domain value and an inverse Fourier transform is performed to obtain the residual inversion image; the residual inversion image is cleansed by a cleaning method to obtain a residual reconstructed image; the compressed sensing reconstructed image is added to the residual reconstructed image to obtain the final reconstructed image. The Chinese patent publication number is CN115841420A, the publication date is March 24, 2023, and the invention patent application name is "A method for super-resolution reconstruction of polarization images based on deep learning". The disclosed method includes collecting high-resolution polarization images based on a polarization camera, and preprocessing to obtain low-resolution polarization images to produce a data set; fusing feature information through a shallow convolutional neural network; densely fusing features through a residual dense module; using a scaling module to amplify the number of image feature channels to a multiple of the corresponding amplification factor; obtaining a super-resolution polarization image through an output module; and optimizing the neural network using polarization perception and pixel mixing loss function.

[0004] However, neither of the two aforementioned invention patent applications considers that the varying sensitivity of different image regions to resolution can affect the image super-resolution effect. Scheduling the reconstruction path based on the local complexity of the image, and using different depth models to process different regions, would help achieve a dynamic balance between reconstruction quality and efficiency. Currently, there is a lack of super-resolution algorithms that combine compressed sensing modeling with content-aware path scheduling, particularly those that are lightweight, high-quality, and adaptable, all-software solutions. Summary of the Invention

[0005] In view of this, the present invention aims to provide an adaptive image super-resolution reconstruction method based on compressed sensing. By combining the efficient data representation capability of compressed sensing with an adaptive path scheduling mechanism, this method can dynamically optimize computing resource allocation while ensuring high-precision reconstruction, significantly improving algorithm efficiency and adaptability.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0007] A method for adaptive image super-resolution reconstruction based on compressed sensing, comprising:

[0008] S1: Use the compressed sensing matrix to sample the input low-resolution image to obtain the corresponding observation vector; perform sparse representation on the low-resolution image to obtain the corresponding sparse coefficients;

[0009] S2: restoring the sparse coefficients obtained in step S1 based on the observation vector and the compressed sensing matrix obtained in step S1; obtaining an optimized reconstructed image based on the restored sparse coefficients and the observation vector;

[0010] S3: Use the Transformer-based path scheduler to perform image complexity judgment on the optimized reconstructed image obtained in step S2 to obtain the path probability, and input the optimized reconstructed image into the fast path and the fine path for processing to obtain the fast reconstructed image and the fine reconstructed image respectively;

[0011] S4: Based on the path probability obtained in step S3, the fast reconstructed image and the fine reconstructed image obtained in step S3 are fused to obtain a corresponding high-resolution image.

[0012] Furthermore, in step S1, the process of sampling the input low-resolution image using the compressed sensing matrix to obtain the corresponding observation vector includes:

[0013] The low-resolution image is segmented into several image blocks, and the observation vector is obtained by the following formula:

[0014] ;

[0015] in, image patches representing low-resolution images, represents the observation vector, represents the compressed sensing matrix;

[0016] The compressed sensing matrix is ​​trained using the following formula:

[0017] ;

[0018] in, represents the loss function for training the compressed sensing matrix, represents a low-resolution image, represents the initial reconstructed image obtained by reverse sparse projection of the observation vector, E represents the unit matrix, represents the loss adjustment parameter, Indicates L2 norm calculation, represents the Frobenius norm.

[0019] Furthermore, step S2 includes:

[0020] S21: Using the sparse representation dictionary, the sparse coefficients are restored using the following formula:

[0021] ;

[0022] in, represents the sparse coefficient, represents the recovery weight parameter, represents a sparse representation dictionary;

[0023] S22: processing the preliminary reconstructed image into blocks to obtain preliminary reconstructed image blocks;

[0024] S23: extracting a context information feature vector from each preliminary reconstructed image block obtained in step S22 through a conditional feature extraction network;

[0025] S24: Based on the context information feature vector obtained in step S23, a multi-layer perceptron is used to dynamically generate an adaptive sparse dictionary, and the recovered sparse coefficients are optimized using the following formula:

[0026] ;

[0027] in, represents the optimized sparse coefficient corresponding to the i-th preliminary reconstructed image block, represents the i-th preliminary reconstructed image block, Represents the context information feature vector corresponding to the i-th preliminary reconstructed image block, represents the adapted sparse dictionary of the i-th context information feature vector, represents the sparse regularization coefficient;

[0028] S25: Reconstruct the image block using the optimized sparse coefficients and the adaptive sparse dictionary obtained in step S24 using the following formula:

[0029] ;

[0030] in, represents the i-th image block after reconstruction;

[0031] S26: All the reconstructed blocks obtained in step S25 are fused using a sliding window stitching method to construct a full-image reconstructed image;

[0032] S27: Construct a shallow convolutional neural network to optimize the full-image reconstructed image to obtain an optimized reconstructed image; the shallow convolutional neural network includes three cascaded convolutional layers, wherein ReLU activation operation is performed on the features output by the first two convolutional layers; the output of the last convolutional layer is subtracted from the corresponding elements of the full-image reconstructed image to obtain the optimized reconstructed image.

[0033] Furthermore, the conditional feature extraction network in step S23 includes multiple convolutional layers, global average pooling layers and fully connected layers; wherein, after the input preliminary reconstructed image block passes through multiple convolutional layers for feature extraction in sequence, the global average pooling layer spatially compresses the extracted features; the output features of the global average pooling layer enter the fully connected layer to obtain the corresponding context information feature vector.

[0034] Furthermore, in the path scheduler in step S3:

[0035] The input optimized reconstructed image is converted into a feature sequence using a block embedding layer;

[0036] Perform continuous multi-layer Transformer encoding on the feature sequence; in each Transformer encoding, perform a multi-head self-attention operation on the input feature sequence, and perform a window attention operation on the output features of the multi-head self-attention operation to obtain the attention feature;

[0037] The attention feature is added to the corresponding element of the feature sequence and then layer normalized. The layer normalized feature is then input into the feedforward network for mapping to obtain the mapping feature.

[0038] The mapping features are pooled, and the pooled features are activated by MLP and Sigmoid to obtain the path probability.

[0039] Furthermore, in the fast path, the ordinary convolution in MobileNetV2 is replaced with depth-wise separable convolution, and a local residual enhancement block is inserted after each layer of Stage3 to Stage5 in the bottleneck module of MobileNetV2;

[0040] In the local residual enhancement block, the input features are sequentially subjected to multiple depth-wise separable convolution layers, and a ReLU activation operation is performed after each convolution layer; the output features of the last depth-wise convolution are added to the corresponding elements of the input features;

[0041] Sub-pixel convolution is performed on the output features of the improved MobileNetV2 to obtain a fast reconstructed image.

[0042] Furthermore, in the refined path, the input features are initially convolved, and then the convolved features are continuously input into the cascaded multi-layer Swin Transformer; the processed features are subjected to channel attention operation, and then the features after the channel attention operation are subjected to sub-pixel convolution to obtain a refined reconstructed image.

[0043] Furthermore, in step S4, the fast reconstructed image and the fine reconstructed image are spliced ​​together, and the spliced ​​features are sequentially subjected to convolution at two different scales and then sigmoid activation operations to obtain fusion coefficients.

[0044] According to the fusion coefficient and path probability, the fusion weight is obtained by the following formula:

[0045] ;

[0046] in, represents the fusion weight, represents the fusion adjustment factor, represents the fusion coefficient, represents the path probability;

[0047] Based on the fusion weights, the corresponding elements of the fast reconstructed image and the fine reconstructed image are added together to obtain a high-resolution image using the following formula:

[0048] ;

[0049] in, represents a high-resolution image, represents the finely reconstructed image, Indicates fast image reconstruction.

[0050] Furthermore, the composite loss function of training steps S2 to S4 is:

[0051] ;

[0052] in, represents the composite loss function, represents the MSE loss, represents the perceived loss, represents the path selection loss, 、 and represents the compound loss weight;

[0053] The path selection loss is:

[0054]

[0055] Where B represents the total number of optimized reconstructed images, represents the path probability corresponding to the b-th optimized reconstructed image, represents the true path label corresponding to the b-th optimized reconstructed image, ,when When , it means the real path label is a refined path. When , it indicates that the real path label is the fast path.

[0056] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0057] (1) In the adaptive image super-resolution reconstruction method based on compressed sensing, the present invention first adopts an optimizable compressed sensing matrix to efficiently sample the input image, reducing redundant information while retaining key structural features, thereby improving the accuracy of subsequent reconstruction; the present invention provides a sparse representation and reconstruction mechanism, which replaces the traditional fixed dictionary sparse reconstruction method and adopts a learnable dictionary model based on image context adaptive adjustment, which has stronger image structure recovery capabilities and is particularly suitable for complex image scenes in compressed sensing recovery; the fast path or the fine path is dynamically selected based on the local complexity of the image to achieve a dynamic balance between computational efficiency and reconstruction quality; the present invention also provides an adaptive path scheduling mechanism, that is, the participation of the fast path and the fine path in feature fusion is dynamically adjusted based on the local complexity of the image to achieve a dynamic balance between computational efficiency and reconstruction quality, thereby improving the final reconstruction quality;

[0058] (2) In the adaptive image super-resolution reconstruction method based on compressed sensing, the present invention uses sub-pixel convolution to perform upsampling in the fast path and the fine path to avoid the checkerboard effect and improve the final reconstruction quality. In addition, the present invention combines MSE loss, perception loss and path selection loss to perform multi-objective optimization, ensuring reconstruction accuracy and perception quality while improving the accuracy of path scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 A flowchart of the adaptive image super-resolution reconstruction method based on compressed sensing driving according to an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a flow chart of a method for adaptive image super-resolution reconstruction based on compressed sensing according to an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the process of steps S1 and S2 described in the embodiment of the present invention;

[0063] Figure 4 This is a flow chart of step S3 according to an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of the fast path and the fine path described in the embodiment of the present invention;

[0065] Figure 6 This is a flow chart of step S4 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0067] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0068] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second" and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0069] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0070] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0071] like Figures 1 to 6 As shown, the adaptive image super-resolution reconstruction method based on compressed sensing driving according to the embodiment of the present invention includes:

[0072] S1: Use the compressed sensing matrix to sample the input low-resolution image to obtain the corresponding observation vector; perform sparse representation on the low-resolution image to obtain the corresponding sparse coefficients.

[0073] In some embodiments, the process of sampling the input low-resolution image using a compressed sensing matrix to obtain a corresponding observation vector includes:

[0074] The low-resolution image is segmented into several image blocks, and the observation vector is obtained by the following formula:

[0075] ;

[0076] in, Represents a low-resolution image image blocks, N represents low-resolution image blocks Dimensions, represents the observation vector, In this embodiment of the present invention, the compressed sensing matrix The initial random Gaussian matrix is ​​used to map the image blocks. Perform projection to obtain the observation vector . Compressed sensing matrix You can learn it later during training.

[0077] In the embodiment of the present invention, the low-resolution image Segmented into several image blocks , and then through the compressed sensing matrix For several image blocks obtained by segmentation Perform low-dimensional sampling (i.e., use a random Gaussian matrix to project the image block) to obtain the observation vector , M represents the dimension of the observation vector. The ratio of dimension M to dimension N is set according to the compression ratio. In the embodiment of the present invention, the compression ratio is 0.25, that is, M / N=0.25.

[0078] The compressed sensing matrix is ​​trained and optimized end-to-end, with the training goal of minimizing the image reconstruction error and maintaining the stability of the matrix structure. Specifically, the compressed sensing matrix is ​​trained using the following formula:

[0079] ;

[0080] in, represents the loss function for training the compressed sensing matrix, represents the initial reconstructed image obtained by reverse sparse projection of the observation vector, E represents the unit matrix, represents the loss adjustment parameter, Represents the L2 norm calculation, which is used to measure the pixel-level reconstruction error. represents the Frobenius norm. Loss function in is a regular term used to constrain the compression matrix to be close to the orthogonal structure, and the loss adjustment parameter Used to balance reconstruction error and regularization term .

[0081] Through the above training optimization strategy, the compressed sensing matrix It can effectively preserve the structure and texture information of the image, while significantly reducing data redundancy and improving subsequent reconstruction efficiency.

[0082] S2: Restore the sparse coefficients obtained in step S1 based on the observation vector and the compressed sensing matrix obtained in step S1; obtain an optimized reconstructed image based on the restored sparse coefficients and the observation vector.

[0083] Compressed sensing theory shows that if an image is sparse in a certain transform domain, the original image can be reconstructed with a small number of observations. Therefore, in some embodiments, step S2 includes:

[0084] S21: Using the sparse representation dictionary, the sparse coefficients are restored using the following formula:

[0085] ;

[0086] in, represents the recovery weight parameter, represents a sparse representation dictionary, Represents sparse coefficients, sparse coefficients The number of non-zero elements in is much less than the total dimension (i.e., "sparseness"). In this embodiment of the present invention, the sparse representation dictionary Sparse coefficient It can be expressed by the following formula:

[0087]

[0088] in, , Indicates 0 norm calculation, that is, sparse coefficient The number of non-zero elements in the sparse coefficient vector is K, which represents the sparsity constant, i.e., the maximum number of non-zero elements allowed in the sparse coefficient vector. .

[0089] In the above formula for recovering sparse coefficients, is the reconstruction error, which is used to measure the sparse coefficient The prediction quality of L1 regularization term, used to enhance the sparsity of coefficients and restore weight parameters The optimization process in step S21 is usually solved by basis pursuit (Basis Pursuit), matching pursuit (OMP), or neural network approximation. In this embodiment of the present invention, a deep network is used to directly model this process in the next stage.

[0090] S22: Process the preliminary reconstructed image into blocks to obtain preliminary reconstructed image blocks. Perform block processing and divide it into multiple preliminary reconstructed image blocks , used for local sparse coding, i represents the number of preliminary reconstructed image blocks, where p represents the preliminary reconstructed image blocks The dimension of , that is, the total number of pixels in each preliminary reconstructed image block, is unified to p = N in the compressed sensing stage and the sparse reconstruction stage.

[0091] S23: Extracting a context information feature vector from each preliminary reconstructed image block obtained in step S22 through a conditional feature extraction network.

[0092] It can be understood that step S23 can be expressed by the following formula:

[0093] ;

[0094] in, Represents the i-th preliminary reconstructed image block The corresponding context information feature vector, f represents the conditional feature extraction network. In some embodiments, the conditional feature extraction network includes multiple convolutional layers, global average pooling layers and fully connected layers; wherein, after the input preliminary reconstructed image block passes through multiple convolutional layers for feature extraction, the global average pooling layer performs spatial compression on the extracted features; the output features of the global average pooling layer enter the fully connected layer to obtain the corresponding context information feature vector. In an embodiment of the present invention, the input preliminary reconstructed image block passes through three convolutional layers in sequence: the convolution kernel size of the first convolution layer is 3×3, the number of channels is 32, the stride is 1, and the activation function is ReLU; the convolution kernel size of the second convolution layer is 3×3, the number of channels is 64, the stride is 2, and batch normalization and ReLU activation processing are performed after convolution; the convolution kernel size of the third convolution layer is 3×3, the number of channels is 128, the stride is 2, and the activation function is ReLU. After feature extraction through three convolutional layers, the obtained features are input into the global average pooling layer to compress the spatial dimension of the features to 1×1, and then input into the fully connected layer to compress the output dimension to the set feature vector dimension. In the embodiment of the present invention, the feature vector dimension is set to 64 to obtain the corresponding context information feature vector.

[0095] S24: Based on the context information feature vector obtained in step S23, a multi-layer perceptron is used to dynamically generate an adaptive sparse dictionary. In this embodiment of the present invention, based on the context information feature vector corresponding to each preliminary reconstructed image block, , using multi-layer perceptron dynamic correspondence to generate adaptive sparse dictionary , and perform sparse coding to solve the optimal optimized sparse coefficient , where k represents the adaptive sparse dictionary The number of atoms in the adaptive sparse dictionary Specifically, the recovered sparse coefficients are optimized by the following formula:

[0096] ;

[0097] in, Represents the sparse regularization coefficient used to control the sparsity of the solution. Multilayer Perceptron Dynamic Correspondence Generates Adaptive Sparse Dictionary , ensuring that each column (atom) of the dictionary is relevant to the image context.

[0098] S25: Reconstruct the image block using the optimized sparse coefficients and the adaptive sparse dictionary obtained in step S24 using the following formula:

[0099] ;

[0100] in, represents the reconstructed i-th image block.

[0101] S26: All the reconstructed blocks obtained in step S25 are merged in a sliding window splicing manner to construct a full-image reconstructed image. Fusion is performed using sliding window stitching to construct a full-image reconstruction image. , n represents the total number of pixels of the full-image reconstructed image.

[0102] S27: Construct a shallow convolutional neural network to optimize the full-image reconstructed image to obtain an optimized reconstructed image. The shallow convolutional neural network includes three cascaded convolutional layers, and performs a ReLU activation operation on the features output by the first two convolutional layers. In this embodiment of the present invention, the output of the last convolutional layer is subtracted from the full-image reconstructed image by corresponding elements to obtain the optimized reconstructed image. In other embodiments, the output of the last convolutional layer can also be directly used as the optimized reconstructed image.

[0103] The shallow convolutional neural network provided by the embodiment of the present invention can effectively approximate the process of introducing the full-image regularization term to fine-tune and optimize the full-image reconstructed image. Specifically, the process of optimizing the full-image reconstructed image to obtain the optimized reconstructed image can be expressed as follows:

[0104] ;

[0105] in, represents the optimized reconstructed image, represents the hyperparameter that controls the strength of the regularization constraint, Represents the regularization term of low-resolution images, regularization term This method is used to measure the difference between optimized images in the perceptual space. This method approximates the above regularization process through a shallow convolutional neural network, improving image restoration accuracy and reducing edge artifacts.

[0106] S3: Use the Transformer-based path scheduler to perform image complexity judgment on the optimized reconstructed image obtained in step S2 to obtain the path probability, and input the optimized reconstructed image into the fast path and fine path for processing to obtain the fast reconstructed image and the fine reconstructed image respectively.

[0107] In the path scheduler provided in some embodiments: a block embedding layer is used to convert the input optimized reconstructed image into a feature sequence, specifically, the input image is divided into multiple image blocks, and the image blocks are mapped into fixed-length vectors (i.e., feature sequences); the feature sequence is continuously encoded with multiple layers of Transformer; in each Transformer encoding, a multi-head self-attention operation is performed on the input feature sequence, and a window attention operation is performed on the output features of the multi-head self-attention operation to obtain attention features; the attention features are added to the corresponding elements of the feature sequence and then layer normalization is performed, and the layer-normalized features are input into the feedforward network for mapping to obtain mapping features; a pooling operation is performed on the mapping features, specifically global average pooling, and the pooled features are processed by MLP (multi-layer perceptron) and Sigmoid activation operation to obtain the path probability.

[0108] In the implementation of the present invention, the block embedding layer is used to embed the input optimized reconstructed image Divide into multiple sizes The image blocks, H and W represent the optimized reconstructed images The height and width of the image are optimized. The width and height of the image are consistent with the width and height of the initial reconstruction image. Each image block is flattened and mapped into a vector of dimension d through linear transformation to form a feature sequence , for the feature sequence Perform continuous multi-layer Transformer encoding. In each Transformer encoding, the input feature sequence X is subjected to a multi-head self-attention (MHSA) operation using the following formula:

[0109] ;

[0110] in, represents a multi-head self-attention operation, Q, K, and V represent the query vector, key vector, and value vector of the feature sequence X, respectively. Represents the dimension of each self-attention, used for normalization, express Function. Perform window attention operation on the feature blocks in the output features of the multi-head self-attention operation to obtain the attention features The present invention performs window attention operation on feature blocks to reduce computational cost.

[0111] Attention Features After adding the corresponding elements of the feature sequence X, perform layer normalization, and then normalize the features after layer normalization Input the feedforward network for mapping to obtain mapping features The process of acquiring mapping features can be expressed as follows:

[0112] ;

[0113] in, represents a feedforward network, express activation function, and represents the linear mapping parameter matrix of the feedforward network, and Represents the bias term. Perform global average pooling to form an image-level feature vector, perform MLP (multi-layer perceptron) processing and Sigmoid activation on the pooled features to judge the image complexity and obtain the path probability .

[0114] The feedforward network provided by this embodiment of the present invention has a two-layer structure: the first layer is a fully connected layer, with an output channel dimension four times the input channel dimension and using ReLU activation; the second layer is a fully connected layer, with an output channel dimension one-quarter the input channel dimension. The feedforward network shares parameters at all positions and is combined with the multi-head attention layer via residual connections and layer normalization to form a standard Transformer Block.

[0115] In some embodiments, a fast path is provided such as Figure 5 As shown in (a), the ordinary convolution in MobileNetV2 is replaced with depthwise separable convolution, and a local residual enhancement block is added to improve the ability to model local structural information. Specifically, the local residual enhancement block is inserted after each layer of Stage 3 to Stage 5 in the bottleneck module of MobileNetV2.

[0116] In the fast path provided by the embodiment of the present invention, the process of depthwise separable convolution can be expressed as follows:

[0117] ;

[0118] in, represents depthwise separable convolution, Indicates that each channel of the input feature x is independently convolved by 3×3. Indicates that 1×1 channel fusion convolution is performed on the convolved features.

[0119] In the local residual enhancement block, the input features are sequentially subjected to multiple depth-wise separable convolutional layers, and a ReLU activation operation is performed after each convolution layer; the output features of the last depth-wise convolution are added to the corresponding elements of the input features; and sub-pixel convolution is performed on the output features of the improved MobileNetV2 to obtain a fast reconstructed image.

[0120] In an embodiment of the present invention, the sub-pixel convolution process can be expressed as follows:

[0121] ;

[0122] in, represents the output of sub-pixel convolution, represents sub-pixel convolution, Indicates the reorganization of features into rows and columns, H x and W x Represents the height and width of the input feature, C represents the number of channels of the input feature, represents the upsampling ratio, and in the embodiment of the present invention, r=4.

[0123] In some embodiments, a refined path such as Figure 5 As shown in (b), the input features are initially convolved, and then the convolved features are continuously input into the cascaded multi-layer Swin Transformer; the processed features are subjected to channel attention operation, and then the features after the channel attention operation (SE) are subjected to sub-pixel convolution to obtain a finely reconstructed image. The embodiment of the present invention adopts the Swin Transformer structure from the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" from the ICCV 2021 conference.

[0124] S4: Based on the path probability obtained in step S3, the fast reconstructed image and the fine reconstructed image obtained in step S3 are fused to obtain a corresponding high-resolution image.

[0125] In some embodiments, the fast reconstructed image and the fine reconstructed image are spliced ​​together, and the spliced ​​features are then subjected to two different scale convolutions in sequence and then sigmoid activation operations to obtain a fusion coefficient. In this embodiment of the present invention, the process of obtaining the fusion coefficient can be expressed as follows:

[0126] ;

[0127] in, represents the fusion coefficient, represents sigmoid activation, express activation function, Indicates a convolution operation with a convolution kernel of 3×3. Indicates a convolution operation with a convolution kernel of 1×1. represents the finely reconstructed image, represents fast reconstruction of the image, Represents a splicing operation.

[0128] According to the fusion coefficient and path probability, the fusion weight is obtained by the following formula:

[0129] ;

[0130] in, represents the fusion weight, represents the fusion adjustment factor, represents the fusion coefficient;

[0131] Based on the fusion weights, the corresponding elements of the fast reconstructed image and the fine reconstructed image are added together to obtain a high-resolution image using the following formula:

[0132] ;

[0133] in, Represents a high-resolution image.

[0134] In some embodiments, the composite loss function of training steps S2 to S4 is:

[0135] ;

[0136] in, represents the composite loss function, represents the MSE loss, represents the perceived loss, represents the path selection loss, 、 and represents the compound loss weight.

[0137] MSE loss It can be expressed by the following formula:

[0138] ;

[0139] in, represents the true high-resolution image corresponding to the low-resolution image I.

[0140] Perceptual loss It can be expressed by the following formula:

[0141] ;

[0142] in, Represents the feature tensor output by the third convolutional layer in the VGG16 network.

[0143] Path selection loss The calculation is the difference between the path probability output by the Transformer-based path scheduler in step S3 and the corresponding true path label. Specifically, the path selection loss is:

[0144]

[0145] Where B represents the total number of optimized reconstructed images, represents the path probability corresponding to the b-th optimized reconstructed image, represents the true path label corresponding to the b-th optimized reconstructed image, ,when When , it means that the true path label is the fine path, that is, the fine reconstructed image output by the fine path accounts for a larger proportion during fusion. When , it indicates that the true path label is the fast path, that is, the fast reconstructed image output by the fast path accounts for a larger proportion during fusion.

[0146] The true path label is obtained by the image entropy and average gradient of the preliminary reconstructed image. Specifically, the image entropy of the preliminary reconstructed image is obtained by the following formula:

[0147] ;

[0148] Among them, S represents the image entropy, It represents the probability of gray level l appearing in the preliminary reconstructed image, L represents the gray level range, L∈[0,255], which determines the resolution of entropy calculation.

[0149] The average gradient of the preliminary reconstructed image is obtained as follows:

[0150] ;

[0151] in, represents the average gradient, and (h,w) represents the pixel point of the image.

[0152] When the average gradient of the initial reconstructed image Less than the preset average gradient , and the image entropy S of the preliminary reconstructed image is less than the preset image entropy When and , the real path label , otherwise the real path label .

[0153] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0154] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An adaptive image super-resolution reconstruction method based on compressed sensing, characterized in that: include: S1: Use the compressed sensing matrix to sample the input low-resolution image and obtain the corresponding observation vector; Performing sparse representation on the low-resolution image to obtain corresponding sparse coefficients; S2: restoring the sparse coefficients obtained in step S1 based on the observation vector and the compressed sensing matrix obtained in step S1; obtaining an optimized reconstructed image based on the restored sparse coefficients and the observation vector; S3: Using a Transformer-based path scheduler to perform image complexity judgment on the optimized reconstructed image obtained in step S2 to obtain a path probability, and input the optimized reconstructed image into the fast path and the fine path for processing to obtain a fast reconstructed image and a fine reconstructed image, respectively; in the path scheduler in step S3: The input optimized reconstructed image is converted into a feature sequence using a block embedding layer; Perform continuous multi-layer Transformer encoding on the feature sequence; in each Transformer encoding, perform a multi-head self-attention operation on the input feature sequence, and perform a window attention operation on the output features of the multi-head self-attention operation to obtain the attention feature; Adding the attention feature to the corresponding element of the feature sequence and performing layer normalization, and then inputting the layer-normalized feature into the feedforward network for mapping to obtain a mapping feature; The mapping features are pooled, and the pooled features are activated by MLP and Sigmoid to obtain the path probability. S4: Based on the path probability obtained in step S3, the fast reconstructed image and the fine reconstructed image obtained in step S3 are fused to obtain a corresponding high-resolution image.

2. The adaptive image super-resolution reconstruction method based on compressed sensing according to claim 1, characterized in that: In step S1, the process of sampling the input low-resolution image using the compressed sensing matrix to obtain the corresponding observation vector includes: The low-resolution image is segmented into several image blocks, and the observation vector is obtained by the following formula: ； in, an image patch representing the low-resolution image, represents the observation vector, represents the compressed sensing matrix; The compressed sensing matrix is ​​trained by the following formula: ； in, represents the loss function for training the compressed sensing matrix, represents the low-resolution image, represents the preliminary reconstructed image obtained by reverse sparse projection of the observation vector, E represents the unit matrix, represents the loss adjustment parameter, Indicates L2 norm calculation, represents the Frobenius norm.

3. The adaptive image super-resolution reconstruction method based on compressed sensing according to claim 2, characterized in that: Step S2 includes: S21: Using the sparse representation dictionary, the sparse coefficients are restored using the following formula: ； in, represents the sparse coefficient, represents the recovery weight parameter, represents the sparse representation dictionary; S22: processing the preliminary reconstructed image into blocks to obtain preliminary reconstructed image blocks; S23: extracting a context information feature vector from each preliminary reconstructed image block obtained in step S22 through a conditional feature extraction network; S24: Based on the context information feature vector obtained in step S23, a multi-layer perceptron is used to dynamically generate an adaptive sparse dictionary, and the recovered sparse coefficients are optimized using the following formula: ； in, represents the optimized sparse coefficient corresponding to the i-th preliminary reconstructed image block, represents the i-th preliminary reconstructed image block, Represents the context information feature vector corresponding to the i-th preliminary reconstructed image block, represents the adapted sparse dictionary of the i-th context information feature vector, represents the sparse regularization coefficient; S25: Reconstruct the image block using the optimized sparse coefficients and the adaptive sparse dictionary obtained in step S24 using the following formula: ； in, represents the i-th image block after reconstruction; S26: All the reconstructed blocks obtained in step S25 are fused using a sliding window stitching method to construct a full-image reconstructed image; S27: Construct a shallow convolutional neural network to optimize the full-image reconstructed image to obtain an optimized reconstructed image; the shallow convolutional neural network includes three cascaded convolutional layers, wherein ReLU activation operation is performed on the features output by the first two convolutional layers; the output of the last convolutional layer is subtracted from the corresponding elements of the full-image reconstructed image to obtain the optimized reconstructed image.

4. The method for adaptive image super-resolution reconstruction based on compressed sensing according to claim 3, characterized in that: The conditional feature extraction network in step S23 includes multiple convolutional layers, global average pooling layers and fully connected layers; wherein, after the input preliminary reconstructed image block passes through multiple convolutional layers in sequence for feature extraction, the global average pooling layer spatially compresses the extracted features; the output features of the global average pooling layer enter the fully connected layer to obtain the corresponding context information feature vector.

5. The adaptive image super-resolution reconstruction method based on compressed sensing according to claim 1, characterized in that: In the fast path, the ordinary convolution in MobileNetV2 is replaced with depthwise separable convolution, and a local residual enhancement block is inserted after each layer of Stage 3 to Stage 5 in the bottleneck module of MobileNetV2; In the local residual enhancement block, the input features are sequentially subjected to multiple depthwise separable convolution layers, and a ReLU activation operation is performed after each convolution layer; the output features of the last depthwise convolution are added to the corresponding elements of the input features; Sub-pixel convolution is performed on the output features of the improved MobileNetV2 to obtain the fast reconstructed image.

6. The adaptive image super-resolution reconstruction method based on compressed sensing according to claim 1, characterized in that: In the refined path, the input features are initially convolved, and then the convolved features are continuously input into the cascaded multi-layer Swin Transformer; the processed features are subjected to channel attention operation, and then the features after the channel attention operation are subjected to sub-pixel convolution to obtain the refined reconstructed image.

7. The adaptive image super-resolution reconstruction method based on compressed sensing according to claim 1, characterized in that: In step S4, the fast reconstructed image and the fine reconstructed image are spliced ​​together, and the spliced ​​features are sequentially subjected to convolution at two different scales and then to a sigmoid activation operation to obtain a fusion coefficient; According to the fusion coefficient and the path probability, the fusion weight is obtained by the following formula: ； in, represents the fusion weight, represents the fusion adjustment factor, represents the fusion coefficient, represents the path probability; Based on the fusion weights, the high-resolution image is obtained by adding corresponding elements of the fast reconstructed image and the fine reconstructed image using the following formula: ； in, represents the high-resolution image, represents the finely reconstructed image, represents the fast reconstructed image.

8. The method for adaptive image super-resolution reconstruction based on compressed sensing according to claim 1, characterized in that: The composite loss function of training steps S2 to S4 is: ； in, represents the composite loss function, represents the MSE loss, represents the perceived loss, represents the path selection loss, 、 and represents the compound loss weight; The path selection loss is: Wherein, B represents the total number of the optimized reconstructed images, represents the path probability corresponding to the b-th optimized reconstructed image, represents the true path label corresponding to the b-th optimized reconstructed image, ,when When , it means the real path label is a refined path. When , it indicates that the real path label is the fast path.

Images