An image compression sensing reconstruction method based on a Transformer enhanced residual self-encoding network

Abstract

The application discloses an image compression sensing reconstruction method based on a residual self-encoding network enhanced by a Transform, and is characterized in that the method comprises the following steps: 1) performing initial linear reconstruction on image observation values y; 2) designing and adopting a residual self-encoding network based on a Transform for feature enhancement to perform deep reconstruction; and 3) performing network training based on a global-local joint loss function. The method can better capture local and global features, effectively enhance feature information, and accurately reconstruct an original image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image compressed sensing reconstruction method based on a Transformer enhanced residual autoencoder network. Background Art

[0002] Compressed sensing (CS) is an emerging image signal acquisition and reconstruction technology that combines sampling and compression processes to overcome the limitations of the Nyquist-Shannon sampling theorem. It can effectively recover and reconstruct image information at sampling rates far below the Nyquist rate, providing a solution for better and faster transmission of massive data.

[0003] The Transformer is an encoder-decoder architecture based on a global self-attention mechanism that can capture global interactions between contextual information and has demonstrated promising performance in several visual tasks. The Swin Transformer combines the strengths of convolutional neural networks (CNNs) and the Transformer, showing great promise in computer vision.

[0004] In recent years, thanks to the powerful feature extraction and representation capabilities of deep learning technology, existing deep learning-based methods have applied deep convolutional neural networks to image compressed sensing tasks and significantly improved image reconstruction performance. However, the convolutional layers of deep convolutional neural networks typically have only a small receptive field, making it challenging to use CNNs to capture long-range pixel correlations, which limits their reconstruction performance in CS tasks. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the existing technology and provide an image compression sensing reconstruction method based on the Transformer enhanced residual autoencoder network. This method can better capture local and global features, effectively enhance feature information, and accurately reconstruct the original image.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] A method for image compressed sensing reconstruction based on a Transformer-enhanced residual autoencoder network includes the following steps:

[0008] 1) Perform initial linear reconstruction of the image observation value y: Use the convolution layer F in the convolutional neural network CNN in To learn the upsampling matrix, generate a series of vectors I = F in *y, use Reshape+Concat operation to reshape the vector I into an image block of size B×B×1, and splice it to get the initial reconstructed image Z in;

[0009] 2) Design and use the residual autoencoder network based on Transformer for feature enhancement to perform deep reconstruction: the initial reconstructed image Z obtained in step 1) in The residual autoencoder network based on Transformer for feature enhancement is used for depth reconstruction to obtain the depth reconstructed image Z:

[0010] First, the initial reconstructed image Z in The convolution with a kernel size of 3×3 is used to perform channel transformation to obtain the feature map Z1, as shown in formula (1):

[0011] Z1=conv(Z in ) (1);

[0012] Secondly, the feature map Z1 is passed through the residual network F which is composed of 5 residual blocks stacked together. R The shallow feature map Z2 is obtained as shown in formula (2):

[0013] Z2=F R (w1,Z1) (2),

[0014] Among them, w1 represents the network parameters;

[0015] Then the shallow feature map Z2 is fed into the Transformer enhanced autoencoder network F C , and obtain the enhanced feature map z3, as shown in formula (3):

[0016] Z3=F C (w2,Z2) (3),

[0017] Among them, w2 represents the network parameters, F C The Transformer enhanced self-encoding network represented by includes: an encoding network F for extracting features from the input shallow feature map Z2 E , a decoding enhancement network F containing a residual SwinTransformer module RSTB for feature enhancement D , and cross-layer connections for supplementing and fusing feature information;

[0018] Finally, the feature maps Z1, Z2, and Z3 are combined with the initial reconstructed image Z in Add and fuse to obtain the final depth reconstructed image Z, as shown in formula (4):

[0019] Z=(Z1+Z2+Z3)+Z in (4);

[0020] 3) Network training based on global-local joint loss function: including:

[0021] 3-1) Construct a training dataset and obtain observations y: Use 200 training images and 200 test images from the BSDS500 dataset, randomly crop them into non-overlapping image blocks of size B×B×1, and use the convolutional layer F in the convolutional neural network CNN on the input image x. s To implement the block-based sampling process, obtain the observation value y = F s *x;

[0022] 3-2) Construct a joint loss function Loss to train the network end-to-end, as shown in formula (5):

[0023] Loss=Loss int +Loss deep (5),

[0024] Among them, Loss int For the initial reconstructed image Z in The loss term is shown in formula (6):

[0025]

[0026] Loss deep is the loss term for the depth reconstruction image Z, as shown in formula (7):

[0027]

[0028] Among them, x i is the original image, Z in is the initial reconstructed image, and Z is the depth reconstructed image.

[0029] The Transformer enhanced autoencoder network F described in step 2) C for:

[0030] 2-1) Use the shallow feature map Z2 as the Transformer to enhance the autoencoder network F C The input first passes through the encoding network F E Perform feature extraction to obtain output feature f2;

[0031] Encoding network F E It consists of two 3×3 convolution kernels and a downsampling operation with a stride of 2. Each convolution is followed by an activation function ReLU, as shown in formulas (8), (9), (10), and (11):

[0032] g1=Conv2(Z2) (8),

[0033] f1=Downsampling(g1) (9),

[0034] g2=Conv2(f1) (10),

[0035] f2=Downsampling(g2) (11),

[0036] Among them, Conv2 is a combination of two 3×3 convolution kernels and the activation function ReLU, and Downsampling is a downsampling operation with a step size of 2;

[0037] 2-2) The coding network F E The output feature f2 is fed into the decoding enhancement network F D , the convolutional layer and the residual SwinTransformer module RSTB are used to enhance the feature map f2, and the output feature f4 is obtained. The decoding enhancement network F D It consists of two 3×3 convolutions, upsampling and a residual Swin Transformer module RSTB. Each convolution is followed by an activation function ReLU. At the same time, the encoding network F E The shallow features g1 and g2 obtained before the two downsamplings are connected to the decoding enhancement network F through cross-layer D After two upsamplings in , the deep features f3 and f4 of corresponding sizes are obtained, and their channel information is fused. Finally, convolution is used to convert the feature map into a single channel to obtain the final output feature Z3, as shown in formulas (12), (13), (14), (15), and (16):

[0038] g3=Conv2(f2) (12),

[0039] f3=Upsampling(RSTB(g3)) (13),

[0040] g4=Conv2(concat(g2,f3) (14),

[0041] f4=Upsampling(RSTB(g4)) (15),

[0042] Z3=conv(concat(g1, f4)) (16),

[0043] Among them, Conv2 is a combination of two 3×3 convolution kernels and the activation function ReLU, Upsampling is an upsampling operation of 2×2 deconvolution, RSTB represents a residual Swin Transformer block RSTB, concat is channel splicing, and conv is a 3×3 convolution.

[0044] The residual Swin Transformer module RSTB process is: using the feature g described in step 2-2) i As the input of the residual Swin Transformer module RSTB, where i = 3, 4, the residual Swin Transformer module RSTB is a residual block consisting of two Swin Transformer layers and a convolutional layer, as shown in formula (17):

[0045] RSTB=conv(H STL (H STL (g i )))+g i (17),

[0046] Among them, g i Represents the input features, conv is a 3×3 convolution, H STL It is the Swin Transformer layer.

[0047] The beneficial effects of this technical solution are:

[0048] This technical solution adopts deep learning technology for compressed sensing image reconstruction, adopts end-to-end training to jointly optimize the sampling and reconstruction networks, and fully utilizes the global self-attention mechanism of the residual Swin Transformer module in the decoding module based on the Transformer-enhanced residual autoencoder network to enhance image features. A cross-layer connection is used between the encoder and decoder to realize the fusion of feature maps of various dimensions, so as to retrieve the edge information lost in the decoding process and effectively improve the quality of compressed sensing reconstruction of images.

[0049] This method can better capture local and global features, effectively enhance feature information, and accurately reconstruct the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A network framework diagram of an embodiment method;

[0051] Figure 2 This is the RSTB framework diagram of the residual Swin Transformer module in the embodiment. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0053] Example:

[0054] Reference Figure 1, an image compressed sensing reconstruction method based on Transformer enhanced residual autoencoder network, comprising the following steps:

[0055] 1) Perform initial linear reconstruction of the image observation value y: Use the convolution layer F in the convolutional neural network CNN in To learn the upsampling matrix, generate a series of vectors I = F in *y, use Reshape+Concat operation to reshape the vector I into an image block of size B×B×1, and splice it to get the initial reconstructed image Z in ;

[0056] 2) Design and use the residual autoencoder network based on Transformer for feature enhancement to perform deep reconstruction: the initial reconstructed image Z obtained in step 1) in The residual autoencoder network based on Transformer for feature enhancement is used for depth reconstruction to obtain the depth reconstructed image Z:

[0057] First, the initial reconstructed image Z in The convolution with a kernel size of 3×3 is used to perform channel transformation to obtain the feature map Z1, as shown in formula (1):

[0058] Z1=conv(Z in ) (1);

[0059] Secondly, the feature map Z1 is passed through the residual network F which is composed of 5 residual blocks stacked together. R The shallow feature map Z2 is obtained as shown in formula (2):

[0060] Z2=F R (w1,Z1) (2),

[0061] Among them, w1 represents the network parameters;

[0062] Then the shallow feature map Z2 is fed into the Transformer enhanced autoencoder network F C , and obtain the enhanced feature map Z3, as shown in formula (3):

[0063] Z3=F C (w2,Z2) (3),

[0064] Among them, w2 represents the network parameters, F C The Transformer enhanced self-encoding network represented by includes: an encoding network F for extracting features from the input shallow feature map Z2 E , a decoding enhancement network F containing a residual SwinTransformer module RSTB for feature enhancement D, and cross-layer connections for supplementing and fusing feature information;

[0065] Finally, the feature maps v1, Z2, and Z3 are combined with the initial reconstructed image Z in Add and fuse to obtain the final depth reconstructed image Z, as shown in formula (4):

[0066] Z=(Z1+Z2+Z3)+Z in (4);

[0067] 3) Network training based on global-local joint loss function: including:

[0068] 3-1) Construct a training dataset and obtain observations y: Use 200 training images and 200 test images from the BSDS500 dataset, randomly crop them into non-overlapping image blocks of size B×B×1, and use the convolutional layer F in the convolutional neural network CNN on the input image x. s To implement the block-based sampling process, obtain the observation value y = F s *x;

[0069] 3-2) Construct a joint loss function Loss to train the network end-to-end, as shown in formula (5)

[0070] Loss=Loss int +Loss deep (5),

[0071] Among them, Loss int For the initial reconstructed image Z in The loss term is shown in formula (6):

[0072]

[0073] Loss deep is the loss term for the depth reconstruction image Z, as shown in formula (7):

[0074]

[0075] Among them, x i is the original image, Z in is the initial reconstructed image, and Z is the depth reconstructed image.

[0076] The Transformer enhanced autoencoder network F described in step 2) C for:

[0077] 2-1) Use the shallow feature map Z2 as the Transformer to enhance the autoencoder network F C The input first passes through the encoding network F EPerform feature extraction to obtain output feature f2;

[0078] Encoding network F E It consists of two 3×3 convolution kernels and a downsampling operation with a stride of 2. Each convolution is followed by an activation function ReLU, as shown in formulas (8), (9), (10), and (11):

[0079] g1=Conv2(Z2) (8),

[0080] f1=Downsampling(g1) (9),

[0081] g2=Conv2(f1) (10),

[0082] f2=Downsampling(g2) (11),

[0083] Among them, Conv2 is a combination of two 3×3 convolution kernels and the activation function ReLU, and Downsampling is a downsampling operation with a step size of 2;

[0084] 2-2) The coding network F E The output feature f2 is fed into the decoding enhancement network F D , the convolutional layer and the residual SwinTransformer module RSTB are used to enhance the feature map f2, and the output feature f4 is obtained. The decoding enhancement network F D It consists of two 3×3 convolutions, upsampling and a residual Swin Transformer module RSTB. Each convolution is followed by an activation function ReLU. At the same time, the encoding network F E The shallow features g1 and g2 obtained before the two downsamplings are connected to the decoding enhancement network F through cross-layer D After two upsamplings in , the deep features f3 and f4 of corresponding sizes are obtained, and their channel information is fused. Finally, convolution is used to convert the feature map into a single channel to obtain the final output feature Z3, as shown in formulas (12), (13), (14), (15), and (16):

[0085] g3=Conv2(f2) (12),

[0086] f3=Upsampling(RSTB(g3))(13),

[0087] g4=Conv2(concat(g2,f3) (14),

[0088] f4=Upsampling(RSTB(g4)) (15),

[0089] Z3=conv(concat(g1, f4))(16),

[0090] Among them, Conv2 is a combination of two 3×3 convolution kernels and the activation function ReLU, Upsampling is the upsampling operation of 2×2 deconvolution, RSTB represents a residual SwinTransformer block RSTB, concat is channel splicing, and conv is a 3×3 convolution.

[0091] The residual Swin Transformer module RSTB process is: using the feature g described in step 2-2) i As the input of the residual Swin Transformer module RSTB, where i = 3, 4, the residual Swin Transformer module RSTB is a residual block consisting of two Swin Transformer layers and a convolutional layer, as shown in formula (17):

[0092] RSTB=conv(H STL (H STL (g i )))+g i (17),

[0093] Among them, g i Represents the input features, conv is a 3×3 convolution, H STL It is the Swin Transformer layer.

Claims

1. A method for image compressed sensing reconstruction based on Transformer enhanced residual autoencoder network, characterized in that: The steps include: 1) Perform initial linear reconstruction of the image observation value y: Use the convolution layer F in the convolutional neural network CNN in To learn the upsampling matrix, generate a series of vectors I = F in *y, use Reshape+Concat operation to reshape the vector I into an image block of size B×B×1, and splice it to get the initial reconstructed image Z in ; 2) Design and use the residual autoencoder network based on Transformer for feature enhancement to perform deep reconstruction: the initial reconstructed image Z obtained in step 1) in The residual autoencoder network based on Transformer for feature enhancement is used for depth reconstruction to obtain the depth reconstructed image Z: First, the initial reconstructed image Z in The convolution with a kernel size of 3×3 is used to perform channel transformation to obtain the feature map Z1, as shown in formula (1): Z1=conv(Z in ) (1); Secondly, the feature map Z1 is passed through the residual network F which is composed of 5 residual blocks stacked together. R The shallow feature map Z2 is obtained as shown in formula (2): Z2=F R (w1,Z1) (2), Among them, w1 represents the network parameters; Then the shallow feature map Z2 is fed into the Transformer enhanced autoencoder network F c , and obtain the enhanced feature map Z3, as shown in formula (3): Z3=F c (w2, Z2) (3), Among them, w2 represents the network parameters, F c The Transformer enhanced self-encoding network represented by includes: an encoding network F for extracting features from the input shallow feature map Z2 E , a decoding enhancement network F containing a residual Swin Transformer module RSTB for feature enhancement D , and cross-layer connections for supplementing and fusing feature information; Finally, the feature maps Z1, Z2, and Z3 are combined with the initial reconstructed image Z in Add and fuse to obtain the final depth reconstructed image Z, as shown in formula (4): Z=(Z1+Z2+Z3)+Z in (4); 3) Network training based on global-local joint loss function: including: 3-1) Construct a training dataset and obtain observations y: Use 200 training images and 200 test images from the BSDS500 dataset, randomly crop them into non-overlapping image blocks of size B×B×1, and use the convolutional layer F in the convolutional neural network CNN on the input image x. s To implement the block-based sampling process, obtain the observation value y = F s *x; 3-2) Construct a joint loss function Loss to train the network end-to-end, as shown in formula (5) Loss=Loss int +Loss deep (5), Among them, Loss int For the initial reconstructed image Z in The loss term is shown in formula (6): Loss deep is the loss term for the depth reconstruction image Z, as shown in formula (7): Among them, x i is the original image, Z in is the initial reconstructed image, Z is the depth reconstructed image; The Transformer enhanced autoencoder network F described in step 2) C for: 2-1) Use the shallow feature map Z2 as the Transformer to enhance the autoencoder network F C The input first passes through the encoding network F E Perform feature extraction to obtain output feature f2; Encoding network F E It consists of two 3×3 convolution kernels and a downsampling operation with a stride of 2. Each convolution is followed by an activation function ReLU, as shown in formulas (8), (9), (10), and (11): g1=Conv2(Z2) (8), f1=Downsampling(g1) (9), g2=Conv2(f1) (10), f2=Downsampling(g2)(11), Among them, Conv2 is a combination of two 3×3 convolution kernels and the activation function ReLU, and Downsampling is a downsampling operation with a step size of 2; 2-2) The coding network F E The output feature f2 is fed into the decoding enhancement network F D , the convolutional layer and the residual SwinTransformer module RSTB are used to enhance the feature map f2, and the output feature f4 is obtained. The decoding enhancement network F D It consists of two 3×3 convolutions, upsampling and a residual Swin Transformer module RSTB. Each convolution is followed by an activation function ReLU. At the same time, the encoding network F E The shallow features g1 and g2 obtained before the two downsamplings are connected to the decoding enhancement network F through cross-layer D After two upsamplings in , the deep features f3 and f4 of corresponding sizes are obtained, and their channel information is fused. Finally, convolution is used to convert the feature map into a single channel to obtain the final output feature Z3, as shown in formulas (12), (13), (14), (15), and (16): g3=Conv2(f2) (12), f3=Upsampling(RSTB(g3)) (13), g4=Conv2(concat(g2,f3) (14), f4=Upsampling(RSTB(g4)) (15), Z3=conv(concat(g1, f4)) (16), Among them, Conv2 is a combination of two 3×3 convolution kernels and the activation function ReLU, Upsampling is an upsampling operation of 2×2 deconvolution, RSTB represents a residual Swin Transformer block RSTB, concat is channel splicing, and conv is a 3×3 convolution.

2. The image compressed sensing reconstruction method based on the Transformer enhanced residual autoencoder network according to claim 1 is characterized in that The residual Swin Transformer module RSTB process is: using the feature g described in step 2-2) i As the input of the residual Swin Transformer module RSTB, where i = 3, 4, the residual Swin Transformer module RSTB is a residual block consisting of two Swin Transformer layers and a convolutional layer, as shown in formula (17): RSTB=conv(H STL (H STL (g i )))+g i (17), Among them, g i Represents the input features, conv is a 3×3 convolution, H STL It is the Swin Transformer layer.

Images