Binocular image super-resolution reconstruction method based on global cross attention

Abstract

The invention relates to a binocular image super-resolution reconstruction method based on global cross attention. The invention provides a three-dimensional image super-resolution network (GCASR) based on global cross attention, and provides a three-dimensional image super-resolution network (GCASR) based on global cross attention. The method comprises the following steps: firstly, in a feature extraction stage, designing a Wide-area Feature Extraction Block (WAFEBlock to extract abundant image features; and secondly, in a feature interaction stage, a global cross attention module (GCAM) is designed, and effective information in a view and effective information between views are effectively combined, so that the quality of image reconstruction is improved. According to the method for constructing the GCASSR, the performance of image reconstruction can be effectively improved.

Description

Technical Field

[0001] The invention relates to a binocular image super-resolution reconstruction method based on global cross attention. Background Art

[0002] With the widespread popularity of smart phones, various social media applications such as Weibo, WeChat, blogs, etc. have emerged in an endless stream, and people's requirements for image quality and resolution are increasing. Not only do people need clearer pictures and videos in daily life, but in scientific research, the clarity of images will also affect the performance of research methods. For example, in the task of medical image segmentation, the segmentation model can better obtain tiny details, edges, textures and other features in the image through high-resolution images, and achieve more accurate segmentation. Of course, the most direct way to improve the resolution of images / videos is to improve the performance of the acquisition equipment, but improving image resolution from a hardware perspective is costly and has technical bottlenecks. Therefore, many researchers study how to improve the resolution of images / videos from a software perspective, namely super-resolution reconstruction (SR).

[0003] Super-resolution reconstruction technology converts input low-resolution (LR) images into high-resolution (HR) images by restoring the high-frequency information of the image. SR technology can save hardware costs and avoid complex physical modeling. It is widely used in medical imaging [1-2], autonomous driving [3-4], and industrial inspection [5-6]. However, SR technology still has a lot of room for improvement. Among them, stereo image super-resolution reconstruction (Stereo Image Super Resolution, StereoSR), as a branch of SR research, is still in its infancy.

[0004] StereoSR aims to restore a pair of LR images collected by a binocular system into a single HR image. Due to the redundant nature of binocular vision, directly applying existing conventional SR techniques will result in significant computational redundancy, hindering deployment in real binocular systems. In recent years, researchers have proposed a number of StereoSR methods. Because the attention mechanism can effectively capture the redundant information of another view, thereby improving the performance of the StereoSR method, more and more attention-based StereoSR methods have been proposed. Song et al. [7] proposed combining self-attention and disparity attention to aggregate information and automatically learn long-range dependencies in images, thereby better capturing global context information and reconstructing high-quality stereo SR images. Recently, Chu et al. [8] proposed a non-local feature super-resolution reconstruction network (Stereo Image Super-Resolution Using NAFNet, NAFSSR), which significantly improved the SR performance by combining non-local attention mechanism and deep feature extraction.

[0005] Although existing methods have improved the performance of image restoration by using the cross-attention mechanism, they have neglected the internal information of the view, which is crucial for capturing the details and texture features of the image and is an important factor in improving the quality of reconstruction. Summary of the invention

[0006] The purpose of the present invention is to provide a binocular image super-resolution reconstruction method based on global cross-attention to address the problems existing in the prior art, and propose a global cross-attention based stereo image super-resolution network (Global Cross-Attention Based Stereo Image Super-Resolution Network, GCASSR). First, in the feature extraction stage, a wide-area feature extraction block (Wade-area Feature Extraction Block, WAFEBlock) is designed to extract rich image features; second, in the feature interaction stage, a global cross-attention module (Global Cross-attention Module, GCAM) is designed to effectively combine the effective information within and between views to improve the quality of image reconstruction. A large number of experiments show that the method GCASSR of the present invention can effectively improve the performance of image reconstruction, surpassing the baseline and most of the existing methods in terms of PSNR / SSIM[9] indicators, and can compete with the latest stereo image reconstruction methods.

[0007] To achieve the above objectives, the technical solution of the present invention is: a binocular image super-resolution reconstruction method based on global cross-attention, and a stereo image super-resolution network GCASSR based on global cross-attention is proposed; firstly, in the feature extraction stage, a wide-area feature extraction block WAFEBlock is designed to extract rich image features; secondly, in the feature interaction stage, a global cross-attention module GCAM is designed to effectively combine the effective information within and between views to improve the quality of image reconstruction.

[0008] In one embodiment of the present invention, the left and right view branches of GCASSR adopt a weight sharing strategy.

[0009] In one embodiment of the present invention, the method is specifically implemented as follows:

[0010] First, GCASSR passes through a 3×3 convolutional layer from the left and right low-resolution images Extract initial features from Where H×W represents the spatial dimension and C represents the number of channels, which can be expressed as:

[0011]

[0012] and are left and right low-resolution images, and is the initial feature extracted from the left and right low-resolution images, C 3×3 It is a 3×3 convolution operation;

[0013] Secondly, GCASSR obtains deep features through N WAFEBlocks, and adds GCAM after the first N-1 WAFEBlocks to obtain information combined with the internal and external information of the view, which is expressed as:

[0014]

[0015] in, and represents the i-th WAFEBlock and GCAM, i∈(1,N), Represents the output after passing through i WAFEBlocks and GCAM;

[0016] Finally, a 3×3 convolutional layer and a pixel shuffle layer are used to upsample the features by a scale factor s, and global residual learning is used to reconstruct the left and right high-resolution images. It is expressed as:

[0017]

[0018] Among them, U↑ represents bilinear interpolation, S ↑ Represents the Pixel Shuffle layer.

[0019] In one embodiment of the present invention, WAFEBlock is divided into three parts: a mobile convolution module MBConv, a feedforward network FFN and a large kernel convolution module LKCM; wherein, LKCM uses a large kernel convolution attention mechanism to capture a wide range of contextual information, enabling the model to understand a wider range of features and structures in the image and extract more representative features.

[0020] In one embodiment of the present invention, MBConv performs the first step of feature extraction through the normalization layer Norm Layer, the convolution layer Conv 1×1, the activation function and the channel attention mechanism operation; FFN performs the second step of feature extraction through the normalization layer, the convolution layer and the gated activation; the left and right branches share weights. For the left view, given the input feature X org , then the first step extraction and the second step extraction process are expressed as:

[0021] X = MBConv(FFN(X org )), (4)

[0022] After the first and second step extraction of features, the third step extraction of features is performed through LKCM, and the output X after the first and second step extraction is used as the input of the third step extraction. The whole process is expressed as:

[0023] X LKCA =LKCA(LN(C 1×1 (C 3×3 (SG(X))))), (5)

[0024] Among them, LKCA represents the large kernel convolution attention mechanism, LN is the normalization layer, and C 1×1 and C 3×3 They represent 1×1 convolution operation and 3×3 convolution operation respectively, X LKCM represents the output after LKCM, SG is the gated activation function;

[0025] Finally, through 1×1 convolution, and weighted with the original features, the specific process is as follows:

[0026] X out =C 1×1 (X LKCA +X org ), (8)

[0027] Among them, X out Indicates the final output.

[0028] In one embodiment of the present invention, SG splits the input into two features X1, X2∈R along the channel dimension H×W×C / 2 , and then calculate the output through the linear gate:

[0029] SG=X1⊙X2, (6)

[0030] where ⊙ is the product of the elements.

[0031] In one embodiment of the present invention, X SG Represents the input features, and the function of LKCA is expressed as:

[0032] LKCA=C 1×1 (C 7×7 (C 5×5 (X SG )))+X SG , (7)

[0033] Among them, C n×n It is an n×n convolution operation.

[0034] In one embodiment of the present invention, GCAM includes two parts: cross attention and self attention, and the formula is expressed as:

[0035]

[0036] Among them, γ L , γ R , α L , α R is a learnable scaling parameter, X L and X R represents the left and right input features, F L and F R represents the features learned through intra-view and inter-view learning, and represents the features of the left and right views after self-learning by self-attention SAM, F R→L and F L→R They respectively represent the features contributed by the right image to the left image and the features contributed by the left image to the right image.

[0037] The features learned by SAM for the left and right views are represented as:

[0038]

[0039] in, and Represents the features of the left and right views after SAM self-learning, and Represents the query, key, and value projection matrices for the left and right views, where Attention is based on scaled dot product attention:

[0040]

[0041] Where Q, K, V represent query, key and value matrices respectively, and C is the number of channels of the matrix;

[0042] The feature representation after mutual learning between the left and right views is:

[0043]

[0044] In one embodiment of the present invention, the method uses the joint loss L in the spatial and frequency domains. COMPOSE To train the network, the specific loss function is as follows:

[0045] L COMPOSE =L SR +λ*L ftt , (16)

[0046]

[0047] in, and represents the left and right real images, and represents the left and right high-resolution images, λ is the weight, and the frequency Charbonnier loss function L ftt As shown below:

[0048]

[0049] in, stands for Fast Fourier Transform.

[0050] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects: stereoscopic image super-resolution reconstruction improves the quality of super-resolution reconstructed images by fusing the information of another view. Therefore, how to fully and effectively utilize the feature information in another view is the key to stereoscopic image super-resolution reconstruction. Although the existing methods use the cross-attention mechanism to improve the performance of image restoration, they neglect the internal information of the view. The internal information of the view is crucial to capturing the details and texture features of the image and is an important factor in improving the reconstruction quality. In view of the problem of insufficient extraction of internal and external features in stereoscopic image super-resolution reconstruction technology, this paper proposes a stereoscopic image super-resolution reconstruction network based on global cross-attention. Specifically, the present invention designs a wide-area feature extraction strategy, introduces large kernel convolution attention in the feature extraction stage to enhance the receptive field, thereby enriching the extraction of image features. In addition, in the feature interaction stage, the present invention proposes global cross-attention to combine the effective information within and between views to improve the image reconstruction quality. On four datasets, KITTI 2012, KITTI 2015, Middlebury and Flickr1024, the present invention conducts a large number of experiments to demonstrate the effectiveness of the proposed method, which surpasses most of the existing methods and can compete with the best methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the GCASSR network architecture diagram of the present invention.

[0053] Figure 2 It is a wide-area feature extraction module.

[0054] Figure 3 For mobile convolution modules and feed-forward networks.

[0055] Figure 4 It is a large kernel convolution attention module.

[0056] Figure 5 It is a global criss-cross attention module.

[0057] Figure 6 It is the self-attention module.

[0058] Figure 7 Visual comparison of StereoSR methods on different datasets (×4).

[0059] Figure 8 Visual comparison of StereoSR methods on Flickr1024 and KITTI2012 datasets (×2). DETAILED DESCRIPTION

[0060] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0061] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0062] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0063] The present invention provides a binocular image super-resolution reconstruction method based on global cross-attention, and proposes a stereo image super-resolution network GCASSR based on global cross-attention. Firstly, in the feature extraction stage, a wide-area feature extraction block WAFEBlock is designed to extract rich image features. Secondly, in the feature interaction stage, a global cross-attention module GCAM is designed to effectively combine the effective information within and between views to improve the quality of image reconstruction.

[0064] The following is the specific implementation process of the present invention.

[0065] 1. Method of the present invention

[0066] 1.1 Network Architecture

[0067] like Figure 1 As shown, this embodiment provides a binocular image super-resolution reconstruction method based on global cross attention, and proposes a stereo image super-resolution network GCASSR based on global cross attention. The network structure diagram is shown in Figure 1 In order to reduce the computational complexity, the left and right view branches of the network adopt a weight sharing strategy.

[0068] First, GCASSR passes through a 3×3 convolutional layer (Conv 3×3) from the left and right low-resolution images. Extract the initial features Where H×W represents the spatial dimension and C represents the number of channels, which can be expressed as:

[0069]

[0070] in, and are left and right low-resolution images, and is the initial feature extracted from the left and right low-resolution images, C 3×3It is a 3×3 convolution operation.

[0071] Secondly, the model obtains deep features through N WAFEBlocks, and adds GCAM after the first N-1 WAFEBlocks to obtain the information combined inside and outside the view, expressed as:

[0072]

[0073] in, and represents the i-th WAFEBlock and GCAM, i∈(1,N), represents the output after passing through i WAFEBlocks and GCAMs. The setting of N is proposed in Section 2, and the WAFEBlock and GCAM modules will be introduced in detail in 1.2 and 1.3.

[0074] Finally, the present invention uses a 3×3 convolutional layer and a pixel shuffle layer

[10] to upsample the features by a scale factor s and adopts global residual learning

[11] to reconstruct the left and right high-resolution images. It can be expressed as:

[0075]

[0076] Among them, U ↑ represents bilinear interpolation, S ↑ Represents the Pixel Shuffle layer.

[0077] 1.2. Wide Area Feature Extraction Module (WAFEBlock)

[0078] Wide area feature extraction module (WAFEBlock) such as Figure 2 As shown in the figure, the module is mainly divided into three parts: mobile convolution module (MBConv), feedforward network (FFN) and large kernel convolution module (Large Kernel Convolution Module, LKCM). Among them, the large kernel convolution module uses the large kernel convolution attention mechanism

[12] Capturing a wide range of contextual information enables the model to understand a wider range of features and structures in the image, thereby extracting more representative features.

[0079] The flowcharts of MBConv and FFN are as follows Figure 3As shown in Figure 2. MBConv extracts features in the first step through the normalization layer (Norm Layer), convolution layer (Conv1×1), activation function and channel attention (ChannelAttention) mechanism. FFN extracts features in the second step through the normalization layer, convolution layer and gated activation (Simple Gate). Since the left and right branches share weights, taking the left view as an example, given the input feature X orh , the process can be expressed as:

[0080] X = MBConv(FFN(X org )), (4)

[0081] After the first and second step feature extraction, the present invention performs the third step feature extraction through LKCM. The present invention uses the output X after the first and second step feature extraction as the input of the third step feature extraction. The whole process can be expressed as:

[0082] X LKCA =LKCA(LN(C 1×1 (C 3×3 (SG(X))))), (5)

[0083] Among them, LKCA represents the large kernel convolution attention mechanism, LN is the normalization layer, and C 1×1 and C 3×3 represents 1×1 and 3×3 convolution operations, X LKCM represents the output after LKCM, and SG is the gated activation function.

[0084] The SG function splits the input into two features X1,X2∈R along the channel dimension H×W×C / 2 , and then calculate the output through the linear gate:

[0085] SG=X1⊙X2, (6)

[0086] where ⊙ is the product of the elements.

[0087] The LKCA attention module is Figure 4 As shown. Use X SG Represents the input features, the function of the LKCA module can be expressed as:

[0088] LKCA=C 1×1 (C 7×7 (C 5×5 (X SG )))+X SG , (7)

[0089] Among them, C n×n It is an n×n convolution operation.

[0090] Finally, through 1×1 convolution, and weighted with the original features, the specific process can be expressed as:

[0091] X out =C 1×1 (X LKCA +X org ), (8)

[0092] Among them, X out Indicates the final output.

[0093] 1.3 Global Cross-Attention Module (GCAM)

[0094] like Figure 5 As shown in the figure, the global cross attention module GCAM mainly includes two parts: cross attention and self-attention. Different from other StereoSR methods, the present invention introduces a self-attention module (SAM) in the feature interaction stage, which can not only learn the information between views, but also learn the information within the view. The formula of the whole process can be expressed as:

[0095]

[0096] Among them, γ L , γ R and α L , α R is a learnable scaling parameter, X L and X R represents the left and right input features, F L and F R represents the features learned through intra-view and inter-view learning, and represents the features of the left and right views after SAM self-learning, F R →L and F L→R They respectively represent the features contributed by the right image to the left image and the features contributed by the left image to the right image.

[0097] The features learned by the SAM module for the left and right views can be expressed as:

[0098]

[0099] in, and Represents the features of the left and right views after SAM self-learning, and Represents the query and key-value projection matrices of the left and right views, where Attention is based on scaled dot product attention

[13] .

[0100]

[0101] Among them, Q, K, and V represent query, key, and value matrices respectively.

[0102] The features learned from each other between the left and right views can be expressed as:

[0103]

[0104] 1.4 Loss Function

[0105] The StereoSR task mainly focuses on restoring high-frequency details. In order to effectively restore clear and sharp high-frequency textures, the present invention utilizes the joint (COMPOSE) loss L in the spatial and frequency domains. COMPOSE To train the network of the present invention. The specific loss function is as follows:

[0106] L COMPOSE =L SR +λ*L ftt , (16)

[0107]

[0108] in, and represents the left and right real images, and represents the left and right high-resolution images, and λ is a weight set to 1.0 based on experience. The present invention uses the frequency Charbonnier loss function

[14] L ftt As shown below:

[0109]

[0110] in, Fast Fourier Transform

[15] , L ftt It can make the network pay more attention to high-frequency information.

[0111] 2. Performance comparison

[0112] The present invention uses 800 stereo images from Flickr1024 as the training set and another 112 stereo images as the validation set.

[16] 、KITTI 2015

[17] , Middlebury

[18] and Flickr1024

[19] Tested on four datasets.

[0113] The number of channels C and the number of blocks N of GCASSR are set to 64 and 32 respectively. GCASSR is optimized using AdamW, with β1=0.9, β2=0.9, and the weights are set to 0 by default. The network is built in PyTorch

[20] The model was implemented on the platform and trained on a workstation with 4 Nvidia RTX 3090 graphics cards. The model was trained for a total of 1×10 5 times, and the learning rate is initially set to 3.0×10 -3 , the cosine strategy is used to decay during training

[21] The strategy gradually decreases to 1×10 -7 .

[0114] After training, the present invention uses two objective indicators, Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), for quantitative evaluation, and qualitative analysis through the visual effect of the reconstructed image. In order to verify the effectiveness of GCASSR, the present invention compares GCASSR with existing SR methods, including four SISR methods (i.e., VDSR

[22] , EDSR

[23] 、RDN

[24] and RCAN

[25] ) and 9 StereoSR methods (i.e. StereoSR

[26] 、PASSRnet

[27] 、IMSSRnet

[28] 、iPASSR

[29] 、SSRDE-Fnet

[30] 、SRRes+SAM

[31] , Steformer

[32] 、NAFSSR-S [8] and CANSSR-S

[33] ).

[0115] 2.1 Quantitative Evaluation

[0116] The GCASSR method proposed in the present invention is compared with the existing SR methods at ×4 and ×2 sampling factors, and the results are shown in Tables 1 and 2. From the two tables, it can be concluded that the reconstruction effect of the proposed method on the four data sets exceeds that of all the methods mentioned in the table.

[0117] Specifically, on the dataset Flickr1024 (×4), it can be seen that the PSNR result of GCASSR is much higher than the PSNR values ​​of other methods in the table. Compared with the baseline NASSR-S, GCASSR is improved by 0.19dB and obtains better results. Compared with some mainstream StereoSR methods, the PSNR of the present invention is 2.37dB, 0.63dB, 0.80dB and 0.49dB higher respectively. Compared with the second best method CANSSR-S in the table, the PSNR of the present invention is improved by 0.08dB and the SSIM is improved by 0.005. In addition, on the dataset Flickr1024 (×2), the method proposed by the present invention is also higher than other methods in the table. Compared with the baseline NAFSSR-S, the method of the present invention is 0.33dB higher. Compared with the second best method CANSSR-S in the table, the PSNR of the present invention is improved by 0.16dB.

[0118] Table 1 PSNR / SSIM comparison results of GCASSR method and other methods (×4)

[0119]

[0120] Table 2 PSNR / SSIM comparison results of GCASSR method and other methods (×2)

[0121]

[0122] Note: The bold font is the method of the present invention, the italic font is the baseline method, and the underlined font is the second best method

[0123] like Figure 7 As shown in the figure, the present invention visualizes the HR images (×4) reconstructed by different StereoSR methods on four datasets: Flickr1024, KITTI 2012, KITTI 2015 and Middlebury. Specifically, in the Flickr1024_test_000019 picture, the GCASSR method of the present invention can clearly restore the window guardrail; in the KITTI2012_test_000004 picture, the vertical line texture on the balcony of the GCASSR reconstructed image is more obvious; in the KITTI2014_test_000002 picture, GCASSR eliminates more letter artifacts; in the GCASSR reconstruction result of Middlebury_test_motorcycle, the wire lines above the switch and the gap between the wood are also clearer, and the image details are better restored.

[0124] 2.2 Qualitative Evaluation

[0125] In addition, Figure 8 As shown, the present invention also shows the visualization results of SR (×2) reconstruction of two datasets Flickr1024 and KITTI 2012, which further proves the superiority of GCASSR. From the visualization results of Flickr1024_test_000010 and KITTI2012_test_000002 pictures, the SR image reconstructed by the method of the present invention has clearer lines and is closer to the reference image.

[0126] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] References:

[0132] [1] Wu Lei, Lü Guoqiang, Zhao Chen, et al. Super-resolution reconstruction of CT images based on multi-scale residual network[J]. Liquid Crystals and Displays, 2019, 34(10): 1006-1012.

[0133] [2] Liu Kewen, Ma Yuan, Xiong Hongxia, et al. Medical image super-resolution reconstruction method based on residual channel attention network[J]. Laser & Optoelectronics Progress, 2020, 57(2): 1014-1023.

[0134] [3]WAN L, SUN Y, SUN L, et al.Deep Learning Based Autonomous VehicleSuper Resolution DOA Estimation for Safety Driving[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 22(7): 4301-4315.

[0135] [4] PARK M, SON H, KIM E. Implicit Point Function for LiDAR Super-Resolution in Autonomous Driving[J]. IEEE Robotics and Automation Letters, 2023, 8(11):7073-7079.

[0136] [5] LE H, LE M. Development of Magnetic Image Super-resolution Model for Nondestructive Testing[J]. Measurement Science and Technology, 2023, 34(12): 5907-5920.

[0137] [6] Xiao Chuangbai, Duan Juan, Yu Jing. POCS super-resolution reconstruction method for sequence images[J]. Journal of Beijing University of Technology, 2009, 35(1): 108-113.

[0138] [7]SONG W,CHOI S,JEONG S,et al.Stereoscopic Image Super-resolutionwith Stereo Consistent Feature[C].AAAI Conference onArtificial Intelligence,2020,34(7):12031-12038.

[0139] [8]CHU

[0140] [9]HORE A,ZIOU D.Image Quality Metrics:PSNR vs.SSIM[C].InternationalConference on Pattern Recognition.2010:2366-2369.

[0141]

[10] LI J, FANG F, MEI K, et al.Multi-scale Residual Network for ImageSuper-resolution[C].European Conference on ComputerVision, 2018:517-532.

[0142]

[11] YANG W, FENG J, YANG J, et al.Deep Edge Guided Recurrent ResidualLearning for Image Super-resolution[J]. IEEE Transactions on Image Processing, 2017, 26(12):5895-5907.

[0143]

[12] LI C,ZENG B,LU Y,et al.LKCA:Large Kernel Convolutional Attention[J].arXiv preprint,2024,arXiv:2401.05738.

[0144]

[13] VASWANI A.Attention is all you need[J].Advances in NeuralInformation Processing Systems,2017:1-11.

[0145]

[14] BARRON J T.A General and Adaptive Robust Loss Function[C].IEEEConference on ComputerVision and Pattern Recognition.2019:4331-4339.

[0146]

[15] DUHANMEL P,VETTERLI M.Fast Fourier Transforms:A Tutorial ReviewandA State of the Art[J].Signal Processing,1990,19(4):259-299.

[0147]

[16] GEIGER A,LENZ P,URTASUN R.Are We Ready for Autonomous Driving?theKitti Vision Benchmark suite[C].IEEE Conference on Computer Vision andPattern Recognition,2012:3354-3361.

[0148]

[17] MENZE M,GEIGERA.Object Scene Flow forAutonomous Vehicles[C].IEEEConference on ComputerVision andPatternRecognition.2015:3061-3070.

[0149]

[18] SCHARSTEIN D, H,KITAJIMA Y,et al.High-resolutionStereo Datasets with Subpixel-accurate Ground Truth[C].German ConferencePattern Recognition,2014:31-42.

[0150]

[19] WANG Y,WANG L,YANG J,et al.Flickr1024:A Large-scale Dataset forStereo Image Super-resolution[C].IEEE International Conference onComputerVisionWorkshops,2019:1-6.

[0151]

[20] PASZKE A,GROSS S,MASSA F,et al.Pytorch:An Imperative Style,High-performance Deep Learning Library[C].Advances inNeural Information ProcessingSystems,2019,32:1-12.

[0152]

[21] LOSHCHILOV I,HUTTER F.SGDR:Stochastic Gradient Descent with WarmRestarts[J].2016,arXiv:1608.03983.

[0153]

[22] KIM J,LEE J K,LEE K M.Accurate Image Super-resolution Using VeryDeep Convolutional Networks[C].IEEE Conference on Computer Vision and PatternRecognition.2016:1646-1654.

[0154]

[23] LIM B,SON S,KIM H,et al.Enhanced Deep Residual Networks forSingle Image Super-resolution[C].IEEE Conference on ComputerVision andPatternRecognitionWorkshops,2017:136-144.

[0155]

[24] ZHANG Y,TIAN Y,KONG Y,et al.Residual Dense Network for ImageSuper-resolution [C].IEEE Conference on ComputerVision andPatternRecognition.,2018:2472-2481.

[0156]

[25] ZHANG Y,LI K,Li K,et al.Image Super-resolution Using Very DeepResidual Channel AttentionNetworks[C].European Conference on ComputerVision,2018:286-301.

[0157]

[26] JEON D S,BAEK S H,CHOI I,et al.Enhancing the Spatial Resolutionof Stereo Images Using A Parallax Prior[C].IEEE Conference on Computer Visionand Pattern Recognition,2018:1721-1730.

[0158]

[27] DONG C,LOY C C,HE K,et al.Learning a Deep Convolutional Networkfor Image Super-resolution[C].European Conference Computer Vision,2014:184-199.

[0159]

[28] LEI J,ZHANG Z,FAN X,et al.Deep Stereoscopic Image Super-resolution via Interaction Module[J].IEEE Transactions on Circuits andSystems for Video Technology,2020,31(8):3051-3061.

[0160]

[29] WANG Y,YING X,WANG L,et al.Symmetric parallax attention forstereo image super-resolution[C].IEEE Conference on Computer Vision andPattern Recognition,2021:766-775.

[0161]

[30] DAIQ,LI J,YIQ,et al.Feedback Network for Mutually Boosted StereoImage Super-resolution and Disparity Estimation[C].International Conferenceon Multimedia,2021:1985-1993.

[0162]

[31] YING X,WANG Y,WANG L,et al.A Stereo Attention Module for StereoImage Super-resolution[J].IEEE Signal Processing Letters,2020,27:496-500.

[0163]

[32] LIN J,YIN L,WANG Y.Steformer:Efficient Stereo Image Super-resolution with Transformer[J].IEEE Transactions on Multimedia,2023,25:8396-8407.

[0164]

[33] LIU A, LI S, CHANG Y, et al. Coarse-to-Fine Cross-View InteractionBasedAccurate Stereo Image Super-Resolution Network[J]. IEEE Transactions onMultimedia, 2024, (26): 7321-7334.

[0165] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A binocular image super-resolution reconstruction method based on global cross attention, characterized in that: A stereo image super-resolution network GCASSR based on global cross attention is proposed. Firstly, in the feature extraction stage, a wide area feature extraction block WAFEBlock is designed to extract rich image features. Secondly, In the feature interaction stage, a global cross-attention module (GCAM) is designed to effectively combine the effective information within and between views to improve the quality of image reconstruction.

2. The binocular image super-resolution reconstruction method based on global cross attention according to claim 1 is characterized in that: The left and right view branches of GCASSR adopt a weight sharing strategy.

3. The binocular image super-resolution reconstruction method based on global cross attention according to claim 2 is characterized in that: The method is specifically implemented as follows: First, GCASSR passes through a 3×3 convolutional layer from the left and right low-resolution images Extract initial features from Where H×W represents the spatial dimension and C represents the number of channels, which can be expressed as: and are the left and right low-resolution images, and is the initial feature extracted from the left and right low-resolution images, C 3×3 It is a 3×3 convolution operation; Secondly, GCASSR obtains deep features through N WAFEBlocks, and adds GCAM after the first N-1 WAFEBlocks to obtain information combined with the internal and external information of the view, which is expressed as: in, and represents the i-th WAFEBlock and GCAM, i∈(1,N), Represents the output after passing through i WAFEBlocks and GCAM; Finally, a 3×3 convolutional layer and a pixel shuffle layer are used to upsample the features by a scale factor s, and global residual learning is used to reconstruct the left and right high-resolution images. It is expressed as: Among them, U ↑ represents bilinear interpolation, S ↑ Represents the Pixel Shuffle layer.

4. The binocular image super-resolution reconstruction method based on global cross attention according to claim 1 or 3, characterized in that: WAFEBlock is divided into three parts: mobile convolution module MBConv, feedforward network FFN and large kernel convolution module LKCM; among them, LKCM uses large kernel convolution attention mechanism to capture a wide range of contextual information, enabling the model to understand a wider range of features and structures in the image and extract more representative features.

5. The binocular image super-resolution reconstruction method based on global cross attention according to claim 4 is characterized in that: MBConv extracts features in the first step through the normalization layer, convolution layer Conv 1×1, activation function and channel attention mechanism operation; FFN extracts features in the second step through the normalization layer, convolution layer and gated activation; the left and right branches share weights. For the left view, given the input feature X org , then the first step extraction and the second step extraction process are expressed as: X=MBConv(FFN(X org )), (4) After the first and second step extraction of features, the third step extraction of features is performed through LKCM, and the output X after the first and second step extraction is used as the input of the third step extraction. The whole process is expressed as: X LKCA =LKCA(LN(C 1×1 (C 3×3 (SG(X))))),(5) Among them, LKCA represents the large kernel convolution attention mechanism, LN is the normalization layer, and C 1×1 and C 3×3 They represent 1×1 convolution operation and 3×3 convolution operation respectively, X LKCM represents the output after LKCM, SG is the gated activation function; Finally, through 1×1 convolution, and weighted with the original features, the specific process is as follows: X out =C 1×1 (X LKCA +X org ),(8) Among them, X out Indicates the final output.

6. The binocular image super-resolution reconstruction method based on global cross attention according to claim 5, characterized in that: SG splits the input into two features X1,X2∈R along the channel dimension H×W×C / 2 , and then calculate the output through the linear gate: SG=X1⊙X2,(6) where ⊙ is the product of the elements.

7. The binocular image super-resolution reconstruction method based on global cross attention according to claim 5, characterized in that: Use X SG Represents the input features, and the function of LKCA is expressed as: LKCA=C 1×1 (C 7×7 (C 5×5 (X SG )))+X SG ,(7) Among them, C n×n It is an n×n convolution operation.

8. The binocular image super-resolution reconstruction method based on global cross attention according to claim 1 or 3, characterized in that: GCAM includes two parts: cross attention and self attention. The formula is expressed as: Among them, γ L , γ R , α L , α R is a learnable scaling parameter, X L and X R represents the left and right input features, F L and F R represents the features learned through intra-view and inter-view learning, and represents the features of the left and right views after self-learning by self-attention SAM, F R→L and F L→R They respectively represent the features contributed by the right image to the left image and the features contributed by the left image to the right image. The features learned by SAM for the left and right views are represented as: in, and Represents the features of the left and right views after SAM self-learning, and Represents the query, key, and value projection matrices for the left and right views, where Attention is based on scaled dot product attention: Where Q, K, V represent query, key and value matrices respectively, and C is the number of channels of the matrix; The feature representation after mutual learning between the left and right views is:

9. The binocular image super-resolution reconstruction method based on global cross attention according to claim 1, characterized in that: The proposed method utilizes the joint loss L in spatial and frequency domains COMPOSE To train the network, the specific loss function is as follows: L COMPOSE =L SR +λ*L ftt ,(16) in, and represents the left and right real images, and represents the left and right high-resolution images, λ is the weight, and the frequency Charbonnier loss function L ftt As shown below: in, stands for Fast Fourier Transform.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images