Binocular image super-resolution reconstruction method based on multi-level reinforced attention mechanism

Abstract

The application provides a binocular image super-resolution reconstruction method based on a multi-level reinforced attention mechanism, in which multiple attentions are used for feature enhancement in neural network training, feature information in views is fully utilized for fusion processing, a loss function related to a frequency domain is used for constraint processing on the frequency domain, low-frequency information and the retention of the overall structure of an image are strengthened, and the binocular image after super-resolution is recovered to better effect, and clearer texture and edge details are recovered.

Description

Technical Field

[0001] This invention relates to the field of binocular image super-resolution technology, and in particular to a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism. Background Technology

[0002] A straightforward way to achieve binocular image super-resolution is to apply single-image super-resolution algorithms to the left and right images respectively. Attention mechanisms are a crucial research area in deep learning, and in recent years, several high-performance single-image super-resolution algorithms have emerged, such as RCAN (based on channel attention), PAN (based on pixel attention), SwinIR (based on Transformer self-attention), and MAN (based on multi-scale large kernel attention). However, reconstructing binocular images independently using only single-image super-resolution methods relies solely on intra-image self-similarity to recover details, neglecting the additional information that can be utilized across views—i.e., cross-view similarity—thus limiting further improvements in super-resolution performance. Therefore, fully utilizing cross-view information can help reconstruct higher-quality super-resolution images, as one view may provide supplementary information about the same scene region relative to another. With societal demands, various adaptable super-resolution reconstruction techniques have been proposed, and binocular image super-resolution reconstruction technology has been applied to research foundations in various fields, demonstrating significant applied research value. This led to the development of PASSRnet based on parallax attention, iPASSR based on bidirectional parallax attention, SwinFSR based on Transformer self-attention, and CVHSSR based on large kernel convolutional attention.

[0003] While numerous attempts have been made to combine various attention mechanisms with stereo image super-resolution in order to effectively extract more features from internal and cross-views, most stereo image super-resolution methods remain unsatisfactory in restoring natural textures and edge details, a problem that remains unresolved. Therefore, how to effectively utilize the inter-view dependencies between different attention features based on multiple attention mechanisms to reconstruct super-resolution stereo images requires further exploration. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism. This method fully utilizes the feature information within the view for fusion processing and uses a frequency domain-related loss function to process the frequency domain, thereby enhancing the preservation of low-frequency information and the overall structure of the image. This results in better recovery of the binocular image after super-resolution, and restores clearer textures and edge details.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism, comprising the following steps:

[0006] Step S1: Establish a stereo image training set; divide the stereo image super-resolution dataset into a training set and a test set. The low-resolution images are generated by bicubic downsampling. During the training phase, the generated low-resolution images are cropped into small patches, and the corresponding high-resolution images are also cropped. At the same time, these small patches are randomly flipped horizontally and vertically to enhance the training data.

[0007] Step S2: Establish and train a binocular super-resolution reconstruction network model based on a multi-level enhanced attention mechanism; the network takes a pair of low-resolution RGB binocular images as input and generates super-resolution binocular images.

[0008] Step S3: Construct the loss function; use the L1 loss function combined with the frequency domain loss function to enhance the supervision of the high-level feature space and constrain the training of the network.

[0009] Step S4: Set training parameters to train the network;

[0010] Step S5: Test network performance; use low-resolution stereo image pairs as test samples and input them into the network model trained in the previous step to obtain super-resolution stereo image pairs. Use objective evaluation metrics and visual effect comparison to verify the effect of super-resolution.

[0011] In a preferred embodiment, specifically, the binocular super-resolution reconstruction network model based on a multi-level enhanced attention mechanism includes two left and right weight-sharing network branches; in each weight-sharing network, a hybrid attention information extraction module is stacked to extract the intraview channels and spatial features of the left and right images; the binocular interactive view attention module is used to capture the globally corresponding information and cross-view information extracted from the left and right binocular images; specifically, it is divided into three parts: intraview feature extraction, interactive view feature fusion, and binocular image reconstruction.

[0012] In a preferred embodiment, step 2 specifically includes the following steps:

[0013] Step S21, In-view Feature Extraction; In the feature extraction stage, the input binocular image is first... The input is fed into a 3×3 convolutional layer to extract shallow features and generate high-dimensional features. Where C is the number of feature channels; then the high-dimensional features are input into the stacked multi-attention enhancement blocks for intraview feature extraction to obtain more local features and interaction information, and restore more accurate texture details; the multi-attention enhancement blocks include a hybrid attention information extraction module and a binocular interactive view attention module;

[0014] The hybrid attention information extraction module is the basic module of the left and right branches of the network. It extracts features from the view more deeply by capturing remote and local dependencies. The hybrid attention information extraction module consists of two sequentially connected modules: the first is a simplified channel and spatial information extraction module, and the second is a feedforward network module for residual information aggregation. The computation process of the two parts is as follows:

[0015] In the first module, after layer normalization, a 1×1 convolutional layer is used to expand the channels of the input feature map. The output is then passed through a 3×3 depthwise convolution to capture the local context of each channel; a cross-activation structure A-unit is then used to further learn an effective representation of the spatial context; the next step is to simplify the channel spatial attention module, fully utilizing both channel and spatial attention mechanisms, given the original input. First, average pooling and 1×1 convolution are used to learn the inter-channel relationships of the feature maps of a given input image, realizing the functions of global spatial information aggregation and channel information interaction, and outputting simplified channel attention features X1. Then, average pooling and max pooling operations are used to aggregate the spatial information of the feature maps, and a simplified spatial attention map is obtained by combining 3×3 convolution and the sigmoid function. Finally, the element-wise multiplication of the input feature maps and the output of the sigmoid layer is used as the output X2 of the simplified spatial attention module. The simplified channel spatial attention module is represented as follows:

[0016]

[0017]

[0018] In the formula W C (·),H AP (·) represent 1×1 convolution and average pooling operations, respectively, H AP,1 (·),H MP,1 (·) represents average pooling and max pooling operations on the first dimension, respectively. H cat (·) indicates concatenation in a form with dimension 1, σ(·) represents the Sigmoid activation function, and Θ represents element-wise multiplication;

[0019] After simplifying the channel and spatial information extraction modules, a 1×1 convolutional inverse transform is performed on the feature mapping channels to produce adaptive feature refinement, resulting in the first module's result. In the second module, after normalizing the output of the previous module, a residual information aggregation feedforward network containing B-units with cross-activation structures is used to improve local context awareness. Specifically, given an input tensor... First, a 1×1 convolutional layer is used to expand X′ to a higher dimension. Where k is the expansion ratio; next, a 3×3 deep convolutional layer is used to encode the information of the neighboring pixel positions of X′1, and then the CAS-B unit is used as the activation function of the deep convolutional layer, and the number of feature channels is halved in the output; finally, the initial input dimension X′2 is remapped through a 1×1 convolutional layer.

[0020] The above process can be represented as:

[0021]

[0022]

[0023] In the formula W C (·),W D These represent layer normalization, 1×1 convolution, and 3×3 depthwise convolution, respectively. CAS.B(·) represents the B unit of the cross-activation structure.

[0024] Finally, just like the previous module, the input of the next module and the output of the convolutional layer are added together to obtain the final result.

[0025] Step S22, cross-view feature fusion: After the hybrid attention information extraction module for the left and right branches, the binocular interactive view attention module is used; the binocular interactive view attention module uses the binocular features generated in the previous step as input to perform bidirectional cross-view interaction and generate interactive features fused with the input features of that view; specifically, given the input binocular view features... After layer normalization and 1×1 convolution operation, the binocular features are obtained. and in It's a 1×1 convolution; then, channel weights are generated by performing a fast 1D convolution of size k. Here, k is adaptively determined through the mapping of the channel dimension C, and the aggregated features are obtained by multiplying the channel weights element-wise with the binocular features. As shown below:

[0026]

[0027]

[0028] In the formula, H MP (·) represent a k×k convolutional layer and a max pooling operation, respectively, and Θ represents element-wise multiplication;

[0029] By calculating the attention matrix once, F is generated simultaneously. R→L ,F L→R Finally, the interactive cross-view information and the internal view information FL and FR are merged by adding them element by element, according to... In the formula It is a query matrix that projects features within the source view (such as the left view). It is a key-value matrix representing the projection of features within the target view (such as the right view); represented as follows:

[0030]

[0031]

[0032] In the formula, γ L and γ R The trainable channels are scaled and initialized to zero for stable training.

[0033] It is a 1×1 convolution;

[0034] Step S23, Binocular Image Reconstruction: After feature extraction, the output is fed to a 3×3 convolutional layer and a spatial attention enhancement module. Finally, a pixel recombination operation is used to upsample the output features to a high-resolution size, and a global residual path is used to further improve the super-resolution performance by utilizing the input binocular image information, thus recovering the super-resolution images of the left and right views.

[0035]

[0036] In the formula H C (·),H E (·),H P (·),H ↑ (·) represent the upsampling operations of convolution, enhanced spatial attention module, pixel recombination, and bilinear interpolation, respectively.

[0037] The enhanced spatial attention module will provide the given input Sending to a 1×1 convolutional layer yields Among them W C (·) is a 1×1 convolution to reduce the channel size of the input features; then, the block uses cross-row convolution and cross-row max pooling layers to reduce the spatial size; after a set of convolutions, in order to extract features, upsampling based on bilinear interpolation is performed to restore the spatial size; combined with residual connectivity, the features are further processed to obtain a 1×1 convolutional layer to restore the channel size; finally, the attention matrix is ​​generated by the Sigmoid function and multiplied by the original input features X″.

[0038] In a preferred embodiment, in step S3, the loss of the total difference is written as:

[0039] L = L SR +λL FFT , (10)

[0041] In the formula, L SR ,L FFT Let represent the L1 reconstruction loss function and the frequency domain loss of the Charbonnier loss, respectively, and let λ represent the hyperparameter used to control the frequency Charbonnier loss function; the parameter λ is set to 0.1 in all experiments.

[0042] SR reconstruction loss; the SR reconstruction loss is essentially an L1 loss function; it uses the pixel-level L1 distance between the super-resolution and the ground truth stereo image to obtain the PSNR; expressed as follows:

[0043]

[0044] In the formula, These are the left and right super-resolution images generated by the model, respectively. These are their high-resolution images;

[0045] Frequency domain loss; introducing frequency Charbonnier loss; expressed as follows:

[0046]

[0047] In the formula, the constant ε is experimentally set to 10. -3 FFT(·) represents Fast Fourier Transform.

[0048] In a preferred embodiment, in step S4, AdamW is used for optimization, where β1 = 0.9, β2 = 0.9, and the weights are defaulted to 0; the learning rate is initially set to 1×10⁻⁶. -3 And reduced to 1×10 using a cosine annealing strategy. -7 The model is trained on 30×90 patches; during training, each batch of 32 samples is evenly distributed across 8 parts, with 2×10 iterations. 5 Second-rate.

[0049] Compared with existing technologies, the present invention has the following beneficial effects: The present invention provides a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism. By constructing a network model based on a multi-level enhanced attention mechanism, the present invention integrates multiple attention mechanisms, comprehensively and efficiently enhancing the interaction between intra-view information and cross-views, better extracting the super-resolution information that is not fully utilized in the left and right views of the binocular image, expanding the receptive field while reducing the computational load; using a new cross-attention module, utilizing an efficient channel attention mechanism, a good balance is achieved in terms of efficient interaction; the channel features and spatial features used for fusion propagate important information forward through the long-range dependencies between features, effectively improving the robustness and generalization ability of detection, improving the super-resolution effect at certain edges, better restoring the natural texture of the image, and obtaining better super-resolution results with less computation. Attached Figure Description

[0050] Figure 1 This is a flowchart of a binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism, according to a preferred embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the binocular super-resolution image network structure according to a preferred embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the hybrid attention information extraction module according to a preferred embodiment of the present invention;

[0053] Figure 4 This is a simplified schematic diagram of the channel space attention module according to a preferred embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the cross-activation structure of a preferred embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the binocular interactive view attention module according to a preferred embodiment of the present invention;

[0056] Figure 7 This is a binocular image super-resolution result diagram shown in a preferred embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] This invention provides a binocular image super-resolution method based on a multi-level enhanced attention mechanism, comprising the following four steps:

[0061] First, a stereo image training set is established. Existing publicly available stereo image super-resolution datasets are used as training samples, with low-resolution images generated through bicubic downsampling. During training, the generated low-resolution images are cropped into small patches, and the corresponding high-resolution images are also cropped. These patches are then randomly flipped horizontally and vertically to enhance the training data.

[0062] Second, the network structure is designed. The overall network consists of three parts: intra-view feature extraction, interactive view feature fusion, and binocular image reconstruction. In the intra-view feature extraction stage, the input binocular image is first fed into a convolutional layer to extract shallow features and generate high-dimensional features. Then, the high-dimensional features are fed into stacked multi-attention enhancement blocks for intra-view feature extraction to obtain more local features and interaction information, thus restoring more accurate texture details. In the cross-view feature fusion stage, to capture the cross-information between the left and right views, a binocular interactive view attention module is used after the hybrid attention information extraction module of the left and right branches. The binocular interactive view attention module uses the binocular features generated by the previous hybrid attention information extraction module as input to perform bidirectional cross-view interaction and generate interactive features fused with the input features of that view. In the binocular image reconstruction stage, after the fused feature extraction, the output is fed into a convolutional layer and a spatial attention enhancement module. Finally, a pixel recombination operation is used to upsample the output feature tensor to restore the super-resolution left and right view images.

[0063] Third, construct the loss function. To enhance the texture details of binocular images and maintain disparity consistency between viewpoints, this invention employs a method combining the mean squared error loss function with a frequency domain loss function to enhance supervision of the high-level feature space and constrain network training. The total difference loss can be written as:

[0064] L = L SR +λL FFT ,

[0065] In the formula, L SR ,L FFTLet represent the mean squared error loss function and the frequency Charbonnier loss function of L1, respectively. λ represents the hyperparameter used to control the frequency Charbonnier loss function, which is set to 0.1 based on past experience.

[0066] Fourth, set training parameters to train the network. Select a suitable optimizer and set parameters such as loss function, learning rate, maximum number of iterations, and batch size to train the network until the final network weight model is obtained.

[0067] Fifth, test network performance. Use low-resolution stereo image pairs as test samples and input them into the network model trained in the previous step to obtain super-resolution stereo image pairs. Use objective evaluation metrics and visual effect comparisons to verify the effect of super-resolution.

[0068] Figure 1 This is a flowchart of the method of the present invention. For binocular images, super-resolution processing is performed according to the following detailed steps:

[0069] Step S1: Establish a stereo image training set. Divide the existing publicly available stereo image super-resolution dataset into a training set and a test set. The low-resolution images are generated through bicubic downsampling. During the training phase, the generated low-resolution images are cropped into small patches, and the corresponding high-resolution images are also cropped. These patches are then randomly flipped horizontally and vertically to enhance the training data.

[0070] Step S2: Establish and train a binocular super-resolution reconstruction network model based on a multi-level enhanced attention mechanism. For example... Figure 2 As shown, the network takes a pair of low-resolution RGB stereo images as input and generates super-resolution stereo images. Specifically, the network contains two left-right weight-sharing network branches. In each weight-sharing network, a hybrid attention information extraction module is stacked to extract the intraview channels and spatial features of the left and right images. The stereo interactive view attention module is used to capture the globally corresponding information and cross-view information extracted from the left and right stereo images. In general, it can be divided into three parts: intraview feature extraction, interactive view feature fusion, and stereo image reconstruction.

[0071] Step S2.1, In-view Feature Extraction. In the feature extraction stage, the input stereo image is first... The input is fed into a 3×3 convolutional layer to extract shallow features and generate high-dimensional features. Where C represents the number of feature channels. The high-dimensional features are then input into a stacked multi-attention enhancement block for intraview feature extraction to obtain more local features and interaction information, thus restoring more accurate texture details. The multi-attention enhancement block includes a hybrid attention information extraction module and a binocular interactive view attention module.

[0072] like Figure 3 As shown, the hybrid attention information extraction module is the basic module of the left and right branches of the network. It can extract features from the view more deeply by capturing remote and local dependencies. The hybrid attention information extraction module consists of two sequentially connected modules. The first is a simplified channel and spatial information extraction module, and the second is a feedforward network module for residual information aggregation. The computation process of the two parts is as follows:

[0073] In the first module, after layer normalization, a 1×1 convolutional layer is used to expand the channels of the input feature map. The resulting output is then passed through a 3×3 depthwise convolution to capture the local context of each channel. This is followed by the use of a cross-activation structure A unit (such as...). Figure 4 (As shown) to further learn an effective representation of the spatial context. The next step is to simplify the channel spatial attention module, such as... Figure 5 As shown, by fully utilizing channel attention and spatial attention mechanisms, less useful information is filtered out, given the original input... First, average pooling and 1×1 convolution operations are used to learn the inter-channel relationships of feature maps in a given input image, realizing the functions of global spatial information aggregation and channel information interaction, and outputting simplified channel attention features. Spatial information from the feature map is aggregated using average pooling and max pooling operations. A simplified spatial attention map is then obtained by combining 3×3 convolution with the sigmoid function. Finally, the element-wise multiplication of the input feature map and the output of the sigmoid layer is used as the output of the simplified spatial attention module. The simplified channel space attention module can be represented as:

[0074]

[0075]

[0076] In the formula W C (·),H AP (·) represent 1×1 convolution and average pooling operations, respectively, H AP,1 (·),H MP,1 (·) represents average pooling and max pooling operations on the first dimension, respectively. H cat (·) indicates concatenation in a form with dimension 1, σ(·) represents the Sigmoid activation function, and Θ represents element-wise multiplication.

[0077] After simplifying the channel and spatial information extraction modules, a 1×1 convolutional inverse transform is performed on the feature mapping channels to generate adaptive feature refinement, resulting in the first module's result.

[0078] In the second module, after normalizing the output of the previous module, the result is passed through a B-unit containing a cross-activation structure (such as...). Figure 4 The residual information aggregation feedforward network (as shown) is used to improve local context awareness. Specifically, given an input tensor... First, a 1×1 convolutional layer is used to expand X′ to a higher dimension. Where k is the expansion ratio. Next, a 3×3 depthwise convolutional layer is used to encode the information of the neighboring pixel positions of X′1, and then a CAS-B unit is used as the activation function of the depthwise convolutional layer, halving the number of feature channels in the output. Finally, the initial input dimension X′2 is remapped through a 1×1 convolutional layer. The above process can be expressed as:

[0079]

[0080]

[0081] In the formula W C (·),W D These represent layer normalization, 1×1 convolution, and 3×3 depthwise convolution, respectively. CAS.B(·) represents the B unit of the cross-activation structure.

[0082] Finally, just like the previous module, the input of the next module and the output of the convolutional layer are added together to obtain the final result.

[0083] Step S2.2, cross-view feature fusion. For example... Figure 6 As shown, to capture the intersection information between the left and right views, a binocular interactive view attention module is used after the hybrid attention information extraction module for the left and right branches. The binocular interactive view attention module uses the binocular features generated in the previous step as input to perform bidirectional cross-view interaction and generate interactive features fused with the input features of that view. Specifically, given the input binocular view features... After layer normalization and 1×1 convolution operation, the binocular features are obtained. and in It's a 1×1 convolution. Then, channel weights are generated by performing a fast 1D convolution of size k. Here, k is adaptively determined through the mapping of the channel dimension C, and the aggregated features are obtained by multiplying the channel weights element-wise with the binocular features. As shown below:

[0084]

[0085]

[0086] In the formula, Θ represents a k×k convolutional layer and a max pooling operation, respectively, and Θ represents element-wise multiplication.

[0087] By calculating the attention matrix once, F is generated simultaneously. R→L ,F L→R Finally, the interactive cross-view information and the internal view information FL and FR are merged by adding them element by element, according to... In the formula It is a query matrix that projects features within the source view (such as the left view). It is the key-value matrix of feature projections within features of the target view (such as the right view). This can be represented as follows:

[0088]

[0089]

[0090] In the formula, γ L and γ R The trainable channels are scaled and initialized to zero for stable training. It is a 1×1 convolution.

[0091] Step S2.3, Binocular Image Reconstruction. After feature extraction, the output is fed into a 3×3 convolutional layer and a spatial attention enhancement module. Finally, a pixel recombination operation is used to upsample the output features to a high-resolution size. Furthermore, to reduce the burden of feature extraction, a global residual path is used in this part to further improve the super-resolution performance by utilizing the input binocular image information, thus recovering the super-resolution images of the left and right views.

[0092]

[0093] In the formula H C (·),H E (·),H P (·),H ↑ (·) represent the upsampling operations for convolution, enhanced spatial attention module, pixel recombining, and bilinear interpolation, respectively.

[0094] The enhanced spatial attention module will provide the given input Sending to a 1×1 convolutional layer yields Among them W CThe first convolution (·) is a 1×1 convolution to reduce the channel size of the input features. Then, the block uses cross-row convolutions and cross-row max-pooling layers to further reduce the spatial size. After a set of convolutions, upsampling based on bilinear interpolation is performed to restore the spatial size in order to extract features. Combined with residual connectivity, the features are further processed to obtain a 1×1 convolutional layer to restore the channel size. Finally, an attention matrix is ​​generated by the sigmoid function and multiplied by the original input features X″.

[0095] Step S3: Construct the loss function. To enhance the texture details of the binocular images and maintain disparity consistency between viewpoints, this invention employs an L1 loss function combined with a frequency domain loss function to enhance supervision of the high-level feature space and constrain network training. The total difference loss can be written as:

[0096] L = L SR +λL FFT (10)

[0097] In the formula, L SR ,L FFT Let represent the L1 reconstruction loss function and the frequency domain loss of the Charbonnier loss, respectively, and λ represent the hyperparameter used to control the frequency Charbonnier loss function. The parameter λ is set to 0.1 in all experiments.

[0098] SR reconstruction loss. The SR reconstruction loss is essentially an L1 loss function. To achieve faster convergence, this invention uses the pixel-level L1 distance between the super-resolution and the ground truth binocular image, avoiding overly smooth textures and thus obtaining a higher PSNR. It can be represented as follows:

[0099]

[0100] In the formula, These are the left and right super-resolution images generated by the model, respectively. These are their respective high-resolution images.

[0101] Frequency domain loss. To better recover high-frequency details in image super-resolution tasks, this invention introduces frequency domain loss using Charbonnier loss. It can be represented as follows:

[0102]

[0103] In the formula, the constant ε is experimentally set to 10. -3 FFT(·) represents Fast Fourier Transform.

[0104] Step S4: Set training parameters for network training. AdamW is used for optimization, where β1 = 0.9, β2 = 0.9, and the weights are set to 0 by default. The initial learning rate is set to 1 × 10⁻⁶. -3 And reduced to 1×10 using a cosine annealing strategy. -7 The model is trained on 30×90 patches. During training, each batch of 32 samples is evenly distributed across 8 parts, with 2×10 iterations. 5 Second-rate.

[0105] Step S5: Test network performance. Use low-resolution stereo image pairs as test samples and input them into the network model trained in the previous step to obtain super-resolution stereo image pairs. Use objective evaluation metrics and visual effect comparisons to verify the effectiveness of the super-resolution.

[0106] To demonstrate the effectiveness of super-resolution, the experiment compared the bicubic interpolation method, existing single-image super-resolution techniques (EDSR), and binocular image super-resolution techniques (PASSRnet, SRResNet+SAM, iPASSR, and NAFSR-L).

[0107] The binocular super-resolution imaging method proposed in this invention is verified from both qualitative and quantitative perspectives.

[0108] 2.1 Qualitative Experimental Results

[0109] In this embodiment of the invention, super-resolution operations were performed on images in the test set, and the results were compared with those obtained by other methods. For example... Figure 7 The image shown is a super-resolution result. Figure 7 In the image, the box in the lower right corner shows the complete image, while the rest is a magnified view of a specific area within that image. As can be seen, compared to other super-resolution methods, the method proposed in this embodiment recovers more details related to edges and textures, verifying the good performance of this embodiment in binocular image super-resolution tasks.

[0110] 2.2 Quantitative Analysis

[0111] This invention presents a quantitative error analysis of super-resolution results for 112 pairs of binocular images in a test set. The methods compared include bicubic interpolation, EDSR, PASSRnet, SRResNet+SAM, iPASSR, and NAFSR-L. Objective quality assessment refers to the quantitative calculation of a target image using a fixed mathematical formula, and evaluating the image quality based on the calculated value. Currently, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) are the main objective evaluation metrics. The PSNR calculation formulas for image I and image K are as follows:

[0112]

[0113]

[0114] Where H and W represent the height and width of images I and K, respectively, is the pixel value at coordinates x and y in image I, is the peak pixel value, and b is the number of bits in the pixel's binary representation, typically set to 8 in natural image processing. PSNR is measured in decibels (dB), and its value is usually between 20 and 40. A higher PSNR value indicates a smaller pixel difference between the reconstructed image and the labeled image, thus demonstrating better performance of the super-resolution model.

[0115] Given a labeled image I and a reconstructed image K, the steps for calculating SSIM are as follows:

[0116]

[0117] μ I ,μ K The pixel mean values ​​of I and K are respectively, σ I ,σ K The variances of I and K are σ, respectively. IK Let SSIM be the covariance of I and K. The value of SSIM ranges from 0 to 1, with a value closer to 1 indicating a higher overall similarity between the two images. SSIM is often used in conjunction with PSNR as an objective quality assessment metric.

[0118] The experimental results, obtained by testing the images in the test set using different methods and averaging the results, are shown in Table 1.

[0119] Table 1. Comparison of PSNR and SSIM of different super-resolution methods on the test set

[0120]

[0121] As can be seen from the results in Table 1, the binocular image super-resolution method proposed in this embodiment of the invention achieved an average peak signal-to-noise ratio of 24.21 dB and a structural similarity of 0.7633. Compared with other super-resolution methods that utilize neural networks, these values ​​indicate that the super-resolution method proposed in this embodiment of the invention performs better on the test set, and improves the super-resolution effect by utilizing the mapping relationship between binocular images.

[0122] Figure 7 The reconstruction results of this invention are compared with those of the bicubic interpolation method, existing single-image super-resolution techniques (EDSR), and binocular image super-resolution techniques (PASSRnet, SRResNet+SAM, iPASSR, and NAFSR-L). This invention can clearly display the edge lines and clearly distinguish the background from the main texture, resulting in good super-resolution results.

Claims

1. A binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism, characterized in that, Includes the following steps: Step S1: Establish a stereo image training set; divide the stereo image super-resolution dataset into a training set and a test set. The low-resolution images are generated by bicubic downsampling. During the training phase, the generated low-resolution images are cropped into small patches, and the corresponding high-resolution images are also cropped. At the same time, these patches are randomly flipped horizontally and vertically to augment the training data. Step S2: Establish and train a binocular super-resolution reconstruction network model based on a multi-level enhanced attention mechanism; the network takes a pair of low-resolution RGB binocular images as input and generates super-resolution binocular images. Step S3: Construct the loss function; use the L1 loss function combined with the frequency domain loss function to enhance the supervision of the high-level feature space and constrain the training of the network. Step S4: Set training parameters to train the network; Step S5: Test network performance; use low-resolution stereo image pairs as test samples and input them into the network model trained in the previous step to obtain super-resolution stereo image pairs. Use objective evaluation metrics and visual effect comparison to verify the effect of stereo image super-resolution. Specifically, the binocular super-resolution reconstruction network model based on the multi-level enhanced attention mechanism contains two left and right weight-sharing network branches; In each weight-sharing network, the hybrid attention information extraction module is stacked to extract the intraview channels and spatial features of the left and right images; the binocular interactive view attention module is used to capture the globally corresponding information and cross-view information extracted from the left and right binocular images; it is specifically divided into three parts: intraview feature extraction, interactive view feature fusion and binocular image reconstruction. Step S2 specifically includes the following steps: Step S21, In-view Feature Extraction; In the feature extraction stage, the input binocular image is first... The input is fed into a 3×3 convolutional layer to extract shallow features and generate high-dimensional features. Where C is the number of feature channels; then the high-dimensional features are input into the stacked multi-attention enhancement blocks for intraview feature extraction to obtain more local features and interaction information, and restore more accurate texture details; the multi-attention enhancement blocks include a hybrid attention information extraction module and a binocular interactive view attention module; The hybrid attention information extraction module is the basic module of the left and right branches of the network. It extracts features from the view more deeply by capturing remote and local dependencies. The hybrid attention information extraction module consists of two sequentially connected modules: the first is a simplified channel and spatial information extraction module, and the second is a feedforward network module for residual information aggregation. The computation process of the two parts is as follows: In the first module, after layer normalization, a 1×1 convolutional layer is used to expand the channels of the input feature map. The output is then passed through a 3×3 depthwise convolution to capture the local context of each channel; a cross-activation structure A-unit is then used to further learn an effective representation of the spatial context; the next step is to simplify the channel spatial attention module, fully utilizing both channel and spatial attention mechanisms, given the original input. First, average pooling and 1×1 convolution operations are used to learn the inter-channel relationships of feature maps for a given input image, realizing the functions of global spatial information aggregation and channel information interaction, and outputting simplified channel attention features. The spatial information of the feature map is aggregated through average pooling and max pooling operations, and then a simplified spatial attention map is obtained by combining 3×3 convolution and the sigmoid function. Finally, the element-wise multiplication of the input feature map and the output of the sigmoid layer is used as the output of the simplified spatial attention module. The simplified channel space attention module is represented as follows: （1） （2） In the formula These are 1×1 convolution and average pooling operations, respectively. These represent average pooling and max pooling operations performed on the first dimension, respectively. This indicates that the components are joined in a format with a dimension of 1. These represent the Sigmoid activation function, This indicates element-wise multiplication; After simplifying the channel and spatial information extraction modules, the feature mapping channels are processed in a 1×1 manner. Inverse convolutional transform is used to produce adaptive feature refinement, resulting in the first module's output. In the second module, after normalizing the output of the previous module, a residual information aggregation feedforward network containing B-units with cross-activation structures is used to improve local context awareness; specifically, given an input tensor... First, a 1×1 convolutional layer is used to... Expanding to higher dimensions Where k is the expansion ratio; next, a 3×3 depthwise convolutional layer is used to... Information about adjacent pixel positions is encoded, and then a CAS-B unit is used as the activation function of the deep convolutional layer, halving the number of feature channels in the output. Finally, the initial input dimension is remapped through a 1×1 convolutional layer. ; The above process can be represented as: （3） （4） In the formula These are layer normalization, 1×1 convolution, and 3×3 depthwise convolution, respectively. The B unit represents the cross-activation structure; Finally, just like the previous module, the input of the next module and the output of the convolutional layer are added together to obtain the final result. Step S22, cross-view feature fusion: After the hybrid attention information extraction module for the left and right branches, the binocular interactive view attention module is used; the binocular interactive view attention module uses the binocular features generated in the previous step as input to perform bidirectional cross-view interaction and generate interactive features fused with the input features of that view; specifically, given the input binocular view features... After layer normalization and 1×1 convolution operation, the binocular features are obtained. and ,in It's a 1×1 convolution; then, channel weights are generated by performing a fast 1D convolution of size k. Where k is adaptively determined through the mapping of the channel dimension C, and the aggregated features are obtained by multiplying the channel weights element-wise with the stereo features. As shown below: （5） （6） In the formula, These represent a k×k convolutional layer and a max pooling operation, respectively. This indicates element-wise multiplication; By calculating the attention matrix once, simultaneously generating Finally, the interactive cross-view information and the internal view information FL and FR are merged by adding them element by element, according to... In the formula It is a query matrix that projects features within the source view (such as the left view). It is a key-value matrix representing the projection of features within the target view (such as the right view); represented as follows: （7） （8） In the formula, and The trainable channels are scaled and initialized to zero for stable training. It is a 1×1 convolution; Step S23, Binocular Image Reconstruction: After feature extraction, the output is fed to a 3×3 convolutional layer and a spatial attention enhancement module. Finally, a pixel recombination operation is used to upsample the output features to a high-resolution size, and a global residual path is used to further improve the super-resolution performance by utilizing the input binocular image information, thus recovering the super-resolution images of the left and right views. ; （9） In the formula These are upsampling operations for convolution, enhanced spatial attention module, pixel recombination, and bilinear interpolation, respectively. The enhanced spatial attention module will provide the given input Sending to a 1×1 convolutional layer yields ,in First, 1×1 convolutions are used to reduce the channel size of the input features. Then, the blocks use cross-row convolutions and cross-row max pooling layers to further reduce the spatial size. After a set of convolutions, upsampling based on bilinear interpolation is performed to restore the spatial size in order to extract features. Combined with residual connectivity, the features are further processed to obtain 1×1 convolutional layers to restore the channel size. Finally, an attention matrix is ​​generated by the sigmoid function and multiplied by the original input features. .

2. The binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism according to claim 1, characterized in that, In step S3, the loss of the total difference is written as: （10） In the formula, These represent the L1 reconstruction loss function and the frequency domain loss of the Charbonnier loss, respectively. Represents the hyperparameters used to control the frequency of the Charbonnier loss function; parameters from all experiments. Set all to 0.1; SR reconstruction loss; the SR reconstruction loss is essentially an L1 loss function; it uses the pixel-level L1 distance between the super-resolution and the ground truth stereo image to obtain the PSNR; expressed as follows: （11） In the formula, These are the left and right super-resolution images generated by the model, respectively. These are their high-resolution images; Frequency domain loss; introducing frequency Charbonnier loss; expressed as follows: （12） In the formula, the constant is experimentally set. Set to 10 -3 ; This represents the Fast Fourier Transform.

3. The binocular image super-resolution reconstruction method based on a multi-level enhanced attention mechanism according to claim 1, characterized in that, In step S4, AdamW is used for optimization, where β1 = 0.9, β2 = 0.9, and the weights are defaulted to 0; the initial learning rate is set to 1×10. -3 And reduced to 1×10 using a cosine annealing strategy. -7 ; The model is trained on 30×90 patches; during training, 32 samples per batch are evenly distributed across 8 parts, with 2×10 iterations. 5 Second-rate.

Images