Three-dimensional image super-resolution reconstruction method based on reliable attention

Abstract

The invention provides a three-dimensional image super-resolution reconstruction method based on reliable attention. Unreliable features are screened through uncertainty estimation and modulated to improve the reconstruction quality of a low-resolution image. Depth features are extracted through a Monte Carlo feature extraction block MCFEBlock, and uncertainty estimation is carried out; uncertainty estimation is provided for feature screening through an MCdropout method; and unreliable features are filtered and modulated through a feature modulation module FMM.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing, image super-resolution reconstruction, etc., and in particular relates to a stereo image super-resolution reconstruction method based on reliable attention, which can be applied to scenes such as industrial detection, medical imaging, flood inundation image recognition and traffic road collapse scene recognition. Background Art

[0002] Different from single image super-resolution reconstruction, stereo image super-resolution reconstruction aims to restore a pair of low-resolution images into a high-resolution image by using complementary information from another perspective. There are many applications in real-world scenarios, such as industrial inspection, where accurate stereo images can more accurately measure manufactured parts to ensure excellent quality control; and in medical imaging, clearer and more detailed images help accurately measure anatomical structures. However, how to make full use of complementary information from another perspective is a challenge.

[0003] To address this problem, some existing technologies have proposed methods that combine attention mechanisms. For example, Wang et al. introduced a parallax attention mechanism (PAM) with a global receptive field along the epipolar lines, which can handle different stereo images with large parallax changes. Subsequently, Ying et al. proposed a general stereo attention module (SAM), which can be integrated into any single image super-resolution reconstruction model to produce clearer stereo images. Song et al. proposed a self-parallax attention mechanism (SPAM) that can simultaneously aggregate information from the original image and another image. Zhu et al. proposed a cross-view network (CVCnet) that can capture cross-view information from the global view and epipolar lines. Lin et al. proposed an efficient Transformer-based stereo image super-resolution network (Stereformer), which cleverly combined the advantages of convolution and Transformer and achieved remarkable performance. Chu et al. developed a NAFSSR network that uses nonlinear activation-free blocks for intra-view feature extraction and a stereo cross-attention module (SCAM) for cross-view feature interaction. Gao et al. proposed a hybrid scale selective fusion network (MSSFNet) that can adaptively select and fuse the most accurate features from two perspectives.

[0004] However, not all features from another view are reliable or useful. Some features are adversely affected by degradation and distortion during reconstruction, making disparity attention unreliable and adversely affecting the performance of image reconstruction.

[0005] Single image super-resolution reconstruction aims to reconstruct a low-resolution image into a high-resolution image. However, this method can only utilize information within the view, and does not significantly improve the super-resolution reconstruction of stereo images. To solve this problem, researchers have proposed a method for super-resolution reconstruction of stereo images, which enhances the performance of image restoration by utilizing complementary information from another perspective. However, learning all information from another perspective is very time-consuming, and not all information is useful. Subsequently, researchers introduced an attention mechanism in the interaction stage of left and right view features to reduce the learning of unnecessary information, which greatly improved the performance of stereo image super-resolution. However, whether the weights assigned by the attention mechanism are absolutely reliable, and whether the attention will be focused on some unreliable or useless features due to factors such as feature degradation and distortion, this is a question worth pondering. As shown in the attached manual Figure 1 As shown in Figure 2, by visualizing the difference between a left-view high-resolution image and the ground-truth image in NAFSSR and the uncertainty during training, we can see that these uncertainties and differences are very close, which indicates that the features recovered in these high-uncertainty regions are not reliable. In the feature interaction stage, disparity attention may focus on unreliable features, which will have a negative impact on the performance of image reconstruction. Summary of the invention

[0006] In view of the above problems existing in the prior art, the present invention proposes a reliable attention-based stereo image super-resolution network (RASSR). Firstly, two modules are proposed to achieve feature quality assessment while maintaining feature extraction capability: Monte Carlo Feature Extraction Block (MCFEBlock) and Feature Compensation Module (FCM). MCFEBlock is responsible for extracting inter-view features and providing uncertainty estimation for subsequent feature screening through the MCdropout method. The FCM module enriches image features by compensating for the information loss caused by dropout through a series of convolution operations including the large kernel convolution attention mechanism (LKCA). In addition, a feature modulation module (FMM) is introduced to filter out unreliable features and modulate them to ensure that their attention and learned features are relatively reliable.

[0007] The present invention specifically adopts the following technical solutions:

[0008] A stereo image super-resolution reconstruction method based on reliable attention;

[0009] Unreliable features are filtered out and modulated through uncertainty estimation to improve the reconstruction quality of low-resolution images;

[0010] Extract deep features and uncertainty estimation through the Monte Carlo feature extraction block MCFEBlock;

[0011] Providing uncertainty estimation for feature screening through MCdropout method;

[0012] Unreliable features are filtered and modulated through the feature modulation module FMM.

[0013] Furthermore, the Monte Carlo feature extraction block MCFEBlock includes a deep information extractor DIE and a feature compensation module FCM; the feature compensation module FCM is used to compensate for feature loss caused by neuron loss during MCdropout training.

[0014] Furthermore, a stereo image super-resolution network based on reliable attention is used for super-resolution reconstruction, including four parts: initial feature extraction, deep feature extraction, feature interaction and image reconstruction; the left and right branches of the network adopt a weight sharing strategy and the feature extraction process is the same:

[0015] First, a 3×3 convolutional layer is used to extract the low-resolution images from the left and right images. Extract the initial features Where H×W represents the spatial dimension and C represents the number of channels, which can be expressed as:

[0016]

[0017] in, and are left and right low-resolution images, and is the initial feature extracted from the left and right low resolutions, C 3x3 It is a 3×3 convolution operation;

[0018] Then, N MCFEBlocks and M NAFBlocks are used for deep feature extraction; each MCFEBlock and NAFBlock is followed by FMM and SCAM, respectively, as follows:

[0019]

[0020]

[0021]

[0022] in, and represents the i-th MCFEBlock and FMM, and represents the j-th NAFBlock and SCAM, i∈(1,N), j∈(1,M), Represents the output after passing through i MCFEBlocks, It represents the output after passing through j NAFBlocks. represents the sum of the output after passing through i+j blocks;

[0023] Finally, a 3×3 convolutional layer and a shuffle layer are used to upsample the features by a scale factor s. Global residual learning is used to predict only the residual between the bilinearly upsampled low-resolution image and the true image. It is expressed as:

[0024]

[0025] Among them, U ↑ represents bilinear interpolation, S ↑ Indicates the shuffle layer; Represents the left and right high-resolution images.

[0026] Furthermore, the depth information extractor DIE is expressed as:

[0027] Pre0=D ro (FFN(D ro (MVConv(F in )))

[0028] Pre1=D ro (FFN(D ro (MBConv(F in )))

[0029] Among them, F in represents the input features; Pre0 and Pre1 represent two different feature outputs; MBConv and FFN refer to the mobile convolutional block and feedforward network used in NAFBlock, respectively; D ro represents a randomly dropped neuron layer;

[0030] During the training phase, a dropout layer is added after each MBConv layer and the feed-forward network FFN respectively; different output features are obtained by cycling DIE twice; the two feature outputs are then concatenated and their pixel-by-pixel standard deviation is calculated as the approximate uncertainty estimate, and the uncertainty is converted into a mask to filter the features.

[0031] Furthermore, the specific method of uncertainty estimation is:

[0032] Based on Monte Carlo inference, the model uncertainty is estimated by using random sampling and repeatedly running the model; during the prediction process, multiple forward passes are made on the input samples, and the uncertainty is estimated based on the mean or variance of these predictions; expressed as:

[0033]

[0034] Among them, Unvertainty represents the pixel-by-pixel standard deviation; Represents a join operation; Mran and Std represent mean and standard deviation operations.

[0035] Furthermore, the method of converting uncertainty into a mask is to adaptively use the mean + standard deviation of each uncertainty estimate as a threshold during training, and compare each pixel value in the uncertainty with the threshold for assignment, which is expressed as follows:

[0036]

[0037] Among them, M represents the mask value, un represents the pixel value of uncertainty; τ is the threshold adaptively obtained during the training process.

[0038] Furthermore, the feature compensation module FCM has two identical branches, which enrich the inter-view features through the multi-convolution module MCM and add the learned features to the opposing features;

[0039] It is expressed as:

[0040]

[0041]

[0042]

[0043] LKCA=C 5×5 (C 7×7 (C 1×1 (SG(*))

[0044] in, is the final compensated feature, and is the feature after interaction, A vg For average operation;

[0045] and are the left and right input features of the first and second predictions, C n×n It is an n×n convolution operation, SG is the SimpleGate function, and LKCA is the large kernel convolution attention mechanism;

[0046] The SimpleGate function splits the input into two features X1,X2∈R along the channel dimension H×W×C / 2 ,

[0047] The output is then calculated through linear gates:

[0048] SG=X1⊙X2

[0049] where ⊙ is the product of the elements.

[0050] Furthermore, after endowing the feature extraction module with quality assessment capability, the feature modulation module FMM uses the uncertainty mask obtained from MCFEBlock to screen the features to be modulated:

[0051] In the modulation process of the left view, for features with lower uncertainty, they are directly output; for features with higher uncertainty, the features are modulated through N cyclic interaction modules LIM, and then the weighted features are output, as shown below:

[0052]

[0053]

[0054]

[0055] in, is the output characteristic after modulation, is a low uncertainty feature, is a high uncertainty feature, M L,R is the high uncertainty mask value;

[0056] Then, the high uncertainty features of the left view and the low uncertainty features of the right view are put into the attention interaction module SCAM for learning, and the reliable features of the right view that are beneficial to the left view are obtained. The left view gradually shifts its attention to the reliable features; in addition, the features in the left view are enriched by channel attention; finally, a 3×3 convolution block is added; the attention of the right view is also improved through the same mechanism; the process is as follows:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] Among them, LIM N Indicates that the LIM module cycles N times, C 3×3 is a 3×3 convolution operation, is the output after each LIM module modulation, is the feature of the left view that is beneficial to the right view, is the feature that the right view is beneficial to the left view, and CA is the channel attention mechanism.

[0063] Furthermore, the attention interaction module SCAM is used to fuse the features of the left and right views to capture similar information; Scaled DotProduct Attention is adopted, which involves calculating the dot product between the query and the key, and then applying the softmax function to generate the weights assigned to the corresponding values:

[0064]

[0065] in, is the query matrix projected by the source cross-view features, is the key and value matrix projected by the target cross-view features:

[0066]

[0067]

[0068] F L =γ L F R→L +X L

[0069] F R =γ R F L→R +X R

[0070] in, and is the projection matrix, γ L and γ R is a learnable scaling parameter, and ATT is Scaled DotProductAttention.

[0071] Furthermore, the loss function adopted by the stereoscopic image super-resolution network is as follows:

[0072]

[0073]

[0074] in, and represents the left and right real images, and represents the left and right high-resolution images, and λ is the weight;

[0075] For the frequency domain loss, the frequency Charbonnier loss function is used as follows:

[0076]

[0077] in, stands for Fast Fourier Transform.

[0078] Since the method and model of the present invention are generally implemented through a computer system in the form of a computer program, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the method as described above is adopted.

[0079] Compared with the prior art, the present invention and its preferred embodiment introduce the concept of uncertainty, and use uncertainty estimation to screen unreliable features and modulate them to improve the reconstruction quality of low-resolution images. Specifically, a reliable attention-based stereo image super-resolution network (RASSR) is proposed. Three modules are mainly designed, namely the Monte Carlo feature extraction module (MCFEBlock), the feature compensation module (FCM) and the feature modulation module (FMM). MCFEBlock is used to extract depth features and uncertainty estimation, FCM is used to compensate for feature loss during MCdropout training, and in the feature interaction process, FMM selects high uncertainty features and modulates them to make their disparity attention relatively reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0081] Figure 1 Visualize the differences and uncertainties.

[0082] Figure 2 This is a diagram of the algorithm network model architecture of an embodiment of the present invention.

[0083] Figure 3 It is a framework diagram of the Monte Carlo feature extraction block (MCFEBlock) of an embodiment of the present invention.

[0084] Figure 4 This is a visualization diagram of the uncertainty and its mask according to an embodiment of the present invention.

[0085] Figure 5 FIG. 4 is a diagram of a feature compensation module architecture (FCM) according to an embodiment of the present invention.

[0086] Figure 6 This is a characteristic modulation module architecture diagram (FMM) of an embodiment of the present invention.

[0087] Figure 7 This is the architecture diagram of the attention interaction module (SCAM) of an embodiment of the present invention.

[0088] Figure 8A visual comparison diagram of high-resolution images (×4) reconstructed by different methods according to embodiments of the present invention.

[0089] Fig. 9 A visual comparison diagram of high-resolution images (×2) reconstructed by different methods according to embodiments of the present invention.

[0090] Fig.10 This is a diagram showing the effects of the embodiments of the present invention. DETAILED DESCRIPTION

[0091] Hereinafter, the specific embodiments of the present application will be described in detail with reference to the accompanying drawings. According to these detailed descriptions, those skilled in the art can clearly understand the present application and can implement the present application. Without violating the principles of the present application, the features in the various embodiments can be combined to obtain new implementations, or certain features in certain embodiments can be replaced to obtain other preferred implementations.

[0092] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail with the accompanying drawings as follows:

[0093] 1 Network Architecture

[0094] like Figure 2 As shown, the reliable attention-based stereo image super-resolution network (RASSR) provided by the embodiment of the present invention is divided into four main parts: initial feature extraction, deep feature extraction, feature interaction and image reconstruction. In order to reduce the computational complexity, the left and right branches of the network adopt a weight sharing strategy. Since the feature extraction process on both sides is the same, the same formula will be used to describe them below:

[0095] First, RASSR is obtained from the left and right low-resolution images through a 3×3 convolutional layer. Extract the initial features Where H×W represents the spatial dimension and C represents the number of channels. The formula can be expressed as:

[0096]

[0097] in, and are left and right low-resolution images, and is the initial feature extracted from the left and right low resolutions, C 3x3 It is a 3×3 convolution operation.

[0098] Then, N MCFEBlocks and M NAFBlocks are used for deep feature extraction. Each MCFEBlock and NAFBlock is followed by FMM and SCAM respectively (the specific structures of NAFBlocks and SCAM can be found in X.Chu, L.Chen, and W.Yu, “Nafssr: stereo image super-resolution using nafnet,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp.1239–1248). The FMM module will be described in detail in Section 5 below, and the whole process can be described as:

[0099]

[0100]

[0101]

[0102] in, and represents the i-th MCFEBlock and FMM, and represents the j-th NAFBlock and SCAM, i∈(1,N), j∈(1,M), Represents the output after passing through i MCFEBlocks, It represents the output after passing through j NAFBlocks. Represents the sum of the outputs after passing through (i+j) blocks.

[0103] Finally, a 3×3 convolutional layer and a shuffle layer are used to upsample the features with a scale factor of s. In order to reduce the burden of feature learning, the solution of the present invention also adopts global residual learning, which only predicts the residual between the bilinearly upsampled low-resolution image and the real image. It can be expressed as:

[0104]

[0105] Among them, U ↑ represents bilinear interpolation, S ↑ Indicates the shuffle layer. Represents the left and right high-resolution images.

[0106] 2 Monte Carlo feature block (MCFEBlock)

[0107] Many existing stereo image super-resolution reconstruction methods focus on designing more complex feature interaction modules to fuse similar information between images. However, the features learned in the feature interaction stage often contain uncertainty, which may cause the reconstruction process to focus on unreliable or useless information, affecting the performance and efficiency of image reconstruction. In order to clearly evaluate feature quality, the present invention provides MCFEBlock to extract in-view information and obtain uncertainty estimates for subsequent feature interactions.

[0108] like Figure 3 As shown, MCFEBlock is divided into two parts: (1) Deep Information Extractor (DIE); (2) Feature Compensation Module (FCM); Specifically, during the training phase, a dropout layer is added after each MBConv layer and feedforward network (FFN), respectively. Considering the uncertainty acquisition and feature interaction in the subsequent FCM module, different output features are obtained by looping DIE twice. Since the neurons discarded by the dropout layer are random each time, the two predictions are different. Therefore, the two feature outputs are concatenated and their pixel-by-pixel standard deviation is calculated as an approximate uncertainty estimate. In addition, the uncertainty is converted into a mask to filter the features. FCM will be described in detail in Section 4 below. The formula of DIE can be expressed as:

[0109] Pre0=D ro (FFN(D ro (MBConv(F in )))

[0110] Pre1=D ro (FFN(D ro (MBConv(F in )))

[0111] Among them, F in Denotes the input features. Pre0 and Pre1 denote two different feature outputs. MBConv and FFN refer to the mobile convolutional block and feedforward network used in NAFBlock, respectively. ro represents a layer of randomly dropped neurons.

[0112] 3 Uncertainty estimation

[0113] Uncertainty estimation is a key issue in machine learning and deep learning, involving the quantification and estimation of uncertainty in model predictions. The prior art proposes various methods for estimating uncertainty, including ensemble methods, Monte Carlo reasoning, and Bayesian reasoning. The method adopted in the present invention is Monte Carlo reasoning, which estimates model uncertainty by using random sampling and repeatedly running the model. During the prediction process, multiple forward passes are made on the input samples, and uncertainty is estimated based on the mean or variance of these predictions. The specific formula can be expressed as:

[0114]

[0115] Among them, Uncertainty represents the pixel-by-pixel standard deviation. Indicates a join operation. Mean and Std indicate mean and standard deviation operations.

[0116] To obtain the uncertainty mask, this embodiment adaptively uses the mean + standard deviation of each uncertainty estimate as a threshold during training. Figure 4 As shown in Figure 2, the uncertainty estimate and its mask are visualized. Each pixel value in the uncertainty is compared with the threshold. For components exceeding the threshold, their uncertainty is assumed to be high and assigned a value of 1. On the contrary, for components below the threshold, their uncertainty is set to 0. The corresponding formula can be expressed as follows:

[0117]

[0118] Where M represents the mask value, un represents the pixel value of uncertainty, and τ is the threshold adaptively obtained during the training process.

[0119] 4Feature Compensation Module (FCM)

[0120] The embodiment of the present invention uses the FCM module to solve the feature loss caused by neuron loss during MCdropout training. Figure 5 As shown, the FCM module has two input branches. The inter-view features are enriched through the multi-convolution module (MCM) and the learned features are added to the adversarial features. LKCA is a large kernel convolution attention mechanism used to capture long-range dependencies within the image and enhance the attention to local information to achieve compensation and generalization. The entire process of FCM can be expressed as:

[0121]

[0122] in, is the final compensated characteristic, and is the feature after interaction, A vg For average operation.

[0123] Subsequently, a multi-convolution module (MCM) is used to compensate for the discarded features and interact between predictions, adding the learned features to another feature to achieve feature enrichment and generalization. The specific formula can be expressed as:

[0124]

[0125]

[0126] LKCA=C 5×5 (C 7×7 (C 1×1 (SG(*))

[0127] in, and are the left and right input features of the first and second predictions, C n×n is an n×n convolution operation, and SG is a SimpleGate function. Specifically, SimpleGate splits the input into two features X1,X2∈R along the channel dimension H×W×C / 2 .

[0128] It then computes the output via linear gates:

[0129] SG=X1⊙X2

[0130] Among them, ⊙ is the product of elements, and LKCA is the large kernel convolution attention mechanism.

[0131] 5Feature Compensation Module (FMM)

[0132] After giving the feature extraction module the ability to evaluate quality, the present invention uses the uncertainty mask obtained from MCFEBlock to screen the features to be modulated. Figure 6 As shown, taking the modulation of the left view as an example, for features with lower uncertainty, they are directly output; for features with higher uncertainty, the features are modulated through N loop interaction modules (LIMs), and then the weighted features are output, as shown below:

[0133]

[0134]

[0135]

[0136] in, is the output characteristic after modulation, is a low uncertainty feature, is a high uncertainty feature, M L,R is the high uncertainty mask value.

[0137] Then, the high uncertainty features of the left image and the low uncertainty features of the right image are put into SCAM for learning, and reliable features of the right image that are beneficial to the left image are obtained. The left image gradually shifts its attention to the reliable features. In addition, channel attention is used to enrich the features in the left view. Finally, a 3×3 convolution block is added. Similarly, the attention of the right image is also improved by this method. The whole process is as follows:

[0138]

[0139]

[0140]

[0141]

[0142]

[0143] Among them, LIM N Indicates that the LIM module cycles N times, C 3×3 is a 3×3 convolution operation, is the output after each LIM module modulation, The features of the left image that are beneficial to the right image, The features of the right image that are beneficial to the left image, CA is the channel attention mechanism.

[0144] like Figure 7 As shown in Figure 1, in order to meet the requirements of stereo image super-resolution reconstruction, SCAM in NAFSSR is used to fuse the features of the left and right views to capture similar information. This method adopts Scaled DotProduct Attention (from A.Vaswani, N.Shazeer, N.Parmar, J.Uszkoreit, L.Jones, ANGomez, Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.), involves computing the dot product between the query and the key, and then applying a softmax function to generate the weights assigned to the corresponding values:

[0145]

[0146] in, is the query matrix projected by the source cross-view features, and is the key and value matrix projected by the target cross-view features:

[0147]

[0148]

[0149] F L =γ L F R→L +X L

[0150] FR =γ R F L→R +X R

[0151] in, and is the projection matrix, γ L and γ R is a learnable scaling parameter, and ATT is Scaled DotProductAttention.

[0152] 6 Loss Function

[0153] Regarding the loss function, in the image super-resolution task, the main focus is on restoring high-frequency details. In order to effectively restore clear and sharp high-frequency textures, this embodiment uses the network guided by spatial and frequency domain losses. The specific loss function is as follows:

[0154]

[0155]

[0156] in, and represents the left and right real images, and represents the left and right high-resolution images, and λ is a weight that is empirically set to 1.0.

[0157] For the frequency domain loss, the frequency Charbonnier loss function is used as follows:

[0158]

[0159] in, represents the fast Fourier transform. By using It can make the network pay more attention to high-frequency information.

[0160] In order to evaluate the effectiveness of the scheme of the present invention, this embodiment uses 800 stereo images from Flickr1024 as a training set and 112 stereo images as a validation set. In addition, the method of the present invention is tested on four data sets, including: KITTI 2012, KITTI 2015, Middlebury and Flickr1024, to prove the effectiveness of the experiment. The network (RASSR) proposed in the present invention is compared with other methods, including various existing single image super-resolution reconstruction methods (i.e., VDSR, EDSR, RDN and RCAN) and stereo image super-resolution reconstruction methods (i.e., IMSSRnet, iPASSR, SSRDE-FNet, SRRes+SAM, Steformer, CANSSR-S and NAFSSR). The above methods are all trained on the same data, and the experimental results of the prior art solutions are extracted from authoritative papers. In addition, the PSNR and SSIM indicators are used for quantitative evaluation to evaluate the experimental results.

[0161] 1 Quantitative comparison

[0162] Table 1 Quantitative results of different methods on the dataset (×4 and ×2).

[0163]

[0164]

[0165] The proposed method, based on the Reliable Attention Stereo Super-Resolution Network (RASSR), is compared with existing stereo super-resolution methods at ×2 and ×4 sampling factors, respectively. As shown in Table 1, the bold font is the best PSNR result, the underlined font is the second best PSNR result, and the method of is superior to most existing methods. Specifically, on the dataset Flickr1024 (×4), it can be seen that the average PSNR result of the two views is higher than most other methods in the table. Compared with the pioneer network (StereoSR) and the early disparity mechanism network (PASSRnet), the PSNR of is improved by 2.29dB and 0.55dB, respectively, far exceeding them. Compared with the baseline NASSR-S, RASSR is improved by 0.11dB, which proves the effectiveness of the proposed method. In addition, compared with recent methods, the proposed RASSR is 0.41dB better than Steformer. It can be seen that although the MCFEBlock, FCM and FMM modules proposed in the present invention are relatively simple in structure and mechanism, they achieve comparable results to the CANSSR-S network, with a PSNR of 23.99 dB.

[0166] 2 Qualitative comparison

[0167] like Figure 8 As shown in Figure 2, a visual comparison of ×4 high-resolution images of different stereo image super-resolution reconstruction methods on the Flickr1024, KITTI 2012, KITTI 2015, and Middlebury datasets is performed. Specifically, in the visualization results of Flickr1024, RASSR significantly outperforms other methods in terms of clarity. The ceiling architectural lines reconstructed by the proposed method show a more obvious sense of depth and are very similar to the real lines. Continuing with the visualization results of KITTI 2012, the proposed technique makes the vertical line texture on the balcony more prominent. Similarly, in the visualization results of KITTI 2015, the method of enhances the readability of the depicted letters. Finally, when visualizing the results of Middlebury, the method of is able to reconstruct clearer gaps between the piano keys.

[0168] In order to verify the effectiveness of the method, this embodiment also performs high-resolution image visualization with a sampling factor of x2 on the Flickr1024 and KITTI 2012 datasets, as shown in Fig. 9 Compared with other methods, RASSR can restore neater and clearer lines.

[0169] In addition, to demonstrate that attention was improved, e.g. Fig.10 As shown, this embodiment takes the right HR images of NAFSSR and RASSR respectively, and visualizes their differences along the epipolar line. And, a heat map (W×1) of the difference between NAFSSR and RASSR is visualized along this epipolar line for comparison. Both the difference and the attention level are from dark to bright. The disparity attention map is visualized for the difference areas intercepted by NAFSSR and RASSR respectively. From the enlarged image, it can be seen that the attention of the NAFSSR method is obviously focused on the area with large difference, while the attention of the RASSR method of the present invention has been shifted to the surrounding reliable areas, which proves the effectiveness of the method.

[0170] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0174] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0175] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of stereo image super-resolution reconstruction methods based on reliable attention under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A stereo image super-resolution reconstruction method based on reliable attention, characterized in that ； Unreliable features are filtered out and modulated through uncertainty estimation to improve the reconstruction quality of low-resolution images; Extract deep features and uncertainty estimation through the Monte Carlo feature extraction block MCFEBlock; Providing uncertainty estimation for feature screening through MCdropout method; Unreliable features are filtered and modulated through the feature modulation module FMM.

2. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 1, characterized in that: The Monte Carlo feature extraction block MCFEBlock includes a depth information extractor DIE and a feature compensation module FCM; the feature compensation module FCM is used to compensate for feature loss caused by neuron loss during MCdropout training.

3. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 2, characterized in that: A stereo image super-resolution network based on reliable attention is used for super-resolution reconstruction, including four parts: initial feature extraction, deep feature extraction, feature interaction and image reconstruction; the left and right branches of the network adopt a weight sharing strategy and the feature extraction process is the same: First, a 3×3 convolutional layer is used to extract the low-resolution images from the left and right images. Extract the initial features Where H×W represents the spatial dimension and C represents the number of channels, which can be expressed as: in, and are left and right low-resolution images, and is the initial feature extracted from the left and right low resolutions, C 3x3 It is a 3×3 convolution operation; Then, N MCFEBlocks and M NAFBlocks are used for deep feature extraction; each MCFEBlock and NAFBlock is followed by FMM and SCAM, respectively, as follows: in, and represents the i-th MCFEBlock and FMM, and represents the j-th NAFBlock and SCAM, i∈(1,N), j∈(1,M), Represents the output after passing through i MCFEBlocks, It represents the output after passing through j NAFBlocks. represents the sum of the output after passing through i+j blocks; Finally, a 3×3 convolutional layer and a shuffle layer are used to upsample the features by a scale factor s. Global residual learning is used to predict only the residual between the bilinearly upsampled low-resolution image and the true image. It is expressed as: Among them, U ↑ represents bilinear interpolation, S ↑ Indicates the shuffle layer; Represents the left and right high-resolution images.

4. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 2, characterized in that: The depth information extractor DIE is expressed as: Pre0=D ro (FFN(D ro (MBConv(F in ))) Pre1❝D ro (FFN(D ro (MBConv(F in ))) Among them, F in represents the input features; Pre0 and Pre1 represent two different feature outputs; MBConv and FFN refer to the mobile convolutional block and feedforward network used in NAFBlock, respectively; D ro represents a randomly dropped neuron layer; During the training phase, a dropout layer is added after each MBConv layer and the feed-forward network FFN respectively; different output features are obtained by cycling DIE twice; the two feature outputs are then concatenated and their pixel-by-pixel standard deviation is calculated as the approximate uncertainty estimate, and the uncertainty is converted into a mask to filter the features.

5. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 1, characterized in that: The specific method of uncertainty estimation is: Based on Monte Carlo inference, the model uncertainty is estimated by using random sampling and repeatedly running the model; during the prediction process, multiple forward passes are made on the input samples, and the uncertainty is estimated based on the mean or variance of these predictions; expressed as: Among them, Uncertainty represents the pixel-by-pixel standard deviation; Represents a join operation; Mean and Std represent mean and standard deviation operations.

6. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 4, characterized in that: The method of converting uncertainty into a mask is to adaptively use the mean + standard deviation of each uncertainty estimate as a threshold during training, and compare each pixel value in the uncertainty with the threshold to assign a value, which is expressed as follows: Among them, M represents the mask value, un represents the pixel value of uncertainty; τ is the threshold adaptively obtained during the training process.

7. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 2, characterized in that: The feature compensation module FCM has two identical branches, enriching the inter-view features through the multi-convolution module MCM and adding the learned features to the opposing features; It is expressed as: in, is the final compensated characteristic, and is the feature after interaction, A vg For average operation; and are the left and right input features of the first and second predictions, C n×n It is an n×n convolution operation, SG is the SimpleGate function, and LKCA is the large kernel convolution attention mechanism; The SimpleGate function splits the input into two features X1,X2∈R along the channel dimension H×W×C / 2 , The output is then calculated through linear gates: SG=X1⊙X2 where ⊙ is the product of the elements.

8. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 1, characterized in that: After endowing the feature extraction module with quality assessment capability, the feature modulation module FMM uses the uncertainty mask obtained from MCFEBlock to screen the features to be modulated: In the modulation process of the left view, for features with lower uncertainty, they are directly output; for features with higher uncertainty, the features are modulated through N cyclic interaction modules LIM, and then the weighted features are output, as shown below: in, is the output characteristic after modulation, is a low uncertainty feature, is a high uncertainty feature, M L,R is the high uncertainty mask value; Then, the high uncertainty features of the left view and the low uncertainty features of the right view are put into the attention interaction module SCAM for learning, and the reliable features of the right view that are beneficial to the left view are obtained. The left view gradually shifts its attention to the reliable features; in addition, the features in the left view are enriched by channel attention; finally, a 3×3 convolution block is added; the attention of the right view is also improved through the same mechanism; the process is as follows: Among them, LIM N Indicates that the LIM module cycles N times, C 3×3 is a 3×3 convolution operation, is the output after each LIM module modulation, is the feature of the left view that is beneficial to the right view, is the feature that the right view is beneficial to the left view, and CA is the channel attention mechanism.

9. The method for stereoscopic image super-resolution reconstruction based on reliable attention according to claim 3, characterized in that: The loss function used by the stereo image super-resolution network is as follows: in, and represents the left and right real images, and represents the left and right high-resolution images, and λ is the weight; For the frequency domain loss, the frequency Charbonnier loss function is used as follows: in, stands for Fast Fourier Transform.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores the computer program capable of running on the processor, characterized in that: When a processor loads and executes a computer program, the method according to any one of claims 1 to 9 is adopted.

Images