Binocular vision image defocus blur restoration method based on cross-view interaction

Abstract

The application discloses a kind of binocular vision image defocus blur recovery methods based on cross-view interaction, belong to computer vision field.The method includes parallel left and right image deblurring network and cross-view feature interaction network between left and right deblurring network.Wherein, left and right image deblurring network is based on the symmetry U-Net structure of encoder-decoder, including feature extraction, reconstruction, recovery stage;Cross-view feature interaction network is embedded in stereo cross-view attention module between each corresponding level of encoder and decoder, and the feature correlation between left and right subgraphs is used to assist deblurring.The application can realize binocular image collaborative deblurring function, solve the problem of image quality damage caused by defocus degradation and binocular feature inconsistency in complex depth of field scene.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and in particular relates to a method for restoring defocus blur in binocular vision images based on cross-view interaction. Background Technology

[0002] Binocular vision technology, as an important means of 3D scene perception, calculates disparity by stereo matching of left and right views and derives the depth information of target objects by combining triangulation geometric models. It has broad application prospects in fields such as robot navigation, vehicle environmental perception, and high-precision industrial inspection. However, in actual imaging, due to the physical limitation of the finite depth of field of the optical system, object points outside the focal plane will form circles of confusion on the photosensitive element, resulting in image defocus blur. Defocus blur not only leads to severe image quality degradation but is also one of the key factors restricting the performance of binocular vision systems. Blur can cause the loss of high-frequency textures in the image and blurring of the left and right views. Figure 1 Defocus blur restoration of binocular vision images leads to decreased accuracy in subsequent feature point matching, increased disparity estimation error, and deviations in depth information calculation, ultimately affecting the reliability of 3D reconstruction. Therefore, researching defocus blur restoration methods for binocular vision images has significant theoretical and practical value.

[0003] Currently, image deblurring methods are mainly divided into traditional image prior-based methods and deep learning-based methods. Traditional blur restoration methods typically treat restoration as a constrained mathematical inverse problem optimization process. These methods heavily rely on manually designed statistical priors (such as dark channel priors or total variational priors), exhibiting poor robustness when facing drastic lighting fluctuations, high-intensity noise, and complex non-uniform blur, and often face problems such as difficulty in blur kernel estimation and high computational cost. Monocular restoration methods based on deep learning utilize neural networks to learn the nonlinear mapping between degraded and sharp images, significantly improving processing efficiency. However, monocular algorithms neglect the correlation characteristics between binocular images and cannot utilize redundant and complementary information across views to assist restoration, resulting in poor left-right consistency of the restoration results. Most existing research on binocular image deblurring focuses on solving motion blur caused by camera shake or object displacement, while restoration methods specifically for defocus blur remain relatively scarce. Defocus blur exhibits high spatial variability, and its degradation degree is closely coupled with the geometric depth of the scene, requiring algorithms to have extremely strong three-dimensional spatial perception capabilities. Furthermore, the scarcity of high-quality binocular defocus datasets directly limits the generalization ability of existing models in real-world physical scenarios.

[0004] In summary, how to effectively mine and utilize the geometric constraints between binocular views to construct a collaborative restoration framework that combines physical fidelity with cross-view information complementarity is a technical problem that urgently needs to be solved in the current field of binocular vision. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of feature matching failure and reduced stereo matching accuracy caused by defocus blur in binocular stereo vision systems under complex depth-of-field environments. This invention proposes a binocular vision image defocus blur restoration method based on cross-view interaction.

[0006] To achieve the objective of this invention, a method for restoring defocus blur in binocular vision images based on cross-view interaction is disclosed, comprising the following steps:

[0007] Step 1: Image Input and Basic Feature Extraction; The left and right out-of-focus blurred images to be restored are compared... Input a two-branch symmetric network, through Convolutional layers upscale an image from its raw channels to a defined feature dimension. Extract basic features;

[0008] Step 2, Feature extraction and downsampling in the encoding stage: Features are extracted using an encoder module that includes an efficient multi-scale attention mechanism (EMA), and the spatial information of the image is losslessly transferred to the channel dimension using a Haar wavelet downsampling (HWD) module to achieve resolution compression.

[0009] Step 3: Cross-view feature interaction; embed a stereo cross-view attention module (SCAM) between each corresponding level of the encoder and decoder, and use the horizontal epipolar constraint of the binocular images to calculate the correlation of the features of the left and right views, so as to achieve complementary fusion of asymmetric defocus information.

[0010] Step 4: Feature restoration in the decoding stage; Image resolution is restored by upsampling, features from the corresponding encoding stage are fused using skip connections, and feature enhancement is performed again through the EMA module and cross-view verification is performed through the SCAM module.

[0011] Step 5: Clear Image Generation; After each level of decoding and cross-view verification is completed, through... The convolutional layer reduces the dimensionality of the feature map to the original image dimension, ultimately generating end-to-end left and right restored image pairs. .

[0012] Furthermore, during network training, a binocular image defocus blur dataset is constructed using a layered stacking depth-of-field technique. Defocus blur is stacked layer by layer on clear binocular image pairs according to the disparity size. The maximum blur kernel size and focus point position of the image are selected. Based on the MiddleBury21 dataset, a corresponding blur-clear dataset is constructed for network training and testing.

[0013] Furthermore, the left and right branch image deblurring network consists of a 4-layer encoder network and a 4-layer decoder network.

[0014] Furthermore, the basic modules of the encoder-decoder network integrate two-layer convolution with an efficient multi-scale attention mechanism.

[0015] Furthermore, during the network training phase, a composite loss function is employed, which is a weighted combination of mean squared error loss and perceptual loss, used to balance pixel-level accuracy with semantic perceptual consistency of the human visual system.

[0016] Furthermore, in step 1, the image input and basic feature extraction are expressed as follows:

[0017]

[0018] The input left and right blurred image pair is first passed through a 3×3 convolutional layer to separate the images. , From increasing the original number of channels to the feature dimension C set in the feature extraction stage, basic features are extracted. , .

[0019] Furthermore, in step 2, the feature extraction stage uses an encoder to extract image features. This stage contains four network layers. Each layer first passes through an encoder module containing an EMA (Image Image Module), and then performs HWD (Hardware-Driven Downsampling) operation, expressed as follows:

[0020]

[0021] Before downsampling, spatial correlations at the current scale are learned through a parallel sub-network; where, The feature output of the i-th layer, The feature output of the i-th layer after processing by the EMA module. For convolutional layers containing EMA modules, This is a downsampling convolutional layer.

[0022] Furthermore, in step 3, after each encoder layer, the left and right views interact across views via SCAM, as shown in the following formula:

[0023]

[0024] in, , These represent the mapping features from the left image to the right image and from the right image to the left image in the i-th layer, respectively. For convolutional layers containing SCAM, , These are the features of the left and right images of the i-th layer, respectively.

[0025] Furthermore, in step 4, the feature restoration stage uses a decoder to restore the image. The decoder first restores the resolution through upsampling and then fuses the features from the corresponding encoding stage through skip connections.

[0026]

[0027] in, The output result is a skip connection fusion of the i-th layer decoder and its corresponding encoder. This represents the decoder output feature result after SCAM processing at layer i+1. Let i be the encoder feature corresponding to the i-th layer, and [,] represent the concatenation operation of the channel dimension. The deepest decoder corresponds to i=4.

[0028] The decoder module containing EMA is used to further enhance the saliency of the fused features, capturing local and global pixel correlations during the reconstruction process;

[0029]

[0030] in, The feature output of the i-th layer decoder;

[0031] Following each level of decoder, the SCAM module performs a final cross-view verification:

[0032]

[0033]

[0034] These are the left and right image decoder output feature results of the i-th layer after SCAM processing, respectively; These are the left and right image feature results output by the i-th layer decoder, respectively; , These are the decoder mapping features from the left image to the right image and from the right image to the left image in the i-th layer, respectively.

[0035] Furthermore, in step 5, the feature map is reduced to the original image dimension using a 3×3 convolutional layer to obtain the left and right restored images. , :

[0036]

[0037] in, , These are the left and right image decoder output feature results after SCAM processing in the first layer, respectively.

[0038] Compared with existing technologies, the significant advancements of this invention are as follows: 1) By establishing a binocular vision image defocus blur restoration network model based on cross-view interaction, the restored image quality outperforms existing binocular deblurring technologies in terms of evaluation metrics such as PSNR and SSIM; 2) By integrating EMA and HWD into a parallel left and right image deblurring network, this invention improves the network's ability to repair defocus edges of complex spatial variations; 3) By introducing a cross-view attention mechanism, this invention utilizes the redundancy and complementarity between binocular views. When a certain view is severely blurred locally, high-frequency textures can be retrieved from the corresponding clear position in the other view, achieving feature reconstruction that monocular algorithms cannot accomplish.

[0039] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0041] Figure 1 This is a diagram of the binocular vision image defocus blur restoration network structure of the present invention;

[0042] Figure 2 This is a diagram of the branched single-image defocusing blur removal network structure of the present invention;

[0043] Figure 3 The flowchart for constructing a binocular vision image defocus blur dataset for this invention is shown below. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] A method for restoring defocus blur in binocular vision images based on cross-view interaction, comprising the following steps:

[0046] Its overall network structure is as follows Figure 1 As shown, the input left and right blurred image pairs are first passed through a 3×3 convolutional layer to... , From increasing the original number of channels to the feature dimension C set in the feature extraction stage, basic features are extracted. , This process can be expressed by the following formula:

[0047]

[0048] Increasing the dimensionality of the feature space allows the network to represent more feature combinations, which helps the network better capture details such as texture and edges in an image. These features will then be further processed and fused in subsequent feature extraction and feature restoration stages.

[0049] Branch-based single-image deblurring network structure as follows Figure 2 As shown, the feature extraction stage uses an encoder to extract image features. This stage contains four network layers. Each layer first passes through an encoder module containing an EMA (Image Equation), and then performs HWD (Hardware-Driven Downsampling) operation. This can be expressed as:

[0050]

[0051] Before downsampling, spatial relationships at the current scale are learned through a parallel sub-network. Because the resolution is higher at this point, EMA can more accurately establish long-range dependencies between object contours and blurred regions, enhancing important features. HWD downsampling, while reducing spatial resolution, losslessly transfers the EMA-enhanced detailed features to the channel dimension through orthogonal transformation.

[0052] The efficient multi-scale attention (EMA) module adopts a grouped parallel attention mechanism, and its processing flow can be divided into three stages: feature grouping, parallel branch inference, and cross-spatial information interaction.

[0053] The first step is feature grouping, for the input feature map. EMA divides it into g subgroups along the channel dimension:

[0054]

[0055] This grouping strategy not only reduces computational complexity but also allows the network to extract multi-scale spatial features in parallel across the three sub-paths without dimensionality reduction.

[0056] Then, within each subgroup, EMA constructs three parallel feature extraction branches, each focusing on a different spatial scale. The first two branches utilize two one-dimensional global average pooling (GAP) methods to capture spatial dependencies in the vertical and horizontal directions, respectively.

[0057] Vertical encoding (X Avg Pool):

[0058]

[0059] Horizontal Avg Pool:

[0060]

[0061] The results of these two encoding paths are concatenated and fed into a 1×1 convolution. Finally, the spatial weights are generated using the Sigmoid function.

[0062] Another branch sets up a 3×3 convolutional kernel in parallel to capture local cross-channel interaction information and expand the feature space. It works in conjunction with the 1×1 branch to ensure that the network can collect spatial information at different scales in the same processing stage.

[0063] Finally, there is cross-spatial information interaction, which achieves richer feature fusion through the aggregation of information from different spatial dimensions. For the outputs of 1×1 and 3×3 branches, 2D global average pooling (Avg Pool) is introduced respectively.

[0064]

[0065] Next, the global pooling output of one branch is multiplied by a matrix (Matmul) with the original feature output of another branch. This cross-dimensional interaction aims to capture pixel-level pairwise relationships and highlight global contextual information for all pixels.

[0066] Finally, the spatial attention maps generated by the two sub-paths are aggregated and applied to the original feature map through a Sigmoid function to generate the final output.

[0067] The Haar wavelet downsampling (HWD) module comprises a lossless feature encoding module and a feature representation learning module. The main responsibility of the lossless feature encoding module is to transform the features and reduce their spatial resolution through the Haar wavelet transform layer. A two-dimensional image with a resolution of H×W will generate four components using the Haar wavelet transform: a low-frequency approximation component A, and high-frequency detail components in the horizontal direction (H), vertical direction (V), and diagonal direction (D). Each component's size is reduced to [value missing]. However, due to the generation of four components, the number of channels in the feature map becomes four times the original. The feature representation learning module after the lossless feature encoding block consists of a standard 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function, where the standard convolutional layer is used to adjust the number of channels in the feature map.

[0068] After each encoder layer, the left and right views interact across views via SCAM, as shown in the following formula:

[0069]

[0070] The core of the 3D cross-view attention mechanism SCAM is based on scaled dot product attention. It achieves weighted aggregation of numerical vectors (V) by calculating the similarity between the query vector (Q) and the key vector (K).

[0071]

[0072] For image tasks, Q is the query vector, representing the information the current pixel wants to find; K is the key vector, representing the feature labels carried by other pixels; V is the value vector, representing the actual feature content that other pixels want to convey; d K The dimension representing the eigenvector corresponds to the feature map channel dimension C in SCAM. The transpose matrices of Q and K are first multiplied by a dot product, and then the result of the dot product is divided by... The scaled result is then normalized using the Softmax function, and finally multiplied by V to obtain the feature representation.

[0073] Before calculating Q, K, and V, the network first receives feature maps from the left and right image branches. Next, the input is subjected to layer normalization to obtain the normalized features. and :

[0074]

[0075] Next, the normalized feature map is projected through a 1×1 convolutional layer, changing the feature representation space to obtain Q, K, and V vectors:

[0076]

[0077]

[0078] in , , , is a learnable projection weight matrix.

[0079] Because binocular vision images are highly symmetrical under epipolar constraints, SCAM calculates the correlation of cross-view features in the horizontal direction (i.e., the W dimension):

[0080]

[0081] SCAM can generate attention matrix by calculating it only once. and Finally, cross-view information is merged by adding elements together. , and information within the view , :

[0082]

[0083] in, , This is a learnable scaling factor, initialized to zero, which allows the network to smoothly transition from pure monocular restoration to binocular joint restoration mode.

[0084] The left and right subimages fuse the features obtained from each other with the features of their own image to obtain the fused features, represented as follows:

[0085]

[0086] The feature restoration stage uses a decoder to restore the image. The decoder first restores the resolution through upsampling and then fuses the features from the corresponding encoding stage through skip connections.

[0087]

[0088] Where [,] represents the concatenation operation of the channel dimension, and the deepest decoder corresponds to i=4.

[0089] The fused feature map contains rich scale information. At this point, a decoder module containing EMA is used to further enhance the saliency of the fused features, capturing the local and global pixel correlations during the reconstruction process.

[0090]

[0091] Following each level of decoder, the SCAM module performs a final cross-view verification:

[0092]

[0093]

[0094] Finally, a 3×3 convolutional layer is used to reduce the dimensionality of the feature map to the original image dimension, resulting in the restored left and right images. , :

[0095]

[0096] During network training, such as Figure 3 As shown, a binocular image defocus blur dataset is constructed using a layered stacking depth-of-field technique. Defocus blur is stacked in layers according to the disparity of the clear binocular image pairs. The maximum blur kernel size and focus point position of the image are selected. Based on the MiddleBury21 dataset, a corresponding blur-clear dataset is constructed for network training and testing.

[0097] A composite loss function combining mean squared error loss and perceptual loss was used when training the network.

[0098]

[0099]

[0100] Where is the order of magnitude of the balancing loss function. Set it to 0.01.

[0101] Specifically, the formulas for the mean squared error loss and the perceived loss function are as follows:

[0102]

[0103]

[0104] in, This represents the feature map extracted by the j-th convolutional layer of the VGG-19 network, where j takes the value 15.

[0105] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0106] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for restoring defocus blur in binocular vision images based on cross-view interaction, characterized in that, Includes the following steps: Step 1: Image input and basic feature extraction; The left and right out-of-focus blurred images to be restored are paired... Input a two-branch symmetric network, through Convolutional layers upscale an image from its raw channels to a defined feature dimension. Extract basic features; Step 2, Feature extraction and downsampling in the encoding stage: Features are extracted using an encoder module that includes an efficient multi-scale attention mechanism (EMA), and the spatial information of the image is losslessly transferred to the channel dimension using a Haar wavelet downsampling (HWD) module to achieve resolution compression. Step 3: Cross-view feature interaction; Stereo cross-view attention module (SCAM) is embedded between each corresponding level of the encoder and decoder. By utilizing the horizontal epipolar constraint of the binocular images, the correlation between the features of the left and right views is calculated, thereby achieving complementary fusion of asymmetric defocus information. Step 4: Feature restoration in the decoding stage; Image resolution is restored by upsampling, features from the corresponding encoding stage are fused using skip connections, and feature enhancement is performed again through the EMA module and cross-view verification is performed through the SCAM module. Step 5: Clear Image Generation; After each level of decoding and cross-view verification is completed, through... The convolutional layer reduces the dimensionality of the feature map to the original image dimension, ultimately generating end-to-end left and right restored image pairs. .

2. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, During network training, a binocular image defocus blur dataset was constructed using a layered stacking depth-of-field technique. Defocus blur was stacked in layers according to the disparity of the clear binocular image pairs. The maximum blur kernel size and focus point position of the image were selected. Based on the MiddleBury21 dataset, a corresponding blur-clear dataset was constructed for network training and testing.

3. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, The left and right branch image deblurring network consists of a 4-layer encoder network and a 4-layer decoder network.

4. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, The basic module of the encoder-decoder network integrates two-layer convolution and an efficient multi-scale attention mechanism.

5. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, During the network training phase, a composite loss function is used, which is a weighted combination of mean squared error loss and perceptual loss, to balance pixel-level accuracy with semantic perception consistency of the human visual system.

6. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, In step 1, the image input and basic feature extraction are expressed as follows: The input left and right blurred image pair is first passed through a 3×3 convolutional layer to separate the images. , From increasing the original number of channels to the feature dimension C set in the feature extraction stage, basic features are extracted. , .

7. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, In step 2, the feature extraction stage uses an encoder to extract image features. This stage consists of four network layers. Each layer first passes through an encoder module containing an EMA (Image Image Module), and then undergoes HWD (Hardware-Driven Downsampling) operation, expressed as follows: Before downsampling, spatial correlations at the current scale are learned through a parallel sub-network; where, The feature output of the i-th layer, The feature output of the i-th layer after processing by the EMA module. For convolutional layers containing EMA modules, This is a downsampling convolutional layer.

8. The method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, In step 3, after each encoder layer, the left and right views interact across views via SCAM. The formula for this step is as follows: in, , These represent the mapping features from the left image to the right image and from the right image to the left image in the i-th layer, respectively. For convolutional layers containing SCAM, , These are the features of the left and right images of the i-th layer, respectively.

9. A method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, In step 4, the feature restoration stage uses a decoder to restore the image. The decoder first restores the resolution through upsampling and then fuses the features from the corresponding encoding stage through skip connections. in, The output result is a skip connection fusion of the i-th layer decoder and its corresponding encoder. This represents the decoder output feature result after SCAM processing at layer i+1. Let i be the encoder feature corresponding to the i-th layer, and [,] represent the concatenation operation of the channel dimension. The deepest decoder corresponds to i=4. The decoder module containing EMA is used to further enhance the saliency of the fused features, capturing local and global pixel correlations during the reconstruction process; in, The feature output of the i-th layer decoder; Following each level of decoder, the SCAM module performs a final cross-view verification: These are the left and right image decoder output feature results of the i-th layer after SCAM processing, respectively; These are the left and right image feature results output by the i-th layer decoder, respectively; , These are the decoder mapping features from the left image to the right image and from the right image to the left image in the i-th layer, respectively.

10. A method for restoring defocus blur in binocular vision images based on cross-view interaction according to claim 1, characterized in that, In step 5, the feature map is reduced to the original image dimension using a 3×3 convolutional layer to obtain the restored left and right images. , : in, , These are the left and right image decoder output feature results after SCAM processing in the first layer, respectively.

Images