Deep learning binocular vision estimation method and device fused with edge optimization

Abstract

The invention discloses a deep learning binocular vision estimation method and device fused with edge optimization, and the method comprises the steps: carrying out the image correction and preprocessing of an initial left-eye image and an initial right-eye image, and obtaining a target left-eye image and a target right-eye image; according to the target left-eye image and the target right-eye image, a target disparity map and an entropy confidence value are obtained through a binocular vision estimation framework; obtaining a target dense depth map according to the target disparity map and the entropy confidence value; wherein the binocular vision estimation framework comprises a deep learning feature extraction operation, a semi-supervised mechanism fusing edge perception and pseudo label correction, and a confidence modeling mechanism. The binocular vision estimation method can improve the robustness and accuracy of binocular vision estimation, and can be widely applied to the technical field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a deep learning binocular vision estimation method and device fusing edge optimization. BACKGROUND

[0002] The world model endows the system with the ability to understand the world, plan actions, and adapt to new environments through environment modeling, prediction, and decision support. Binocular depth estimation can provide three-dimensional geometric information of the scene, helping the world model accurately reconstruct the three-dimensional structure of the environment. This is crucial for scenarios that require spatial understanding, such as robot navigation, autonomous driving, virtual reality (VR), or augmented reality (AR).

[0003] Traditional binocular depth estimation algorithms rely on explicit feature matching and regularized disparity optimization, which are prone to disparity errors in scenes with sparse texture, occlusion, or weak light, limiting their robustness and accuracy in real-world tasks. In addition, existing commercial depth cameras (such as ZED) lack edge details and depth continuity, making it difficult to meet the demand for high-precision three-dimensional perception. SUMMARY

[0004] Therefore, the embodiments of the present application mainly aim to provide a deep learning binocular vision estimation method and device fusing edge optimization, so as to solve at least one of the problems in the prior art, and improve the robustness and accuracy of binocular vision estimation.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a deep learning binocular vision estimation method fusing edge optimization, which comprises:

[0006] performing image correction and preprocessing operations on the initial left and right images to obtain target left and right images;

[0007] obtaining a target disparity map and an entropy confidence value through a binocular vision estimation framework according to the target left and right images;

[0008] obtaining a target dense depth map according to the target disparity map and the entropy confidence value;

[0009] The binocular vision estimation framework comprises a deep learning feature extraction operation, a semi-supervised mechanism fusing edge perception and pseudo-label correction, and a confidence modeling mechanism.

[0010] In some embodiments, the step of obtaining a target disparity map and an entropy confidence value through a binocular vision estimation framework according to the target left and right images comprises the following steps:

[0011] generating an initial disparity map through a deep learning feature extraction operation according to the target left and right images;

[0012] The initial disparity map is optimized by a semi-supervised mechanism of fusing edge perception and pseudo-label correction to obtain the target disparity map.

[0013] Each pixel point of the target disparity map is modeled by a confidence modeling mechanism to obtain the entropy confidence value.

[0014] In some embodiments, the initial disparity map is generated from the target left and right view images by a deep learning feature extraction operation, including the following steps:

[0015] The target left and right view images are subjected to a semantic feature extraction operation to obtain left and right view features.

[0016] The left and right view features are subjected to explicit matching or implicit matching to generate the initial disparity map.

[0017] In some embodiments, the initial disparity map is optimized by a semi-supervised mechanism of fusing edge perception and pseudo-label correction to obtain the target disparity map, including the following steps:

[0018] An edge-guided loss function, a pseudo-label supervised loss function, and a disparity smoothing regularization term are constructed.

[0019] An edge loss weight, a smoothing regularization weight, and a pseudo-label confidence threshold are pre-set.

[0020] The initial disparity map is semi-supervised optimized according to the edge-guided loss function, the pseudo-label supervised loss function, the disparity smoothing regularization term, the edge loss weight, the smoothing regularization weight, and the pseudo-label confidence threshold to obtain the target disparity map.

[0021] In some embodiments, the initial disparity map is optimized by a semi-supervised mechanism of fusing edge perception and pseudo-label correction to obtain the target disparity map, and the formula used includes:

[0022]

[0023] In the formula, represents an edge-guided loss; represents a disparity gradient; e i,j represents an image edge response; represents a pseudo-label supervised loss; d i,j represents a disparity value of an initial disparity image pixel; represents a disparity value generated by a pseudo-label; represents a disparity smoothing regularization term; represents a gradient of a disparity in the x direction; denotes the gradient of disparity in y direction; e denotes natural base; denotes the gradient of gray image in x direction; denotes the gradient of gray image in y direction; (i,j) denotes a pixel point; Ω p denotes the high-confidence pseudo-label region.

[0024] In some embodiments, the confidence modeling mechanism is used to model the confidence of each pixel point of the target disparity map to obtain the entropy confidence value, including the following steps:

[0025] Obtaining the probability distribution of each pixel point of the target disparity map under different disparities;

[0026] Through confidence estimation, the information entropy of the pixel point is obtained according to the probability distribution;

[0027] The information entropy is subjected to negative value and normalization operation to obtain the entropy confidence value.

[0028] In some embodiments, the confidence modeling mechanism is used to model the confidence of each pixel point of the target disparity map to obtain the entropy confidence value, and the formula used includes:

[0029]

[0030] In the formula, H i,j denotes the information entropy of the pixel point (i,j); D max denotes the maximum disparity range; P(d| i,j) denotes the probability distribution of the pixel point under different disparities d; denotes the entropy confidence value; H min denotes the minimum information entropy; H max denotes the maximum information entropy.

[0031] In some embodiments, the target dense depth map is obtained according to the target disparity map and the entropy confidence value, including the following steps:

[0032] The target disparity map is converted into an initial dense depth map through binocular geometric relationship;

[0033] The initial dense depth map is optimized according to the entropy confidence value to obtain the target dense depth map.

[0034] To achieve the above purpose, another aspect of the embodiment of the present application proposes a depth learning binocular vision estimation device fusing edge optimization, which comprises:

[0035] A correction and preprocessing module is used to perform image correction and preprocessing operation on the initial left and right eye images to obtain target left and right eye images;

[0036] a binocular vision estimation module configured to obtain a target disparity map and an entropy confidence value by a binocular vision estimation framework according to the target left-eye image and the target right-eye image;

[0037] a depth map generation module configured to obtain a target dense depth map according to the target disparity map and the entropy confidence value;

[0038] The binocular vision estimation framework comprises a deep learning feature extraction operation, a semi-supervised mechanism of fusion edge perception and pseudo label correction, and a confidence modeling mechanism.

[0039] In some embodiments, the binocular vision estimation module comprises:

[0040] a deep learning feature extraction submodule configured to generate an initial disparity map by a deep learning feature extraction operation according to the target left-eye image and the target right-eye image;

[0041] a semi-supervised submodule configured to optimize the initial disparity map by a semi-supervised mechanism of fusion edge perception and pseudo label correction to obtain the target disparity map;

[0042] a confidence modeling submodule configured to model the confidence of each pixel point of the target disparity map by a confidence modeling mechanism to obtain the entropy confidence value.

[0043] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0044] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0045] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method described above.

[0046] The embodiments of the present application at least have the following beneficial effects: the present application provides a deep learning binocular vision estimation method and device with fusion edge optimization, which obtains target left and right eye images through image correction and preprocessing operations on the initial left and right eye images; obtains a target disparity map and an entropy confidence value through a binocular vision estimation framework according to the target left and right eye images; and obtains a target dense depth map according to the target disparity map and the entropy confidence value; wherein the binocular vision estimation framework includes a deep learning feature extraction operation, a semi-supervised mechanism with fusion edge perception and pseudo-label correction, and a confidence modeling mechanism, which can improve the robustness and accuracy of binocular vision estimation. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0048] Figure 1 is a flowchart of the deep learning binocular vision estimation method with fusion edge optimization provided by the embodiments of the present application;

[0049] Figure 2 is a deep algorithm framework diagram of the deep learning binocular vision estimation with fusion edge optimization provided by the embodiments of the present application;

[0050] Figure 3 is a comparison diagram of the depth estimation effect of the ZED camera self-provided and the depth estimation effect of the method of the present application;

[0051] Figure 4 is a hardware structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0053] It should be noted that although the functional modules are divided in the system schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200" in the description and claims and the above-mentioned drawings can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0054] The terms "at least one", "multiple", "each", "any" and the like used herein include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0056] Existing deep learning estimation methods have a parallax transition blur problem at the target edge, which easily causes depth map edge distortion or discontinuity, affecting the accuracy of three-dimensional structure restoration. And most existing methods only output a single disparity value, lack of reliability analysis, and cannot identify and eliminate low-quality matching areas, resulting in uncontrollable depth estimation results quality. Most methods rely on a large number of high-quality disparity true values for supervised learning, with high training cost, and do not have self-adaptive ability to real unlabeled scenes, limiting the generalization and deployment range of the model. In actual scenes, some methods have poor recovery ability in detail areas, especially in complex environments with occlusion, repeated texture or uneven lighting, there is still a high risk of mismatch.

[0057] In view of this, as shown in the present application, an edge fusion and optimization deep learning binocular vision estimation method is provided, which can include but is not limited to steps S100 to S300: Figure 1

[0058] Step S100, image correction and preprocessing operation is performed on the initial left and right eye images to obtain target left and right eye images;

[0059] ​Step S200, according to the target left and right image, through the binocular vision estimation framework, the target disparity map and entropy confidence value are obtained;

[0060] Step S300, according to the target disparity map and the entropy confidence value, the target dense depth map is obtained;

[0061] The binocular vision estimation framework includes a deep learning feature extraction operation, a semi-supervised mechanism of fusion edge perception and pseudo label correction, and a confidence modeling mechanism.

[0062] In steps S100 to S300 of some embodiments, the input left and right binocular images are received, and camera intrinsic correction and brightness normalization are performed to improve the consistency of the images. Then, a deep neural network (such as a residual convolutional network) is used to extract features from the binocular images to obtain high-level semantic feature maps. By constructing a cost volume between the left and right image features, the initial disparity value of each pixel is estimated using methods such as soft argmin. An edge-guided semi-supervised mechanism is introduced to optimize the disparity results using partial pseudo-label data, reducing errors and blurring at the edges. The confidence of each pixel is calculated based on the cost distribution or output residual, and the estimation reliability is determined. Finally, the disparity map is converted to a dense depth map based on the known binocular camera baseline and focal length, realizing end-to-end high-precision depth estimation.

[0063] In step S100 of some embodiments, the calibrated initial left and right images are input, and image distortion correction, brightness normalization, and other preprocessing operations are performed to obtain target left and right images, to ensure the robustness and consistency of subsequent feature extraction.

[0064] For example, in a Stereo (stereovision) system, calibrated left and right images are input. Before formally entering the subsequent processing flow, a series of preprocessing operations will be performed on the two images. First, the initial left and right images are distorted and corrected, and the parameters obtained by camera calibration are used to correct the distortion of the left and right images using corresponding mathematical models and algorithms to restore the true geometric shape of the images, so that the objects in the images can more accurately reflect the layout of the actual scene.

[0065] Then, brightness normalization processing is performed. Due to the differences between the light sensing elements of the left and right cameras or the influence of different lighting conditions during shooting, the brightness of the left and right eye images may be inconsistent. Such brightness differences may interfere with subsequent image feature extraction and matching processes, reducing the accuracy and reliability of matching. The brightness normalization operation can be based on a certain algorithm. Optionally, the brightness distribution of the left and right eye images is adjusted to a similar range by calculating the global brightness mean and variance of the image, or a local contrast normalization method is used to eliminate the brightness difference between the left and right images, so that subsequent processing can be performed on images with similar brightness characteristics.

[0066] Optionally, other operations such as noise reduction processing can also be performed in the preprocessing stage. Because the image may be disturbed by various noises during the acquisition process, these noises will affect the quality of the image and the accuracy of the detailed information. A median filter, a Gaussian filter, or other filtering algorithms can be used to perform noise reduction processing on the left and right eye images, while trying to preserve the effective detailed information in the image, providing a clearer and more accurate image basis for subsequent feature extraction, feature matching, and other key steps.

[0067] In some embodiments, step S200 can include but is not limited to steps S210 to S230:

[0068] Step S210, according to the target left and right eye images, generating an initial disparity map through deep learning feature extraction operation;

[0069] Step S220, optimizing the initial disparity map through a semi-supervised mechanism that fuses edge perception and pseudo-label correction to obtain the target disparity map;

[0070] Step S230, modeling the confidence of each pixel point of the target disparity map through a confidence modeling mechanism to obtain the entropy confidence value.

[0071] In steps S210 to S230 of some embodiments, feature extraction, feature matching, and disparity estimation operations are performed on the target left and right eye images obtained after preprocessing, and on the basis of maintaining the deep neural network feature learning ability, a semi-supervised optimization mechanism that fuses edge perception is introduced to enhance the retention ability of structural boundaries. At the same time, a pixel-level confidence estimation strategy is used to model and filter the output disparity result, so that the subsequent generated depth map is superior to existing methods in terms of accuracy continuity, edge smoothness, and overall reliability, and is particularly suitable for SLAM, three-dimensional reconstruction, and other scenes with high three-dimensional geometric accuracy requirements.

[0072] In some embodiments, step S210 can include but is not limited to steps S211 to S212:

[0073] In step S211, semantic feature extraction is performed on the target left and right view images to obtain left and right image features.

[0074] In step S212, explicit matching or implicit matching is performed on the left and right image features to generate the initial disparity map.

[0075] In step S211 of some embodiments, a deep neural network is used to extract high-level semantic features from the target left and right view images to obtain left and right image features, thereby enhancing the matching discrimination and adapting to areas with unclear texture or blurred disparity.

[0076] In some embodiments, in the semantic feature extraction operation on the target left and right view images using a deep neural network (DNN), the relevant control parameters are reasonably set to ensure that the network can efficiently learn features with distinguishing characteristics. Optionally, for the network structure, ResNet18 is used as the backbone network, which has good balance, reduces the computational complexity while ensuring a certain depth, and can effectively alleviate the gradient vanishing problem, which is beneficial to the training and feature extraction of the network. The output channel number is set to 64, so that 64 feature maps of different dimensions are generated when extracting features, representing the features of the image from different aspects and enriching the feature information to provide sufficient feature basis for the subsequent matching process. For the convolution kernel size, the convolution kernel size is pre-set to 3x3, which reduces the amount of calculation and parameters while maintaining a certain receptive field, so that the network can capture local texture and edge features when performing convolution operation on image features, without causing the training and inference process to be too slow due to excessive calculation, ensuring the efficiency and effectiveness of the network. For Batch Size, it is set to 8, i.e. the network will process 8 images simultaneously for training in each iteration. In addition, the input image normalization mean is [0.485, 0.456, 0.406] and the standard deviation is [0.229, 0.224, 0.225]. After normalizing the image with the uniform mean and standard deviation, the input data has a relatively consistent distribution, so the network can learn effective features more quickly, reduce unstable factors in the training process, and improve the stability and accuracy of the model.

[0077] In some embodiments, after the relevant control parameters are reasonably set, the pixel values of the images are normalized according to the given normalized mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] before the target left and right image is input into the deep neural network (DNN). This step can accelerate the convergence speed of the network and improve the stability of the network. Then, the deep neural network is used to extract the high-level semantic features of the image. Exemplarily, ResNet18 is used as the backbone network, and ResNet18 includes multiple residual blocks which are connected by a skip connection to alleviate the gradient vanishing problem, so that the network can learn deep features more effectively. In each convolutional layer of the deep neural network, a 3x3 convolutional kernel is used to slide the window to extract local features, and the convolution operation can capture the edges, textures and other information in the image and convert them into higher-level feature representations. Moreover, the output channel number of the network is 64, and each input image will be converted into a 64-dimensional feature map.

[0078] In step S212 of some embodiments, an initial disparity map is generated by constructing a multi-scale cost volume and performing explicit or implicit matching of the left and right image features. Exemplarily, the left and right image features are represented as F L (i,j) and F R (i,j), respectively. By shifting the right image features, a disparity cost volume C(i,j,d) is constructed as follows:

[0079] C(i,j,d) = F L (i,j) · F R (i,j-d)

[0080] And the left image features are spliced with the shifted right image features as follows:

[0081] C ′ (i,j,d) = concat(F L (i,j), F R (i,j-d))

[0082] Finally, a three-dimensional cost volume C is formed for subsequent decoding processing. The three-dimensional cost volume is activated by softmax to convert into a disparity probability distribution:

[0083]

[0084] Finally, the disparity is obtained by soft-argmin:

[0085]

[0086] where d, d′ is the disparity; concat is the concatenation operation; D max represents the maximum disparity range; P(d| i, j) represents the probability distribution of the pixel point under different disparities d.

[0087] In some embodiments, step S220 can include but is not limited to steps S221 to S223:

[0088] Step S221, constructing an edge-guided loss function, a pseudo-label supervised loss function, and a disparity smoothness regularization term;

[0089] Step S222, pre-setting an edge loss weight, a smooth regularization weight, and a pseudo-label confidence threshold;

[0090] Step S223, performing semi-supervised optimization on the initial disparity map according to the edge-guided loss function, the pseudo-label supervised loss function, the disparity smoothness regularization term, the edge loss weight, the smooth regularization weight, and the pseudo-label confidence threshold to obtain the target disparity map.

[0091] In step S221 of some embodiments, the total loss is composed of three loss functions, i.e., an edge-guided loss function, a pseudo-label supervised loss function, and a disparity smoothness regularization term. Exemplarily, the edge-guided loss is:

[0092]

[0093] The pseudo-label supervised loss is:

[0094]

[0095] The disparity smoothness regularization term is:

[0096]

[0097] In the formula, represents the edge-guided loss; represents the disparity gradient; e i,j represents the image edge response; represents the pseudo-label supervised loss; d i,j represents the disparity value of the initial disparity image pixel point (i, j); represents the disparity value generated by the pseudo-label; represents the disparity smoothness regularization term; represents the gradient of the disparity in the x direction; represents the gradient of the disparity in the y direction; e represents the natural base; represents the gradient of the pixel point (i, j) in the x direction in the grayscale image; represents the gradient of pixel point (i, j) in the gray-scale image in the y direction; (i, j) represents a pixel point; Ω p represents a high-confidence pseudo-label region.

[0098] In step S222 of some embodiments, the edge loss weight, the smoothing regularization weight, and the pseudo-label confidence threshold are pre-set. Exemplarily, the edge loss weight is pre-set to 1.0 to emphasize the importance of edge information in the optimization process. The smoothing regularization weight is set to 0.1 to balance the smoothness and detail preservation. The pseudo-label confidence threshold is set to 0.7 to filter out high-confidence pseudo-labels in the pseudo-labels. Moreover, the proportion of pseudo-label data in the training can be set to 30%.

[0099] In step S223 of some embodiments, the initial disparity map is semi-supervisedly optimized according to the total loss composed of the edge-guided loss function, the pseudo-label supervised loss function, and the disparity smoothing regularization term, and the pre-set edge loss weight, the smoothing regularization weight, and the pseudo-label confidence threshold, to obtain the target disparity map. Exemplarily, through the semi-supervised optimization mechanism of fusing edge perception and pseudo-label correction, the initial disparity map is first aligned with the true edge at the object boundary by using the edge loss (weight 1.0) to ensure sharp contours; at the same time, the disparity smoothness in the non-edge region is constrained by the smoothing regularization (weight 0.1) to suppress noise. On this basis, high-confidence (threshold 0.7) pseudo-label data (proportion 30%) are introduced for supervised training to correct systematic deviations in the initial disparity map through the pseudo-label loss. After the joint optimization of the three, the optimized disparity estimation result maintains high accuracy in the edge region and continuity in the flat region, and finally outputs the target disparity map with clear boundaries and consistent structure.

[0100] In some embodiments, step S230 can include but is not limited to steps S231 to S233:

[0101] In step S231, the probability distribution of each pixel point of the target disparity map under different disparities is obtained.

[0102] In step S232, the information entropy of the pixel point is obtained according to the probability distribution through confidence estimation.

[0103] In step S233, the information entropy is subjected to a negative value taking operation and a normalization operation to obtain the entropy confidence value.

[0104] In step S231 of some embodiments, the constructed three-dimensional cost volume is converted into the probability distribution P(d| i, j) of each pixel under different disparities through a softmax operation:

[0105]

[0106] In step S232 of some embodiments, the interpretability and output reliability of the depth map are further improved by introducing a pixel-level confidence modeling mechanism. Then, the confidence estimation according to the probability distribution, i.e., the confidence estimation based on the parallax probability entropy, is as follows:

[0107]

[0108] In the formula, H i,j represents the information entropy of the current pixel point (i, j) position, representing uncertainty.

[0109] In step S233 of some embodiments, the confidence is defined as the normalized result of the negative value of the information entropy, and the entropy confidence value can be obtained by taking the negative value and normalizing the information entropy, as follows:

[0110]

[0111] In the formula, represents the entropy confidence value of the current pixel point (i, j) position; H min represents the minimum information entropy; H max represents the maximum information entropy.

[0112] In some embodiments, step S300 can include but is not limited to steps S310 to S320:

[0113] Step S310, converting the target parallax map into an initial dense depth map through the binocular geometric relationship;

[0114] Step S320, optimizing the initial dense depth map according to the entropy confidence value to obtain the target dense depth map.

[0115] In step S310 of some embodiments, based on the basic principle of binocular vision, i.e., the parallax is inversely proportional to the depth, and combined with the formula z = (f B) / d, the formula is applied to the target parallax map (each pixel has a parallax value) pixel by pixel to obtain the depth value corresponding to each pixel, and then an initial dense depth map (each pixel has a depth value) is generated. Wherein, z represents the depth value (the distance from the target point to the camera); f represents the focal length of the camera (in pixels); B represents the distance between the optical centers of the two cameras (baseline length); d represents the parallax value (the horizontal position difference of the corresponding pixels in the left and right images).

[0116] In step S320 of some embodiments, after obtaining the initial dense depth map, the depth values of the low-confidence regions can be smoothed or filtered using the entropy confidence values, reducing noise, and the depth values of the low-confidence regions can also be filled or corrected using the depth values of the surrounding high-confidence regions. By optimizing, filtering, or repairing the initial dense depth map using the entropy confidence values, a final target dense depth map with better quality can be obtained.

[0117] In some embodiments, as shown in FIG. 3, the binocular vision estimation process of the binocular vision estimation algorithm using fusion edge-optimized deep learning is as follows: Figure 2

[0118] Step 1, Stereo input and preprocessing: input the calibrated left and right eye images, and perform image distortion correction, brightness normalization, and other preprocessing operations to ensure the robustness and consistency of subsequent feature extraction.

[0119] Step 2, Feature extraction: use a deep neural network (DNN) to extract high-level semantic features of the images, obtaining left and right image features, enhancing the matching discrimination, and adapting to areas with unclear texture or blurred disparity.

[0120] Step 3, Feature matching and disparity estimation: construct a multi-scale cost volume, and perform explicit or implicit matching on the left and right image features to generate an initial disparity map.

[0121] Step 4, Edge-aware semi-supervised optimization (ESS): introduce an edge-aware semi-supervised learning mechanism, combining partial pseudo-labels and prior edge information to optimize the initial disparity map and improve the estimation effect at the boundary. During system training, three loss functions are introduced to form the total loss, including an edge-guided loss function, a pseudo-label supervised loss function, and a disparity smoothing regularization term.

[0122] Step 5, Disparity map generation and confidence estimation: the decoder outputs matching costs (or probability distributions) under multiple disparity hypotheses, which are converted into probability distributions for each pixel under different disparities through a softmax operation. To further improve the interpretability and output reliability of the depth map, a pixel-level confidence modeling mechanism is introduced, based on the entropy of the disparity probability to estimate the confidence of the pixel. The information entropy is taken as a negative value and normalized to obtain the entropy confidence value.

[0123] Step 6, Depth calculation and output: use the binocular geometric relationship to convert the disparity map into a dense depth map, which can be used as input for downstream three-dimensional perception systems.

[0124] Reference Figure 3 ​In some embodiments, the method of the present application is compared with the depth estimation effect of ZED camera in actual scene. From Figure 3 It can be seen directly from the depth map generated by ZED that there are many problems such as holes, edge breaks and structure blur, while the depth map output by the method of the present application still maintains good continuity and structural clarity in the object edge, illumination change area and low texture area, fully embodying the advantages of edge perception mechanism and confidence screening strategy. Further, the two methods are quantitatively compared from the three indexes of mean absolute error (MAE), mean square error (MSE) and root mean square error (RMSE), as shown in Table 1, the method of the present application is obviously better than the ZED device in all error indexes, the average absolute error is reduced by about 28.8%, and the mean square error and the root mean square error are reduced by more than 50% and 30% respectively. The results show that the present application has significant advantages in improving the depth estimation accuracy, detail restoration ability and robustness, and is suitable for high-precision three-dimensional reconstruction and perception tasks.

[0125] Table 1

[0126] Method Depth estimation of ZED camera Method of the invention Mean absolute error 0.104 0.074 Mean squared error 0.030 0.013 Root mean squared error 0.173 0.116

[0127] The embodiment of the present application also provides a depth learning binocular vision estimation device fused with edge optimization, which can realize the above-mentioned method, and the device comprises:

[0128] A correction and preprocessing module is configured to perform image correction and preprocessing operation on the initial left and right eye images to obtain target left and right eye images.

[0129] A binocular vision estimation module is configured to obtain a target disparity map and an entropy confidence value through a binocular vision estimation framework according to the target left and right eye images.

[0130] A depth map generation module is configured to obtain a target dense depth map according to the target disparity map and the entropy confidence value.

[0131] The binocular vision estimation framework comprises a deep learning feature extraction operation, a semi-supervised mechanism fused with edge perception and pseudo-label correction, and a confidence modeling mechanism.

[0132] The binocular vision estimation module comprises:

[0133] A deep learning feature extraction submodule is configured to generate an initial disparity map through a deep learning feature extraction operation according to the target left and right eye images.

[0134] A semi-supervised submodule is configured to optimize the initial disparity map through a semi-supervised mechanism fused with edge perception and pseudo-label correction to obtain the target disparity map.

[0135] The confidence modeling submodule is configured to model the confidence of each pixel point of the target disparity map by a confidence modeling mechanism to obtain the entropy confidence value.

[0136] It can be understood that the content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0137] The embodiment of the application further provides an electronic device, which comprises a processor and a memory, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0138] It can be understood that the content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0139] Reference Figure 4 , Figure 4 The electronic device of another embodiment is illustrated, which comprises:

[0140] The processor 401 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the application;

[0141] The memory 402 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 402 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the application are implemented by software or firmware, the related program codes are stored in the memory 402 and are called and executed by the processor 401 to implement the method of the embodiments of the application;

[0142] The input / output interface 403 is used to realize information input and output;

[0143] The communication interface 404 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth or the like).

[0144] The bus 405 is configured to transmit information between various components (for example, the processor 401, the memory 402, the input / output interface 403 and the communication interface 404) of the device.

[0145] The processor 401, the memory 402, the input / output interface 403 and the communication interface 404 are communicatively connected to each other in the device through the bus 405.

[0146] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method described above.

[0147] It can be understood that the contents in the above method embodiments are all applicable to the storage medium embodiment, the storage medium embodiment specifically realizes the functions of the above method embodiments, and the beneficial effects achieved by the storage medium embodiment are the same as the beneficial effects achieved by the above method embodiments.

[0148] The embodiment of the present application further provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above.

[0149] To sum up, the method and device for estimating binocular vision of deep learning fusion edge optimization according to the embodiment of the present application have the following advantages:

[0150] 1. The estimation accuracy is significantly improved: the embodiment of the present application introduces a deep neural network for end-to-end feature extraction and disparity estimation, which has stronger robustness and accuracy in weak texture, large light change or edge region compared with traditional methods, and can generate a more continuous and detailed depth map.

[0151] 2. The edge details are well maintained: the embodiment of the present application adopts an edge-aware semi-supervised optimization mechanism, which effectively improves the problem of blurred disparity estimation in the edge region of the object in the traditional method, and improves the quality of three-dimensional structure restoration.

[0152] 3. No need for a large amount of labeled data: the embodiment of the present application realizes a partially supervised learning process by combining pseudo-labels and confidence modeling, reduces the dependence on high-quality label data, and has good generalization ability.

[0153] In some alternative embodiments, the function / operations mentioned in the block diagrams can not occur in the order mentioned in the operational illustrations. For example, depending on the involved function / operation, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in reverse order, depending upon the functionality / operations involved. Furthermore, embodiments presented and described in the flowcharts are only examples of implementing the present application. Alternative embodiments can be implemented where various operations are changed, omitted, and / or added. Additionally, the order in which operations are described is not necessarily the order in which the operations are performed.

[0154] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features described can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary for an understanding of the present application. Rather, the actual implementation of the modules, in light of the attributes, functions, and internal relationships of the various functional modules disclosed herein, will be apparent to one of ordinary skill in the art, given the benefit of this disclosure. Accordingly, the present application is not limited to the specific embodiments illustrated herein, but is applicable for use in all domains of engineering consistent with the technical concepts disclosed herein. It will also be appreciated that the particular conceptual configuration disclosed is merely illustrative, and not intended to limit the scope of the present application, which is defined by the appended claims and their equivalents.

[0155] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the technical solutions that make essential contributions to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0156] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be embodied in non-transitory computer-readable media, executed by one or more computing devices, and / or in any other way. The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer. In this context, a "computer-readable medium" can be any means that can store the program for use by or in connection with the instruction execution system, apparatus, or device.

[0157] The computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer.

[0158] It should be understood that aspects of the application can be implemented in hardware, software, firmware, or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and / or the like.

[0159] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The illustrative examples described in this specification are not necessarily to be construed as being limiting, as the specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0160] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary and are not to be construed as limiting the scope of the application. The scope of the application is defined by the appended claims and their equivalents.

[0161] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the described embodiment, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A deep learning binocular vision estimation method with fusion edge optimization, characterized in that, The method comprises the following steps: image correction and preprocessing operations are performed on the initial left and right eye images to obtain target left and right eye images; According to the target left and right eye images, a binocular vision estimation framework is used to obtain a target disparity map and an entropy confidence value; According to the target disparity map and the entropy confidence value, a target dense depth map is obtained. The binocular vision estimation framework includes a deep learning feature extraction operation, a semi-supervised mechanism that fuses edge perception and pseudo-label correction, and a confidence modeling mechanism.

2. The method of claim 1, wherein, According to the target left and right eye images, a binocular vision estimation framework is used to obtain a target disparity map and an entropy confidence value, which comprises the following steps: According to the target left and right eye images, a deep learning feature extraction operation is used to generate an initial disparity map; The initial disparity map is optimized by a semi-supervised mechanism that fuses edge perception and pseudo-label correction to obtain the target disparity map; The confidence of each pixel point in the target disparity map is modeled by a confidence modeling mechanism to obtain the entropy confidence value.

3. The method of claim 2, wherein, According to the target left and right eye images, a deep learning feature extraction operation is used to generate an initial disparity map, which comprises the following steps: Semantic feature extraction is performed on the target left and right eye images to obtain left and right image features; Explicit matching or implicit matching is performed on the left and right image features to generate the initial disparity map.

4. The method of claim 2, wherein, The initial disparity map is optimized by a semi-supervised mechanism that fuses edge perception and pseudo-label correction to obtain the target disparity map, which comprises the following steps: An edge-guided loss function, a pseudo-label supervised loss function, and a disparity smoothing regularization term are constructed; The edge loss weight, the smoothing regularization weight, and the pseudo-label confidence threshold are pre-set; The initial disparity map is semi-supervised optimized according to the edge-guided loss function, the pseudo-label supervised loss function, the disparity smoothing regularization term, the edge loss weight, the smoothing regularization weight, and the pseudo-label confidence threshold to obtain the target disparity map.

5. The method of claim 2, wherein, The formula used in the semi-supervised mechanism that fuses edge perception and pseudo-label correction to optimize the initial disparity map to obtain the target disparity map includes: In the formula, denotes the edge guidance loss; denotes the disparity gradient;e i,j denotes the image edge response; denotes the pseudo-label supervision loss;d i,j denotes the disparity value of an initial disparity image pixel point; denotes the disparity value generated by the pseudo label; denotes the disparity smoothing regularization term; denotes the gradient of the disparity in the x direction; denotes the gradient of the disparity in the y direction;e denotes the natural base; denotes the gradient in the x direction in the grayscale image; denotes the gradient in the y direction in the grayscale image;(i,j) denotes a pixel point;Ω p denotes a high-confidence pseudo label region.

6. The method of claim 2, wherein, The confidence of each pixel point in the target disparity map is modeled by a confidence modeling mechanism to obtain the entropy confidence value, which comprises the following steps: The probability distribution of each pixel point in the target disparity map under different disparities is obtained; The information entropy of the pixel point is obtained according to the probability distribution through confidence estimation; The information entropy is negated and normalized to obtain the entropy confidence value.

7. The method of claim 2, wherein, The formula used in the confidence modeling mechanism to model the confidence of each pixel point in the target disparity map to obtain the entropy confidence value includes: In the formula, H i,j represents the information entropy of the pixel point (i, j); D max represents the maximum disparity range; P(d|i, j) represents the probability distribution of the pixel point under different disparities d; represents the entropy confidence value; H min represents the minimum information entropy; H max represents the maximum information entropy.

8. The method of claim 1, wherein, According to the target disparity map and the entropy confidence value, a target dense depth map is obtained, which comprises the following steps: The target disparity map is converted into an initial dense depth map through binocular geometric relationship; The initial dense depth map is optimized according to the entropy confidence value to obtain the target dense depth map.

9. A deep learning binocular vision estimation device with fusion edge optimization, characterized in that, It comprises: The correction and preprocessing module is configured to perform image correction and preprocessing on the initial left-eye image and the initial right-eye image to obtain target left-eye images and target right-eye images; The binocular vision estimation module is configured to obtain a target disparity map and an entropy confidence value by using a binocular vision estimation framework according to the target left-eye images and the target right-eye images. The depth map generation module is configured to obtain a target dense depth map according to the target disparity map and the entropy confidence value. The binocular vision estimation framework includes a deep learning feature extraction operation, a semi-supervised mechanism of fusing edge perception and pseudo-label correction, and a confidence modeling mechanism.

10. The apparatus of claim 9, wherein, The binocular vision estimation module includes: The deep learning feature extraction submodule is configured to generate an initial disparity map by using a deep learning feature extraction operation according to the target left-eye images and the target right-eye images. The semi-supervised submodule is configured to optimize the initial disparity map by using a semi-supervised mechanism of fusing edge perception and pseudo-label correction to obtain the target disparity map. The confidence modeling submodule is configured to model the confidence of each pixel point of the target disparity map by using a confidence modeling mechanism to obtain the entropy confidence value.

Images