Knowledge embedding-based binocular vision depth estimation method and system

Abstract

The invention discloses a binocular vision depth estimation method and system based on knowledge embedding, and the method comprises the steps: collecting data through employing a binocular camera and rotating laser scanning, carrying out the preprocessing of the data, constructing a data set, and dividing the data set into a training set and a test set. Secondly, constructing a binocular vision depth estimation model based on a convolutional neural network, and constructing a physical knowledge embedding loss function according to the relationship between the real parallax and the predicted parallax; and then training the binocular vision depth estimation model based on physical knowledge embedding by using the training data set. And finally, using the trained binocular vision depth model to predict the stereo image of the test set to obtain a disparity map which is consistent with the original image in size and conforms to the physical characteristics of the scene, and completing depth estimation. According to the method, the disparity map conforming to the physical characteristics of the scene can be obtained, and the precision of disparity estimation and the generalization performance of the model are improved.

Description

TECHNICAL FIELD

[0001] The application relates to the fields of computer vision, machine learning, and the like, and proposes a binocular vision depth estimation method and system based on knowledge embedding. BACKGROUND

[0002] Depth estimation is one of the core topics in the field of computer vision. With the rapid development of deep learning technology, data-driven depth estimation methods have made breakthrough progress and achieved remarkable results in the fields of autonomous driving, intelligent monitoring, and three-dimensional reconstruction. Such methods learn the mapping relationship from pixels to depth through large-scale labeled data-driven neural networks, and exhibit excellent prediction capability in structured roads, indoor scenes, and other standardized environments. However, when the model is migrated to other scenes, its insufficient generalization becomes a key bottleneck restricting practical application.

[0003] For example, in the complex background of industrial scenes, the edge blur phenomenon caused by densely arranged industrial components, and the lack of local features caused by low-texture materials, which destroys the spatial continuity. Although existing improvement schemes have improved the robustness of the model by introducing attention mechanisms, multi-scale feature fusion, and other strategies, there is still an inherent contradiction between precision and efficiency. Especially in industrial automation scenarios that require real-time response, the high computational overhead of complex models and the precision loss in edge blur and low-texture areas form a dilemma.

[0004] To address the problems of complex background, edge blur, and low texture, this paper proposes a binocular vision depth estimation method based on knowledge embedding for industrial scenes to improve the robustness and efficiency of the depth estimation model. SUMMARY

[0005] To solve the above technical problems, the application improves the depth estimation accuracy and model generalization ability, and proposes a binocular vision depth estimation method and system based on knowledge embedding.

[0006] A binocular vision depth estimation method based on knowledge embedding, the overall process of which includes the following steps:

[0007] S1. Use a binocular camera and a rotating laser scanner to collect data, preprocess the data, construct a data set, and divide the training set and the test set.

[0008] S2. Based on a convolutional neural network, a binocular vision depth estimation model is constructed.

[0009] S3. According to the relationship between the true disparity and the predicted disparity, a physical knowledge embedding loss function is constructed to incorporate the physical prior knowledge of left and right errors and structural differences into the model training process.

[0010] S4. Using the training data set, the binocular vision depth estimation model is trained based on the physical knowledge embedding, the model hyperparameters are adjusted until the trained model reaches the required accuracy.

[0011] S5. Using the trained binocular vision depth model to predict the stereo images of the test set, obtaining the disparity map consistent with the scene physical characteristics of the original picture size, completing the depth estimation.

[0012] The above-mentioned binocular vision depth estimation method based on knowledge embedding, the specific process of constructing the training data set for preprocessing and dividing the training set and the test set in step S1 is as follows:

[0013] Using a binocular camera to collect the left and right views of the camera, and recording the true depth of the left and right views by a rotating laser scanner to construct a binocular depth data set, and then using the intrinsic parameters of the binocular camera, including translation vector, rotation matrix, distortion coefficient, rotation matrix, baseline distance and projection matrix to correct the collected left and right views.

[0014] The above-mentioned binocular vision depth estimation method based on knowledge embedding, the specific structure of the binocular vision depth estimation model constructed in step S2 is as follows:

[0015] The structure of the binocular depth estimation neural network includes a residual convolution feature encoder, a context encoder, a correlation pyramid and a multi-level gate control loop unit. The residual convolution feature encoder is composed of a convolution layer, an instance normalization layer, a ReLU activation function layer, two residual blocks and a 1x1 convolution layer; the context encoder is composed of a convolution layer, a batch normalization layer, a GeLU activation function layer, two residual blocks and a 1x1 convolution layer; the correlation pyramid is a stack of several layers of average pooling; each gate control loop unit in the multi-level gate control loop unit is a multi-layer original Figure 1 / 4, 1 / 8 and 1 / 32 pixel size feature values cross-connect and interact with information, where 1 / 4 pixel size is used to obtain the final disparity, and the remaining two are to capture global features.

[0016] The binocular vision depth estimation model is implemented as follows:

[0017] S2.1 The input of the binocular vision depth estimation model is a pair of corrected left and right views of the test set, which are input to the residual convolution feature encoder to extract the features of the left and right views and obtain the dense feature maps of the left and right views. At the same time, the left view is input to the context encoder to capture global context features.

[0018] S2.2 The dense feature maps of the left and right views are input to the pyramid structure to construct a three-dimensional correlation volume. The dot product of the points with the same y coordinate is calculated by the following formula to calculate the dot product similarity of the left view features and the right view features:

[0019]

[0020] In the formula, C xyz represents the dot product similarity matrix, f l-xyD represents a point in the left feature map with dimension index D, row index x, and column index y, f r-xzD represents a point in the right feature map with dimension index D, row index x, and possible column index z considered when calculating the correlation. Then, the calculated three-dimensional correlation volume is average-pooled, and a 4-layer three-dimensional correlation volume is output.

[0021] S2.3 The context features and the 4-layer correlation volume are input into a multi-level gated recurrent unit to iteratively obtain the final disparity field. Then, the disparity map output by the model is upsampled by 9x9 convex combination to obtain a disparity map consistent with the resolution of the original image.

[0022] The above-mentioned knowledge embedding-based binocular vision depth estimation method, according to the relationship between the true disparity and the predicted disparity, constructs a physical knowledge embedding loss function, and integrates the physical prior knowledge of left and right errors and structural differences into the model training process. The specific process is as follows:

[0023] S3.1 The binocular vision depth estimation model loss function consists of a true prediction error term and a physical prior knowledge constraint term, and the calculation formula is as follows:

[0024]

[0025] In the formula, α is an adaptive weight, L1 represents the true prediction error, L lr and L struct is the physical prior knowledge constraint, and L t is the final loss.

[0026] S3.2 The calculation formula of the adaptive weight α in the true prediction error term is as follows:

[0027]

[0028] Where n represents the total number of dimensions, and i represents the dimension.

[0029] The calculation formula of the true prediction error L1 is as follows:

[0030]

[0031] In the formula, d pred (x) represents the predicted disparity of point x in the left view, and d left (x) represents the true disparity of point x in the left view.

[0032] S3.3 The physical prior knowledge constraint term is composed of left-right disparity consistency loss L lr and disparity structure consistency loss L struct .

[0033] The prediction consistency of the real left-right disparity map corresponding points is used to ensure the consistency of the depth prediction of the left and right views in the physical space, and the left-right disparity consistency loss L lr obtained by low-dimensional mapping is as follows:

[0034]

[0035] In the formula, d pred (x) is the predicted disparity of point x in the left view, d right (x-d pred (x)) represents the real disparity of the right view point position x-d pred (x) corresponding to point x.

[0036] The structure consistency of the real disparity and the predicted disparity emphasizes the global spatial continuity of the disparity map, and the disparity structure consistency loss L struct obtained by low-dimensional mapping is as follows:

[0037]

[0038] In the formula, and are the horizontal and vertical gradients of the predicted disparity of the left view point (x, y), and are the horizontal and vertical gradients of the real disparity of the left view point (x, y).

[0039] In another aspect of the present application, a binocular vision depth estimation system based on knowledge embedding is also provided, which comprises a data acquisition module, a model main structure module, a model training module, a depth estimation module and a result display module, and the specific contents are as follows:

[0040] The data acquisition module is used to acquire left and right view data and real depth of left and right views through a binocular camera and a rotating laser scanner respectively, to obtain a corrected data set after preprocessing, and to divide a training set and a test set.

[0041] The model structure module is based on a convolutional neural network and constructs a binocular vision depth estimation model.

[0042] The model training module constructs a physical knowledge embedding loss function according to the relationship between the real disparity and the predicted disparity, uses a training data set, and trains the binocular vision depth estimation model based on physical knowledge embedding.

[0043] The depth estimation module is configured to perform depth estimation on the preprocessed left and right images in the test set by using the trained binocular vision depth estimation model to output a depth map.

[0044] The result display module is configured to display the corrected left and right images and the corresponding depth map through a display.

[0045] The effect of the present application is that:

[0046] 1. A depth estimation neural network is constructed, which is based on a residual convolution combined with a context encoder as a starting pyramid structure and a multi-level gate control loop unit as the main body, balancing the precision and recognition speed, and realizing the overall process of depth estimation based on binocular vision.

[0047] 2. A physical constraint knowledge embedding method based on left and right disparity consistency and disparity structure consistency is proposed, which focuses on solving the problem of local feature loss caused by edge blur and low texture material, so that the model can enhance the disparity matching result in the iterative training process.

[0048] 3. The knowledge embedding method is quantitatively analyzed on the dataset. Experimental results show that the model enhances the understanding ability of subtle differences between binocular images, and the depth estimation accuracy is improved to a certain extent. Compared with the original model, the endpoint error is reduced by 3% to 5%, and the comprehensive mismatch rate is reduced by 5% to 8%. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more intuitively understand the technical solutions described in this paper, the following will briefly introduce the drawings in the examples. The features and advantages of the technical solutions can be more clearly understood by referring to the drawings, which are schematic and should not be understood as any limitation on this paper. Those skilled in the art can obtain other drawings from these drawings without creative labor. Among them:

[0050] Figure 1 is the overall process of the binocular vision depth estimation method based on knowledge embedding proposed in this paper;

[0051] Figure 2 is the overall framework of the binocular vision depth estimation model based on knowledge embedding proposed in this paper;

[0052] Figure 3 is the structure of the residual convolution feature encoder and the context encoder in the model;

[0053] Figure 4 is the structure of the multi-level gate control loop unit in the model;

[0054] Figure 5 is the decline curve of the evaluation results in the training process;

[0055] Figure 6is to initialize the parameters of the trained model to preview and compare the details of the prediction results. DETAILED DESCRIPTION

[0056] In order to enable a clearer understanding of the technical solutions herein, the following further describes the technical solutions herein in conjunction with the accompanying drawings and specific examples.

[0057] In the following description, the knowledge embedding-based binocular vision depth estimation method proposed herein is set forth, but the technical solutions herein can also be implemented in other ways different from those described herein, and therefore the scope of protection of the technical solutions herein is not limited by the specific examples disclosed below.

[0058] Figure 1 The overall flow of the knowledge embedding-based binocular vision depth estimation method proposed herein is shown, which includes the following steps:

[0059] S1. Use binocular cameras and rotating laser scanners to collect data, pre-process the data, construct a data set, and divide the training set and the test set;

[0060] S2. Based on a convolutional neural network, a binocular vision depth estimation model is constructed;

[0061] S3. According to the relationship between the real disparity and the predicted disparity, a physical knowledge embedding loss function is constructed, and the physical prior knowledge of left-right error and structural difference is integrated into the model training process;

[0062] S4. Using the training data set, the binocular vision depth estimation model is trained based on the physical knowledge embedding, the model hyperparameters are adjusted, and the trained model reaches the required accuracy;

[0063] S5. Using the trained binocular vision depth model to predict the stereo images of the test set, obtaining a disparity map consistent with the scene physical characteristics with the same size as the original image, completing the depth estimation.

[0064] The knowledge embedding-based binocular vision depth estimation method, the model construction and training example is as follows:

[0065] S1. Use binocular cameras and rotating laser scanners to collect data, pre-process the data, construct a data set, and divide the training set and the test set

[0066] S1.1 Use binocular cameras to collect left and right views of the cameras, and record the real depth of the left and right views by rotating laser scanners. The point cloud density of the depth is about 30% of the image pixels, and a total of 200 pairs of data are collected. Using the intrinsic parameters of the binocular camera, including translation vector, rotation matrix, distortion coefficient, rotation matrix, baseline distance and projection matrix, the left and right views collected are corrected, and the correction formula is as follows:

[0067] T new =R rect T T new ≈[B,0,0] T (1)

[0068] In the formula, T represents the translation vector of the right camera to the left camera, R rect represents the common rotation matrix, and B represents the baseline distance of the camera.

[0069] R1=R rect R2=R rect R T (2)

[0070] In the formula, R T represents the transpose of the rotation matrix of the right camera to the left camera, and R1 and R2 represent the rotation matrices of the left and right cameras respectively.

[0071] P1=K new [R1|0] (3)

[0072] In the formula, P1 represents the projection matrix of the left camera, K new represents the common intrinsic parameter of the camera.

[0073]

[0074] In the formula, u1 v1 and u2 v2 are the corresponding points of x1 y1 and x2 y2 on the left and right views after correction, and z1 and z2 are both equal to 1.

[0075] After obtaining the preprocessed data set, the data is divided into a training set and a test set according to a ratio of 6:4, wherein the training set is used to train the model and adjust the parameters, and the test set is used to evaluate the prediction ability of the model.

[0076] S2. Constructing a binocular vision depth estimation model

[0077] The connection between the modules of the binocular vision depth estimation model constructed by the method is shown in Figure 2 The model structure includes a residual convolution feature encoder, a context encoder, a correlation pyramid, and a multi-level gate control loop unit. The structures of the residual convolution feature encoder and the context encoder are as shown in Figure 3As shown, the left figure is a residual convolutional feature encoder, and the right figure is a context encoder. The residual convolutional feature encoder is composed of a 7x7 convolutional layer, an instance normalization layer, a ReLU activation function layer, two residual blocks, and a 1x1 convolutional layer. The context encoder is composed of a 7x7 convolutional layer, a batch normalization layer, a GeLU activation function layer, two residual blocks, and a 1x1 convolutional layer. The correlation pyramid is a stack of four layers of 2-step 2-pixel-size average pooling. Each gate control cycle unit in the multi-level gate control cycle unit is a multi-layer original Figure 1 The structure of a single gate control unit is as shown in Figure 4 As shown in the figure, Δd represents the parallax increment. According to d t+1 = Δd + d t , d t+1 is obtained, and the final parallax is obtained through step-by-step iteration output.

[0078] S3. Constructing a loss function

[0079] The loss function of the binocular vision depth estimation model described in S3.1 is composed of a real prediction error term and a physical prior knowledge constraint term, and the calculation formula is as follows:

[0080]

[0081] In the formula, α is an adaptive weight, L1 represents the real prediction error, L lr and L struct are physical prior knowledge constraints, and L t is the final loss.

[0082] S3.2 The calculation formula of the real prediction error L1 in the real prediction error term is as follows:

[0083]

[0084] In the formula, d pred (x) represents the predicted parallax of point x in the left view, and d left (x) represents the real parallax of point x in the left view.

[0085] The calculation formula of the adaptive weight α is as follows:

[0086]

[0087] Where n represents the total number of dimensions, and i represents the current dimension.

[0088] S3.3 The physical prior knowledge constraint term is composed of two parts: the left-right parallax consistency loss L lr and the parallax structure consistency loss L struct .

[0089] The prediction consistency of real left and right disparity map corresponding points is used to ensure the consistency of depth prediction of left and right views in physical space, and the left and right disparity consistency loss L obtained by low-dimensional mapping is as follows: lr The calculation formula is as follows:

[0090]

[0091] In the formula, d pred (x) is the predicted disparity of point x in the left view, d right (x-d pred (x)) represents the real disparity of the right view point position x-d pred (x) corresponding to point x.

[0092] The structure consistency of real disparity and predicted disparity emphasizes the global spatial continuity of the disparity map, and the disparity structure consistency loss L obtained by low-dimensional mapping is as follows: struct The calculation formula is as follows:

[0093]

[0094] In the formula, and are the horizontal and vertical gradients of the predicted disparity of the left view point (x, y), and are the horizontal and vertical gradients of the real disparity of the left view point (x, y).

[0095] S4 model initialization and model training

[0096] In the overall training process, the model learning rate needs to be initialized to 0.001, the total training round is 300, the size of each batch is 8, the model learning rate scheduler is AdamW optimizer, and the data iteration update times of the multi-level gate control cycle unit in a single round are 8, 16 and 24 respectively, and the variable pair is constructed whether to perform knowledge embedding, that is, according to the above loss function, the physical priori knowledge is embedded into the binocular depth estimation model for restriction, the collected training set is used to train the model, and a trained binocular depth estimation model capable of outputting a disparity map conforming to physical characteristics is obtained.

[0097] The end-point error (End-Point Error, EPE) and the comprehensive mismatch rate (D1-all) are used as evaluation indexes, and the results of 1 pixel precision and 5 pixel precision are given, wherein the end-point error directly calculates the average absolute error of the predicted disparity and the real disparity, and reflects the absolute value of the overall error, and the calculation formula is as follows:

[0098]

[0099] In the formula, dpred (x) represents the true disparity of point x. gt (x) represents the true disparity of point x.

[0100] The comprehensive false matching rate emphasizes the reliability of obstacle detection in practical applications, and it counts the proportion of pixels whose disparity error exceeds 3 pixel accuracy or exceeds 5% of the true disparity value among all pixels, and the calculation formula is as follows:

[0101]

[0102] In the formula, δ is an indicator function, which is 1 when the condition is met, and 0 when it is not met, and when |d pred (x) - d gt (x) | > 3, that is, the absolute error exceeds 3 pixels or |d pred (x) - d gt (x) | > 0.05·d gt (x), that is, the absolute error exceeds 5%. The calculation formulas of 1 pixel accuracy and 5 pixel accuracy are as follows:

[0103]

[0104] In the formula, d pred (x) represents the true disparity of point x. gt (x) represents the true disparity of point x, and δ is an indicator function.

[0105] The evaluation results of the test set during the model training process are shown in Table 1. Figure 5 The figure shows the binocular depth estimation model, where 8, 16 and 24 represent the number of data iteration updates in a single round of multi-level gate control loop unit, and KE represents whether knowledge embedding is performed, such as stereo 8 -KE represents a binocular depth estimation model with knowledge embedding and a single round of multi-level gate control loop unit iteration update number of 8.

[0106] The final evaluation specific data is shown in Table 1.

[0107] Table 1: Comparison of binocular depth estimation model evaluation before and after different parameter knowledge embedding

[0108]

[0109] The binocular vision depth estimation model prediction process of S5 is as follows:

[0110] S5.1 The input of the binocular vision depth estimation model is a pair of corrected test set left and right images with size HxWx3, which are input to the residual convolution feature encoder to extract the features of the left and right images to obtain the dense feature map f l ∈RH / 4×W / 4×256 and the dense feature map f of the right view r ∈R H / 4×W / 4×256 Simultaneously, the left view is input into the context encoder to capture global contextual features c∈R. H / 4×W / 4×256 .

[0111] S5.2 The dense feature maps of the left and right images are input into a pyramid structure to construct a 3D related volume. The dot product similarity between the features of the left and right images is calculated by performing a dot product operation on points at the same y-coordinate using the following formula:

[0112]

[0113] In the formula, C xyz Let f represent the dot product similarity matrix. l-xyD This represents a point in the feature map of the left image with dimension index D, and row and column indices x and y, respectively. r-xzD This represents the dimension index D, row index x, and possible column indices z in the right-hand feature map, considered when calculating correlation. Then, average pooling is applied to the calculated 3D correlation volume to output a 4-layer 3D correlation volume C0∈R. H / 4×W / 4×D / 4 C1∈R H / 4×W / 4×D / 8 C2∈R H / 4×W / 4×D / 16 C3∈R H / 4×W / 4×D / 32 .

[0114] S5.3 inputs the context features and the relevant volumes of the four layers into the multi-level gated loop unit, and based on the disparity increment Δd output by the gate control unit, according to d t+1 =Δd+d t Get d t+1 The final disparity is obtained through iterative steps. Then, the disparity map output by the model is input into the upsampling module, and after 9×9 convex combination upsampling, a disparity map with the same resolution as the original image is obtained. An example of the output result of the above model is shown below. Figure 6 As shown.

[0115] In this example, stereo depth estimation models were constructed based on different update counts of multi-level gated loop units, and the variable of whether or not knowledge embedding was performed was controlled. The model with 8 updates had the worst evaluation metric, while the model with 24 updates had the best. After knowledge embedding, the evaluation metric was significantly better than that without. Figure 5 We can see the changes in the evaluation results during the training of the six models. Figure 6 The differences in prediction results among the six models are shown. The model with the best prediction results is the one that is trained with knowledge embedding and a multi-level gated recurrent unit that updates 24 times per round.

[0116] Another aspect of the present application also provides a binocular vision depth estimation system based on knowledge embedding, comprising a data acquisition module, a model main structure module, a model training module, a depth estimation module and a result display module, specifically as follows:

[0117] The data acquisition module is used for collecting left and right view data and real depth of left and right views through binocular cameras and rotating laser scanning respectively, pre-processing to obtain corrected data set, and dividing training set and test set.

[0118] The model structure module is based on a convolutional neural network to construct a binocular vision depth estimation model, and the proposed structure can better capture global scene structure information and increase the accuracy of disparity estimation by preserving features at different scales.

[0119] The model training module is used for constructing a physical knowledge embedding loss function according to the relationship between real disparity and predicted disparity, using the training data set, and training the binocular vision depth estimation model based on physical knowledge embedding, which can detect and correct the wrong disparity value caused by light change and texture loss, alleviate the problem of local feature loss caused by edge blur, and improve the accuracy of disparity estimation.

[0120] The depth estimation module is used for depth estimation of the pre-processed left and right images in the test set by the trained binocular vision depth estimation model to output a depth map.

[0121] The result display module is used for displaying the corrected left and right images and the corresponding depth map through a display.

Claims

1. A binocular vision depth estimation method based on knowledge embedding, characterized in that, Includes the following steps: S1. Collect data using a binocular camera and rotating laser scanning, preprocess the data, construct a dataset, and divide it into training and test sets; S2. Construct a binocular vision depth estimation model based on convolutional neural networks; S3. Construct a physical knowledge embedding loss function based on the relationship between the actual disparity and the predicted disparity; S4. Using the training dataset, train the binocular vision depth estimation model based on physical knowledge embedding; S5. Use the trained binocular vision depth model to predict the stereo images in the test set, obtain a disparity map that is the same size as the original image and conforms to the physical characteristics of the scene, and complete the depth estimation.

2. The knowledge-embedded binocular vision depth estimation method according to claim 1, characterized in that, The process of acquiring and preprocessing data using a binocular camera and a rotating laser scanner specifically includes: acquiring the left and right views of the camera using a binocular camera, while simultaneously recording the true depth of the left and right views using a rotating laser scanner to construct a binocular depth dataset; and then using binocular camera intrinsic parameters, including translation vector, rotation matrix, distortion coefficient, baseline distance, and projection matrix, to correct the acquired left and right views.

3. The knowledge-embedded binocular vision depth estimation method according to claim 1, characterized in that, The specific structure of the binocular vision depth estimation model is as follows: The structure of the binocular depth estimation neural network includes a residual convolutional feature encoder, a context encoder, a correlation pyramid, and a multi-level gated recurrent unit. The residual convolutional feature encoder consists of a convolutional layer, an instance normalization layer, a ReLU activation function layer, two residual blocks, and a cascaded 1×1 convolutional layer. The context encoder consists of a convolutional layer, a batch normalization layer, a GeLU activation function layer, two residual blocks, and a 1×1 convolutional layer. The correlation pyramid is a stack of several layers of average pooling. Each gated recurrent unit in the multi-level gated recurrent unit is a structure in which multiple layers of pixel-sized feature values ​​of 1 / 4, 1 / 8, and 1 / 32 of the original image are cross-connected and interact with each other.

4. The knowledge-embedded binocular vision depth estimation method according to claim 2, characterized in that, The specific implementation process of the binocular vision depth estimation model is as follows: The input to the S2.1 binocular vision depth estimation model is a pair of calibrated test set left and right views, which are fed into two residual convolutional feature encoders to extract features from the left and right images to obtain dense feature maps of the left and right views; at the same time, the left view is fed into the context encoder to capture global context features. S2.2 Dense feature maps of the left and right views are input into a pyramid structure to construct a 3D related volume; the dot product similarity between the features of the left and right images is calculated by performing a dot product operation on points at the same y-coordinate. In the formula, f l-xyD This represents a point in the feature map of the left image with dimension index D, and row and column indices x and y, respectively. r-xzD This indicates that the dimension index is D, the row index is x, and the column index z in the right figure is considered when calculating the correlation. Then, the calculated 3D correlation volume is averaged and pooled to output a 4-layer 3D correlation volume. S2.3 inputs the context features and the relevant volume of the 4 layers into a multi-level gated recurrent unit and iterates to obtain the final disparity field; then the disparity map output by the model is obtained and upsampled by convex combination to obtain a disparity map with the same resolution as the original image.

5. The knowledge-embedded binocular vision depth estimation method according to claim 4, characterized in that, The specific implementation process of step S3 is as follows: The loss function of the binocular vision depth estimation model described in S3.1 consists of two parts: the true prediction error term and the physical prior knowledge constraint term. In the formula, α is the adaptive weight, L1 represents the true prediction error, and L... lr and L struct To constrain error based on prior physical knowledge, L t The total loss; The formula for calculating the adaptive weight α in the true prediction error term S3.2 is as follows: Where n represents the total number of dimensions and i represents the dimension; the formula for calculating the true prediction error L1 is as follows: In the formula d pred (x) represents the predicted disparity of point x in the left view, d left (x) represents the true disparity of point x in the left view; S3.3 The physical prior knowledge constraint term is determined by the left-right parallax consistency loss L. lr and disparity structure consistency loss L struct It consists of two parts; Based on the physical prior knowledge that the prediction consistency of corresponding points in the actual left and right disparity maps is used to ensure the consistency of depth prediction in physical space, a left and right disparity consistency loss L is constructed. lr This corrects erroneous parallax values ​​caused by changes in lighting or missing textures. The calculation formula is as follows: In the formula, d pred (x) is the predicted disparity of point x in the left view, d right (xd pred (x) represents the position xd of the right view point corresponding to point x. pred The true parallax of (x); The structural consistency between true and predicted disparity emphasizes the global spatial continuity of the disparity map. The disparity structural consistency loss L obtained through mapping... struct The calculation formula is as follows: In the formula, and These are the horizontal and vertical gradients of the predicted disparity for the left view point (x, y), respectively. and These are the horizontal and vertical gradients of the true disparity of the left view point (x, y), respectively.

6. A knowledge-embedded binocular vision depth estimation system, used to implement the knowledge-embedded binocular vision depth estimation method according to any one of claims 1 to 5, characterized in that, It includes a data acquisition module, a model main structure module, a model training module, a depth estimation module, and a results display module. Data acquisition module: used to acquire left and right view data and the actual depth of left and right views through binocular cameras and rotating laser scanning, respectively, perform preprocessing to obtain the corrected dataset, and divide it into training set and test set; Model structure module: Based on convolutional neural networks, a binocular vision depth estimation model is constructed; Model training module: Based on the relationship between the actual disparity and the predicted disparity, a physical knowledge embedding loss function is constructed, and the binocular vision depth estimation model is trained using the training dataset based on the physical knowledge embedding. Depth estimation module: Used to estimate the depth of the preprocessed left and right images in the test set and output depth maps using a trained binocular vision depth estimation model; Results display module: Used to display the corrected left and right images and corresponding depth maps on the monitor.

Images