Stereo matching method and system of binocular camera

Abstract

The invention discloses a stereo matching method and system for a binocular camera. The method comprises the following steps: acquiring an original stereo image to be processed by using the binocular camera; inputting the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model; wherein the stereo matching model is obtained by training sample images based on a pre-constructed deep learning network, and each group of sample images comprises a left view and a right view. The problems that in the prior art, the model volume of a stereo matching network is large, and the redundant calculation amount is large are solved, the model volume of the stereo matching network is reduced, and the parameter amount is reduced under the condition that the prediction effect is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a stereo matching method and system for binocular cameras. Background Art

[0002] The rapid development of artificial intelligence technology and intelligent network technology has brought great convenience to people's transportation. The integration of intelligent network technology and the traditional automobile industry has promoted the rapid progress of active safety performance in the field of intelligent vehicles. The intelligent driving system is mainly composed of three modules: perception, decision-making and control. Among them, the perception module is mainly used to enable the intelligent driving system to obtain and recognize all environmental information such as the distance, speed, direction, traffic signs, etc. of surrounding vehicles or pedestrians. Therefore, as a module that directly connects the intelligent driving system with the driving scene, it is a key module to ensure the safety and stability of intelligent driving. For the intelligent driving system, if the driving scene can be reconstructed in three dimensions through the perception module and its high-precision map can be obtained, it will be of great significance to improve the overall perception ability of the intelligent driving system.

[0003] As people's demand for safer and more convenient travel grows, intelligent driving technology is booming. The ability to perceive and understand the environment is the basis and premise of the automotive intelligent system. Intelligent vehicles collect views through binocular cameras, analyze the surrounding environment after perceiving it, and detect driving conditions by providing information to the control system.

[0004] Binocular stereo matching is an important part of the binocular vision 3D reconstruction task. It obtains disparity by matching pixels between a pair of left and right views, and then recovers depth information from the plane image. Stereo matching ranging using binocular cameras has the advantages of simple implementation, low cost, and the ability to measure distance under non-contact conditions. Although stereo matching has broad application prospects, it is still challenging to ensure high reasoning accuracy and real-time running speed at the same time. The left and right images obtained in reality have problems such as inconsistent lighting, large areas without texture or repeated texture, target occlusion and object transparency, which puts strict requirements on the robustness of stereo matching.

[0005] Currently, data-driven deep neural networks for stereo matching are a hot topic in computer vision, achieving higher accuracy than traditional stereo matching methods. Existing end-to-end stereo matching networks basically use 3D convolution as the only means of cost aggregation, which is essentially a local aggregation method. In order to achieve the desired effect, it is often necessary to stack many convolution layers, which makes the model too large and has redundant calculations.

[0006] In view of this, providing a stereo matching method and system for a binocular camera to reduce the model size of the stereo matching network and reduce the number of parameters while ensuring the prediction effect has become an urgent problem to be solved by those skilled in the art. Summary of the invention

[0007] To this end, the embodiments of the present invention provide a stereo matching method and system for a binocular camera, in order to solve the problems in the prior art that the model volume of the stereo matching network is large and the amount of redundant calculation is large, thereby reducing the model volume of the stereo matching network and reducing the amount of parameters while ensuring the prediction effect.

[0008] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0009] The present invention provides a stereo matching method of a binocular camera, the method comprising:

[0010] Using a binocular camera to collect the original stereo image to be processed;

[0011] Inputting the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model;

[0012] The stereo matching model is obtained by training based on a pre-built deep learning network using sample images, and each group of sample images includes a left eye view and a right eye view.

[0013] In some embodiments, the training process of the stereo matching model includes:

[0014] Construct a dataset containing multiple sample images;

[0015] Inputting each of the sample images in the data set into the deep learning network, and extracting the depth features of the left view and the right view from the sample images;

[0016] Calculate the matching cost of the depth features of the left view and the right view, and construct a cost volume based on the matching cost results;

[0017] The cost volume is aggregated by local costs and propagated by global disparity to obtain an initial disparity map;

[0018] The initial disparity map is optimized by sub-pixel interpolation to obtain a predicted disparity map.

[0019] In some embodiments, the network structure of the pre-built deep learning network includes:

[0020] A preprocessing module, wherein the preprocessing module is used to perform standardization processing and data enhancement on the input sample image to obtain a preprocessed sample image;

[0021] A feature extraction network, wherein the feature extraction network is used to extract the depth features of the left view and the right view from the preprocessed sample image;

[0022] A cost volume construction module, wherein the cost volume construction module is used to calculate the matching cost of the depth features of the left view and the depth features of the right view, and to construct a cost volume according to the matching cost result;

[0023] A cost aggregation module, in which the cost body is subjected to local cost aggregation and global disparity propagation to obtain an initial disparity map;

[0024] A disparity fine-tuning network is used to optimize the initial disparity map through sub-pixel interpolation to obtain a predicted disparity map.

[0025] In some embodiments, the feature extraction network comprises:

[0026] The standard convolutional layer includes a kernel size of , stride length is , the padding width is The convolutional layer;

[0027] Hourglass structure,The hourglass structure consists of multiple downsampling standard convolutional layers and upsampling transposed convolutional layers.

[0028] In some embodiments, the matching cost of the depth features of the left view and the right view is calculated, and a cost volume is constructed according to the matching cost result, specifically including:

[0029] Calculate the absolute difference between the depth feature of the left view and the depth feature of the right view to obtain the feature matching cost when the candidate disparity is 0;

[0030] Keep the depth feature of the left view unchanged and shift the depth feature of the right view to the right by one pixel.

[0031] Calculate the absolute difference between the depth feature of the left view and the depth feature of the translated right view to obtain the feature matching cost when the candidate disparity is 1;

[0032] The target candidate disparity range is set to D, and D feature matching costs can be obtained by analogy. All feature matching costs are residually connected along the disparity dimension to obtain the cost body.

[0033] In some embodiments, the cost aggregation module includes a local cost aggregation module;

[0034] The local cost aggregation module is used to perform local cost aggregation on the cost volume through two 3D convolution structures to obtain a local cost aggregation result.

[0035] In some embodiments, the cost aggregation module further includes an SGM semi-global aggregation layer;

[0036] The SGM semi-global aggregation is used to perform global disparity propagation on the local cost aggregation results, and realize pixel-level regression from the cost body to the disparity through the WTA strategy, replacing the candidate disparity with the minimum matching cost at each pixel position in the cost body as the prediction value to obtain the initial disparity map.

[0037] The present invention also provides a stereo matching system of a binocular camera, the system comprising:

[0038] An image acquisition unit, used for acquiring an original stereo image to be processed using a binocular camera;

[0039] A result generating unit, used for inputting the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model;

[0040] The stereo matching model is obtained by training based on a pre-built deep learning network using sample images, and each group of sample images includes a left eye view and a right eye view.

[0041] The present invention also provides an intelligent terminal, which includes: a data acquisition device, a processor and a memory;

[0042] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method as described above.

[0043] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium contains one or more program instructions, and the one or more program instructions are used to execute the method as described above.

[0044] The stereo matching method and system of the binocular camera provided by the present invention uses the binocular camera to collect the original stereo image to be processed; the original stereo image is input into the pre-trained stereo matching model to obtain the disparity map output by the stereo matching model; wherein the stereo matching model is obtained by training with sample images based on a pre-built deep learning network, and each group of sample images includes a left eye view and a right eye view. The method is based on CNN and semi-global matching algorithms, and the traditional semi-global matching (Semi-Global Matching, SGM) algorithm is integrated into the neural network and participates in the training. The SGM module in the obtained stereo matching model does not require learnable parameters, which ensures the lightweight of the stereo matching network, has a good effect of predicting disparity in real road scenes, and has a small number of parameters. Thus, the problem of large model volume and large redundant calculation amount of the stereo matching network in the prior art is solved, the model volume of the stereo matching network is reduced, and the number of parameters is reduced while ensuring the prediction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0046] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0047] Figure 1 This is a flow chart of a stereo matching method for a binocular camera provided by the present invention;

[0048] Figure 2 The second flowchart of the stereo matching method of the binocular camera provided by the present invention;

[0049] FIG3 is one of the network structure diagrams of the deep learning network provided by the present invention;

[0050] FIG4 is a second network structure diagram of the deep learning network provided by the present invention;

[0051] Figure 5 This is a structural block diagram of the stereo matching system of the binocular camera provided by the present invention. DETAILED DESCRIPTION

[0052] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] First, the terms involved in the present invention are explained as follows:

[0054] Neural network: also known as artificial neural network, is a mathematical model based on the basic principles of neural networks in biology. After understanding the structure of the human brain and the response mechanism of external stimuli, it uses network topology knowledge as the theoretical basis to simulate the human brain's nervous system's processing mechanism of complex information.

[0055] Disparity map: Disparity is the difference in direction when observing the same target from two points at a certain distance. The disparity map is a two-dimensional image that stores the disparity values ​​of all pixels in a single view after stereo correction. In a binocular vision system, taking the disparity map of the left view as an example, the disparity value at the pixel position p is equal to the column coordinates of the matching point of the pixel on the right image minus the column coordinates of the pixel on the left image.

[0056] Stereo matching: Stereo matching is a technology that recovers depth information from planar images. The binocular stereo matching system simulates the principle of human visual perception, installs two digital cameras on the same horizontal line and undergoes stereo correction. The disparity map is obtained by matching the target points between the stereo image pairs, and then the depth information can be obtained.

[0057] Cost volume: The matching cost is a measure of the difference between the pixel to be matched and the candidate pixel. The smaller the cost, the greater the correlation and the greater the probability of being a homonymous point. Before searching for homonymous points, each pixel often specifies a disparity search range D, and uses a high-dimensional matrix C of size W×H×D (W is the image width, H is the image height) to store the matching cost value of each pixel at each disparity. This matrix is ​​called a cost volume.

[0058] Cost aggregation: Cost aggregation is a method of associating the matching cost of each pixel with the pixels around it, including local aggregation and (semi) global aggregation; the common method of local aggregation is to weight the sum of the matching costs under the same disparity in a window of fixed size. (Semi) global aggregation is essentially the propagation of disparity, allowing the disparity of areas with high signal-to-noise ratio to propagate to areas with low signal-to-noise ratio, so that the cost of all points can better represent the true correlation.

[0059] In order to solve the problems existing in the prior art, the stereo matching method of binocular cameras provided by the present invention is a stereo matching method combining CNN local filtering and semi-global matching. According to tests, the disparity map predicted by this method has good effect, the model volume is small, and it is applicable to various complex road scenes.

[0060] In general, the stereo matching method of the binocular camera provided by the present invention first standardizes the binocular stereo images to the same domain, uses data enhancement to enrich the trainable samples of various scenes, and enhances the generalization performance of the network; uses a feature extraction network (Feature extractor) with shared parameters for left and right views to extract depth features, and downsamples the image to 1 / 3 size; then performs a simple arithmetic difference on the left and right view features to calculate the matching cost and construct a 4D cost body; then the cost body is sequentially passed through stacked 3D convolution modules and SGM layers, and residual connections between multiple 3D convolution layers are used to achieve local cost aggregation. The SGM layer realizes global disparity propagation and predicts the initial disparity map at the pixel level; finally, the sub-pixel interpolation of the disparity is completed with the help of a refinement network, and the geometric structure of small objects is restored to output the final predicted disparity map.

[0061] Please refer to Figure 1 , Figure 1 This is one of the flow charts of the stereo matching method of the binocular camera provided by the present invention.

[0062] In a specific implementation, the stereo matching method of a binocular camera provided by the present invention comprises the following steps:

[0063] S110: using a binocular camera to collect an original stereo image to be processed;

[0064] S120: inputting the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model;

[0065] The stereo matching model is obtained by training based on a pre-built deep learning network using sample images, and each group of sample images includes a left eye view and a right eye view.

[0066] In some embodiments, Figure 2 As shown, the training process of the stereo matching model includes the following steps:

[0067] S210: Construct a dataset containing multiple sample images. In a specific usage scenario, a stereo matching dataset is selected. In the process of dataset division, SceneFlow is a large synthetic dataset generated by a virtual engine, which is used to pre-train a deep stereo matching network. The dataset contains 35,454 pairs of training images and 4,370 pairs of test images. The resolution of all images is 960×540, and all training images provide dense disparity maps as true values. Use the real-world driving scene dataset KITTI. The KITTI dataset is a dataset for research in the field of autonomous driving jointly sponsored by the Karlsruhe Institute of Technology in Germany and the Chicago Campus of Toyota Technological University, including KITTI2012 and KITTI2015. KITTI2012 contains 194 training image pairs with true values ​​taken from lidar and 195 test image pairs without true values. The resolution of all images is 370×1226. KITTI2015 contains 200 training image pairs and 200 test image pairs. The resolution of all images is 375×1242. Compared with KITTI2012, its scenes are more complex and the tasks are more challenging. This embodiment divides the training images of each data set into a training set and a test set at a ratio of 8:2, and after pre-training on the SceneFlow data set to obtain a converged model, it is migrated to the KITTI data set to learn road scenes.

[0068] S220: Inputting each of the sample images in the data set into the deep learning network, and extracting the depth features of the left view and the right view from the sample images;

[0069] S230: Calculate the matching cost of the depth features of the left view and the right view, and construct a cost volume according to the matching cost result;

[0070] S240: The cost volume is aggregated by local costs and propagated by global disparity to obtain an initial disparity map;

[0071] S250: Optimizing the initial disparity map by sub-pixel interpolation to obtain a predicted disparity map.

[0072] During the training process, network parameters need to be set, including the number of training rounds, batch size, learning rate, learning rate momentum, weight decay coefficient, SGM layer penalty coefficient, and loss component coefficient, etc. In addition, there are parameters for data preprocessing, including random cropping size, image hue, scaling factor, horizontal flip rate, etc.

[0073] Input the training set data for forward propagation, and calculate the total loss based on the prediction results. The total loss is the weighted sum of the local aggregation prediction loss, the semi-global aggregation prediction loss, and the fine-tuning network prediction loss. Then, backpropagate based on the total loss, and optimize and update the parameters based on the accumulated gradients. After each round of training, evaluate the validation set once with the latest model, and retain the evaluation results and model parameters.

[0074] During the training process, the smooth L1 loss function is used to calculate the three losses separately:

[0075]

[0076] Final total loss:

[0077]

[0078]

[0079] in, is the predicted disparity after local cost aggregation, is the predicted disparity output by the SGM layer, is the disparity predicted by the disparity fine-tuning network, is the true disparity of the training image. represents the rounding function, since and They are all pixel-level predictions, so the corresponding supervisory signals need to be constrained to the pixel level to ensure stable training of the network. , and They are the weight coefficients corresponding to the three losses in the total loss.

[0080] For the SceneFlow dataset, this embodiment uses the evaluation indicator end-point-error (EPE) to evaluate the model performance. The specific expression is:

[0081]

[0082] in, is the disparity map predicted by the model, Represents the pixel position, there are a total of pixels, and EPE reflects the absolute error of the disparity map prediction.

[0083] For the KITTI2012 dataset, in addition to using EPE to evaluate model performance, the evaluation indicators are also used Pixel Error Rate ( ):

[0084]

[0085] in, Represents an indicator function, which counts as 1 if the condition is met and counts as 0 if the condition is not met. Indicates that the error between the predicted disparity map and the true value is greater than The pixel ratio of each unit reflects the relative error of the disparity map prediction.

[0086] For the KITTI2015 dataset, the evaluation indicator divergence rate (D1) is used to evaluate the model performance. The specific expression is:

[0087]

[0088]

[0089] It represents the ratio of outliers in the predicted disparity map, reflecting the relative error of the disparity map prediction. When all pixels in the true value are considered, the evaluation index is -all; when only the pixels of the foreground target are considered, the evaluation index is -fg.

[0090] The smaller the evaluation indicators are, the more accurate the disparity prediction is.

[0091] In some embodiments, Figure 3 As shown in the figure, the network structure of the pre-built deep learning network includes five parts, namely, a preprocessing module, a feature extraction network, a cost volume construction module, a cost aggregation module and a disparity fine-tuning network.

[0092] The preprocessing module is used to perform standardization and data enhancement on the input sample image to obtain the preprocessed sample image. In a specific usage scenario, the preprocessing module consists of a standardization module and a data enhancement module; the standardization module standardizes the input RGB image according to the standard mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225) of ImageNet to ensure that the input image is in the normalized domain space; then the image enhancement module is used to randomly crop, adjust the hue, flip and rotate the standardized image to generate a new stereo image and image truth value for the input of the feature extraction network.

[0093] The feature extraction network is used to extract the depth features of the left view and the right view from the preprocessed sample image; the feature extraction network includes a standard convolution layer and an hourglass structure, and the standard convolution layer includes a kernel size of , stride length is , the padding width is The hourglass structure includes multiple downsampling standard convolution layers and upsampling transposed convolution layers.

[0094] More specifically, the feature extraction network uses the backbone of GANet, and the left and right inputs share the network weights. The basic modules used by this network are as follows:

[0095] Standard convolutional layer: The standard convolutional layer includes a kernel size of k×k, a stride of s, and a padding width of Convolution, batch normalization, ReLU activation function. The input image is passed through in sequence to obtain the output features;

[0096] Hourglass structure: The hourglass structure consists of multiple down-sampling standard convolutions and up-sampling transposed convolutions. The input size is The features of are sequentially passed through 4 standard convolutional layers with a kernel size of 3 and a stride of 2. The output feature sizes after downsampling in each layer are , , , . Then, after four transposed convolutions, the output size is , , , The upsampled features of the same size are connected through concat residuals to reduce the loss of underlying information. A complete hourglass structure is finally output. Top-level features of the size.

[0097] like Figure 3 As shown, the size after preprocessing is For a stereo image pair, the underlying features are first obtained through the initial convolution block in the feature extraction network. The convolution block consists of three layers of standard convolutions with kernel sizes of 3, 5, and 3 and strides of 1, 3, and 1, respectively. The output underlying features are Then two stacked hourglass structures are used to obtain a larger receptive field and more context information. The same-size down-sampled and up-sampled features between the two hourglass structures are also connected using concat residuals, and the final output is The top-level features of are used to construct the cost volume.

[0098] The cost volume construction module is used to calculate the matching cost of the depth features of the left view and the right view, and construct the cost volume according to the matching cost result; the process of constructing the cost volume includes the following steps:

[0099] Calculate the absolute difference between the depth feature of the left view and the depth feature of the right view to obtain the feature matching cost when the candidate disparity is 0;

[0100] Keep the depth feature of the left view unchanged and shift the depth feature of the right view to the right by one pixel.

[0101] Calculate the absolute difference between the depth feature of the left view and the depth feature of the translated right view to obtain the feature matching cost when the candidate disparity is 1;

[0102] The target candidate disparity range is set to D, and D feature matching costs can be obtained by analogy. All feature matching costs are residually connected along the disparity dimension to obtain the cost body.

[0103] Specifically, when the cost volume is constructed, the left and right feature maps obtained by feature extraction have the same size , and do the arithmetic absolute difference between the two to get the matching cost when the candidate disparity is 0. Keep the left feature unchanged, translate the right feature to the right by one pixel, and then do the arithmetic absolute difference between the two to get the cost when the candidate disparity is 1. Assuming that the candidate disparity range is D, D feature matching costs can be obtained by analogy. Concat all costs along the disparity dimension to get the shape The cost body constructed in this way does not occupy much memory, is very effective and conforms to the traditional sense of the cost body, and is suitable for the subsequent SGM semi-global cost aggregation.

[0104] The cost body is subjected to local cost aggregation and global disparity propagation in the cost aggregation module to obtain an initial disparity map. The cost aggregation module includes a local cost aggregation module and the cost aggregation module also includes an SGM semi-global aggregation layer; the local cost aggregation module is used to perform local cost aggregation on the cost body through two 3D convolution structures to obtain a local cost aggregation result. The SGM semi-global aggregation is used to perform global disparity propagation on the local cost aggregation result, and realize pixel-level regression from the cost body to the disparity through the WTA strategy, replacing the candidate disparity with the minimum matching cost at each pixel position in the cost body as the prediction value to obtain the initial disparity map.

[0105] like Figure 4 As shown in Figure 1, the cost aggregation module is divided into two parts: local cost aggregation and SGM semi-global aggregation. The calculation process is:

[0106] The cost volume generated in the previous step is input into two basic 3D convolution structures. The basic structure is two layers of stacked 3D standard convolution with batch normalization and ReLU activation function, with a kernel size of 3×3×3 and a stride of 1. The number of channels and the size of the cost volume are not changed. The two basic 3D convolution structures are connected by concat residuals.

[0107] The locally aggregated cost volume is input into the SGM layer for global disparity propagation. The SGM layer uses the left view to adaptively adjust the penalty coefficient:

[0108]

[0109]

[0110] Among them, R, G, and B are the three channels of the original image, and the weighted average of each channel can be converted into a grayscale image I. is the corresponding pixel position where the penalty coefficient needs to be calculated, Pixel Neighborhood of Is a neighborhood The next pixel position along the inner propagation path, and are the pixel values ​​at the corresponding positions on the grayscale image, It is a fixed value set artificially, representing the initial penalty coefficient. In pixels Adaptive penalty coefficient. Using adaptive penalty coefficient can make SGM algorithm more robust to various complex scenes. Then, the WTA strategy is used inside the SGM layer to realize pixel-level regression from cost volume to disparity, that is, replace the candidate disparity with the minimum matching cost at each pixel position in the cost volume as the predicted value. Finally, the disparity map is restored to the original view size using "trilinear interpolation". , get the initial disparity map .

[0111] The disparity fine-tuning network is used to optimize the initial disparity map by sub-pixel interpolation to obtain a predicted disparity map. Specifically, the disparity fine-tuning network consists of 6 dilated convolutional layers with a kernel size of 3×3 and expansion rates of 1, 2, 4, 8, 1, 1. The large receptive field is used to capture the geometric structure of the internal object, and the small receptive field is used to restore the sub-pixel disparity value. The left view is spliced ​​as input, and the above-mentioned hole convolution layers are passed in sequence to output the predicted disparity residual. The initial disparity map is added to the disparity residual to obtain the final predicted disparity map. .

[0112] The stereo matching method of binocular cameras provided by the present invention has the following technical effects:

[0113] With the help of perturbation technique, the SGM algorithm is integrated into the CNN stereo matching model in a fully differentiable way.

[0114] The existing end-to-end stereo matching network basically uses 3D convolution as the only means of cost aggregation, which is essentially a local cost aggregation. With the help of image filters with different weights and different receptive fields, the cost body is only locally aggregated, which can be calculated at a faster speed. However, in reality, in order to achieve the ideal effect, it is often necessary to stack many 3D convolution layers, which is not conducive to efficient operation. The traditional global optimization algorithm is very complex to calculate and occupies a large amount of memory. Therefore, the present invention uses the optimization algorithm in semi-global stereo matching (SGM) to make up for the shortcomings of local cost aggregation.

[0115] SGM can globally optimize the cost volume and effectively avoid the redundant calculation caused by the continuous accumulation of receptive fields by local cost aggregation to obtain context information, but its calculation process is not differentiable and cannot be directly connected to the end-to-end stereo matching network. We found that by introducing a new perturbation interpolation mathematical technique, the predicted discrete disparity output from the SGM layer can realize the gradient backpropagation of this layer and ensure the end-to-end training of the network.

[0116] This technique is described as follows: Let the linear optimization layer Receive continuous cost volume from upstream neural network , and returns the discrete solution ,Then Minimization solution of the layer To calculate the loss . Due to the optimization layer About A piecewise constant function of , so the gradient is almost zero everywhere, but with the help of a hyperparameter representing the interpolation range Continuous interpolation can obtain continuous gradients. In this process, the optimization process needs to be repeated when calculating the reverse gradient, but only needs to be calculated once during inference testing, so it will not affect the inference time.

[0117] Substitute the SGM algorithm into the linear optimization layer , we can get the back propagation gradient of discrete disparity prediction to cost volume according to the above formula, and realize fully differentiable training. It is worth noting that The output of the layer does not have to be used directly to calculate the loss. It can also be connected to other downstream neural networks to achieve differentiable training. The pixel-level disparity map output by the Combine Net model in the present invention at the SGM layer is used as input to the disparity fine-tuning network.

[0118] Also with the help of interpolation perturbation technology, the WTA strategy is used to predict the disparity map and achieve differentiable training.

[0119] In the previous stereo matching network, the cost volume obtained through a series of cost aggregation modules will be regressed through the soft-argmax function to obtain a continuous disparity map:

[0120]

[0121] in, is each candidate disparity The price, represents the softmax function, is the maximum candidate disparity, is the predicted disparity.

[0122] However, this method has its drawbacks: the cost volume is inverted as the disparity distribution at each pixel, which is a discrete categorical distribution, while soft-argmax uses the mean of this distribution for estimation. When the predicted distribution is multi-peaked, the mean may deviate from the peak area and incorrectly predict a low-probability value, which is more obvious when multiple peaks are far apart. In addition, the average value of the disparity distribution is by no means physically equivalent to the true disparity value.

[0123] In the present invention, the WTA strategy, i.e., the argmin operation of the hard threshold, is adopted to obtain the disparity map from the cost volume regression. The WTA strategy can perfectly avoid the shortcomings of the existing stereo matching network in disparity regression and achieve accurate disparity prediction. Although the argmin operation cannot be directly derived, the above-mentioned interpolation perturbation technology can also be used to substitute the argmin optimization function into the linear optimization layer. , obtain the reverse gradient from the disparity map to the cost volume to ensure the differentiable training of the network.

[0124] Thus, the stereo matching method of the binocular camera provided by the present invention uses the binocular camera to collect the original stereo image to be processed; the original stereo image is input into the pre-trained stereo matching model to obtain the disparity map output by the stereo matching model; wherein the stereo matching model is obtained by training with sample images based on a pre-built deep learning network, and each group of sample images includes a left eye view and a right eye view. The method is based on CNN and semi-global matching algorithms, and the traditional semi-global matching (Semi-Global Matching, SGM) algorithm is integrated into the neural network and participates in the training. The SGM module in the obtained stereo matching model does not require learnable parameters, which ensures the lightweight of the stereo matching network, has a good effect of predicting disparity in real road scenes, and has a small amount of parameters. Thus, the problem of large model volume and large amount of redundant calculation of the stereo matching network in the prior art is solved, the model volume of the stereo matching network is reduced, and the amount of parameters is reduced while ensuring the prediction effect.

[0125] In addition to the above method, the present invention also provides a stereo matching system of a binocular camera, such as Figure 5 As shown, the system comprises:

[0126] An image acquisition unit 510 is used to acquire an original stereo image to be processed using a binocular camera;

[0127] A result generating unit 520 is used to input the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model;

[0128] The stereo matching model is obtained by training based on a pre-built deep learning network using sample images, and each group of sample images includes a left eye view and a right eye view.

[0129] In some embodiments, the training process of the stereo matching model includes:

[0130] Construct a dataset containing multiple sample images;

[0131] Inputting each of the sample images in the data set into the deep learning network, and extracting the depth features of the left view and the right view from the sample images;

[0132] Calculate the matching cost of the depth features of the left view and the right view, and construct a cost volume based on the matching cost results;

[0133] The cost volume is aggregated by local costs and propagated by global disparity to obtain an initial disparity map;

[0134] The initial disparity map is optimized by sub-pixel interpolation to obtain a predicted disparity map.

[0135] In some embodiments, the network structure of the pre-built deep learning network includes:

[0136] A preprocessing module, wherein the preprocessing module is used to perform standardization processing and data enhancement on the input sample image to obtain a preprocessed sample image;

[0137] A feature extraction network, wherein the feature extraction network is used to extract the depth features of the left view and the right view from the preprocessed sample image;

[0138] A cost volume construction module, wherein the cost volume construction module is used to calculate the matching cost of the depth features of the left view and the depth features of the right view, and to construct a cost volume according to the matching cost result;

[0139] A cost aggregation module, in which the cost body is subjected to local cost aggregation and global disparity propagation to obtain an initial disparity map;

[0140] A disparity fine-tuning network is used to optimize the initial disparity map through sub-pixel interpolation to obtain a predicted disparity map.

[0141] In some embodiments, the feature extraction network comprises:

[0142] The standard convolutional layer includes a kernel size of , stride length is , the padding width is The convolutional layer;

[0143] Hourglass structure,The hourglass structure consists of multiple downsampling standard convolutional layers and upsampling transposed convolutional layers.

[0144] In some embodiments, the matching cost of the depth features of the left view and the right view is calculated, and a cost volume is constructed according to the matching cost result, specifically including:

[0145] Calculate the absolute difference between the depth feature of the left view and the depth feature of the right view to obtain the feature matching cost when the candidate disparity is 0;

[0146] Keep the depth feature of the left view unchanged and shift the depth feature of the right view to the right by one pixel.

[0147] Calculate the absolute difference between the depth feature of the left view and the depth feature of the translated right view to obtain the feature matching cost when the candidate disparity is 1;

[0148] The target candidate disparity range is set to D, and D feature matching costs can be obtained by analogy. All feature matching costs are residually connected along the disparity dimension to obtain the cost body.

[0149] In some embodiments, the cost aggregation module includes a local cost aggregation module;

[0150] The local cost aggregation module is used to perform local cost aggregation on the cost volume through two 3D convolution structures to obtain a local cost aggregation result.

[0151] In some embodiments, the cost aggregation module further includes an SGM semi-global aggregation layer;

[0152] The SGM semi-global aggregation is used to perform global disparity propagation on the local cost aggregation results, and realize pixel-level regression from the cost body to the disparity through the WTA strategy, replacing the candidate disparity with the minimum matching cost at each pixel position in the cost body as the prediction value to obtain the initial disparity map.

[0153] In the above specific implementation, the stereo matching system of the binocular camera provided by the present invention uses the binocular camera to collect the original stereo image to be processed; the original stereo image is input into the pre-trained stereo matching model to obtain the disparity map output by the stereo matching model; wherein the stereo matching model is based on a pre-built deep learning network and is obtained by training using sample images, and each group of sample images includes a left eye view and a right eye view. The system is based on CNN and semi-global matching algorithms, and the traditional semi-global matching (Semi-Global Matching, SGM) algorithm is integrated into the neural network and participates in the training. The SGM module in the obtained stereo matching model does not require learnable parameters, which ensures the lightweight of the stereo matching network, has a good effect of predicting disparity in real road scenes, and has a small number of parameters. Thus, the problem of large model volume and large redundant calculation amount of the stereo matching network in the prior art is solved, the model volume of the stereo matching network is reduced, and the number of parameters is reduced while ensuring the prediction effect.

[0154] The present invention also provides an intelligent terminal, which includes: a data acquisition device, a processor and a memory;

[0155] The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to execute the method as described above.

[0156] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer-readable storage medium, which contains one or more program instructions. The one or more program instructions are used to be executed by a binocular camera depth calibration system as described above.

[0157] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0158] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0159] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0160] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0161] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0162] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0163] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0164] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A stereo matching method for binocular cameras, characterized in that: The method comprises: Using a binocular camera to collect the original stereo image to be processed; Inputting the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model; The stereo matching model is obtained by training based on a pre-built deep learning network using sample images, and each group of sample images includes a left eye view and a right eye view.

2. The stereo matching method of binocular cameras according to claim 1, characterized in that: The training process of the stereo matching model includes: Construct a dataset containing multiple sample images; Inputting each of the sample images in the data set into the deep learning network, and extracting the depth features of the left view and the right view from the sample images; Calculate the matching cost of the depth features of the left view and the right view, and construct a cost volume based on the matching cost results; The cost volume is aggregated by local costs and propagated by global disparity to obtain an initial disparity map; The initial disparity map is optimized by sub-pixel interpolation to obtain a predicted disparity map.

3. The stereo matching method of binocular cameras according to claim 2, characterized in that: The network structures of the pre-built deep learning networks include: A preprocessing module, wherein the preprocessing module is used to perform standardization processing and data enhancement on the input sample image to obtain a preprocessed sample image; A feature extraction network, wherein the feature extraction network is used to extract the depth features of the left view and the right view from the preprocessed sample image; A cost volume construction module, wherein the cost volume construction module is used to calculate the matching cost of the depth features of the left view and the depth features of the right view, and to construct a cost volume according to the matching cost result; A cost aggregation module, in which the cost body is subjected to local cost aggregation and global disparity propagation to obtain an initial disparity map; A disparity fine-tuning network is used to optimize the initial disparity map through sub-pixel interpolation to obtain a predicted disparity map.

4. The stereo matching method of binocular cameras according to claim 3, characterized in that: The feature extraction network includes: The standard convolutional layer includes a kernel size of , stride is s, padding width is The convolutional layer; Hourglass structure,The hourglass structure consists of multiple downsampling standard convolutional layers and upsampling transposed convolutional layers.

5. The stereo matching method of binocular cameras according to claim 3, characterized in that: Calculate the matching cost of the depth features of the left view and the right view, and construct a cost volume based on the matching cost results, including: Calculate the absolute difference between the depth feature of the left view and the depth feature of the right view to obtain the feature matching cost when the candidate disparity is 0; Keep the depth feature of the left view unchanged and shift the depth feature of the right view to the right by one pixel. Calculate the absolute difference between the depth feature of the left view and the depth feature of the translated right view to obtain the feature matching cost when the candidate disparity is 1; The target candidate disparity range is set to D, and D feature matching costs can be obtained by analogy. All feature matching costs are residually connected along the disparity dimension to obtain the cost body.

6. The stereo matching method of binocular cameras according to claim 3, characterized in that: The cost aggregation module includes a local cost aggregation module; The local cost aggregation module is used to perform local cost aggregation on the cost volume through two 3D convolution structures to obtain a local cost aggregation result.

7. The stereo matching method of binocular cameras according to claim 6, characterized in that: The cost aggregation module also includes an SGM semi-global aggregation layer; The SGM semi-global aggregation is used to perform global disparity propagation on the local cost aggregation results, and realize pixel-level regression from the cost body to the disparity through the WTA strategy, replacing the candidate disparity with the minimum matching cost at each pixel position in the cost body as the prediction value to obtain the initial disparity map.

8. A stereo matching system for a binocular camera, characterized in that: The system comprises: An image acquisition unit, used for acquiring an original stereo image to be processed using a binocular camera; A result generating unit, used for inputting the original stereo image into a pre-trained stereo matching model to obtain a disparity map output by the stereo matching model; The stereo matching model is obtained by training based on a pre-built deep learning network using sample images, and each group of sample images includes a left eye view and a right eye view.

9. An intelligent terminal, characterized in that: The intelligent terminal includes: a data acquisition device, a processor and a memory; The data acquisition device is used to acquire data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes one or more program instructions, and the one or more program instructions are used to execute the method according to any one of claims 1-7.

Images