Stereoscopic matching method and device in stereoscopic vision

Abstract

The invention provides a stereo matching method and device in stereo vision, and the method comprises the steps: obtaining a left view and a right view collected by a camera; performing feature extraction on the left view and the right view through a sub-network, and constructing a fusion cost space of the sub-network; obtaining a sub-network disparity map through the fusion cost space; and filtering the main network through the sub-network disparity map to obtain a target disparity map. According to the scheme provided by the invention, the stereo matching efficiency and accuracy can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of control, and in particular to a stereo matching method and device in stereo vision, especially binocular stereo vision. Background Art

[0002] The main goal of stereo computer vision is to simulate the function of the human eye through the collaborative work of software and hardware, and to make depth predictions based on the collected two-dimensional images, so as to achieve accurate restoration of three-dimensional scenes. In order to achieve this goal, stereo vision technology has been widely developed and applied in various fields. Stereo vision technology is usually divided into binocular vision and multi-viewing vision. Among them, binocular stereo vision is more widely used because it is simpler and more efficient. The existing binocular stereo vision technology is mainly used in many fields such as autonomous driving, three-dimensional reconstruction, remote sensing mapping, and medical testing.

[0003] Stereo matching is a key component in stereo vision systems, and has a significant impact on the accuracy and speed of 3D reconstruction. Stereo matching is to restore the depth information of a scene by analyzing multiple planar images. The specific method is to find matching point pairs in these images and use the triangulation principle to calculate the depth of their corresponding physical points in space. In order to achieve stereo matching, image processing, signal processing, and optics are generally involved.

[0004] Although existing stereo matching has made some progress, there are still some problems when facing complex scenes: in areas lacking obvious textures, it is difficult for the matching algorithm to find reliable matching points; image differences under different lighting conditions will lead to reduced matching accuracy; noise in the image will interfere with the performance of the matching algorithm; in areas with repeated textures, the matching algorithm is prone to false matches. Summary of the invention

[0005] The main purpose of the present invention is to overcome the defects of the above-mentioned related technologies and provide a stereo matching method and device in stereo vision to solve the technical problem of low matching accuracy in stereo matching in the related technologies, which is particularly suitable for binocular stereo vision.

[0006] On one hand, the present invention provides a stereo matching method in stereo vision, comprising:

[0007] Get the left view and right view captured by the camera;

[0008] Extracting features of the left view and the right view through a sub-network, and constructing a fusion cost space of the sub-network;

[0009] Obtaining a sub-network disparity map through the fusion cost space;

[0010] The main network is filtered through the sub-network disparity map to obtain a target disparity map.

[0011] In some implementations, extracting features from the left view and the right view through a sub-network and constructing a fusion cost space of the sub-network includes:

[0012] Extracting structural features of the left view and the right view through a feature extraction module in the sub-network;

[0013] Construct a structural cost space according to the extracted structural features, and construct a convolution cost space according to the convolution method;

[0014] The features of different dimensions in the structural cost space and the convolutional cost space are grouped and mapped according to the channel dimension, and then spliced ​​to obtain the fused cost space.

[0015] In some implementations, extracting structural features from the left view and the right view by a feature extraction module in a sub-network includes:

[0016] Taking the left view and the right view as target images;

[0017] Obtain feature values ​​of feature points of the target image within a 3x3 window;

[0018] Taking the current feature point as the central feature point, comparing the central feature point with the eight surrounding feature points to generate an 8-bit binary string corresponding to the current feature point;

[0019] Convert the binary string into unsigned 8-bit integer data, and use the 8-bit unsigned integer data as the structural feature value of the current feature point;

[0020] Using the word embedding method, the structural feature values ​​of each feature point are converted into feature vectors that can be learned by the neural network, and the similarity between two structural features is represented by the cosine value between the feature vectors.

[0021] In some implementations, the current feature point is used as the central feature point, and the central feature point is compared with the surrounding eight feature points to generate an 8-bit binary string corresponding to the current feature point, including:

[0022] An 8-bit binary string corresponding to the current feature point is generated according to the following formula:

[0023]

[0024] Among them, C(x,y) is a bit string, indicating the structural relationship between two pixels, I(x,y) indicates the eigenvalue of the central feature point (x,y), I(x+i,y+j) indicates the eigenvalue of the points around the central feature point, n′ is the largest integer not greater than half of n (window length), n is the length of the window, m′ is the largest integer not greater than half of m (window height), m is the height of the window, is the bit-by-bit concatenation operation, ξ thr is the learnable threshold.

[0025] In some implementations, the sub-network disparity map is obtained by the fusion cost space:

[0026] Through the Hadamard product operation, the fusion cost space and the structure cost space are downsampled multiple times to achieve multiple filtering to obtain a sub-network disparity map.

[0027] In some embodiments, the main network is a deep convolutional stereo matching network, and accordingly, filtering the main network through the sub-network disparity map to obtain a target disparity map includes:

[0028] In the feature extraction stage, the deep convolutional stereo matching network obtains features extracted by the sub-network from the left view and the right view;

[0029] The deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view by combining the features extracted from the left view and the right view by the sub-network through the spatial pyramid module;

[0030] The deep convolutional stereo matching network performs filtering through the sub-network disparity map in the cost aggregation stage to obtain a target disparity map.

[0031] In some embodiments, the deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view through a spatial pyramid module in combination with features extracted from the left view and the right view by a sub-network, including:

[0032] The deep convolutional stereo matching network uses four-scale average pooling layers to compress the feature data of the left view and the right view into multiple dimensions to obtain multiple levels of context information;

[0033] Through convolution and upsampling, multiple levels of context information are converted to the same resolution;

[0034] The context information of multiple levels converted to the same resolution is concatenated and aggregated to obtain context information of multiple scales.

[0035] Another aspect of the present invention provides a stereo matching device in stereo vision, comprising:

[0036] An acquisition unit, used for acquiring a left view and a right view captured by a camera;

[0037] An extraction unit, used to extract features of the left view and the right view through a sub-network, and construct a fusion cost space of the sub-network;

[0038] A first generating unit, configured to obtain a sub-network disparity map through the fusion cost space;

[0039] The second generating unit is used to filter the main network through the sub-network disparity map to obtain a target disparity map.

[0040] Another aspect of the present invention provides a storage medium having a computer program stored thereon, wherein the program implements the steps of any of the aforementioned methods when executed by a processor.

[0041] In yet another aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0042] According to the technical solution of the present invention, on the basis of the main network, a sub-network is introduced to optimize the stereo matching process, reducing the computational complexity of the network. Furthermore, through the use of multi-level feature fusion and deep prior information, the algorithm can achieve faster running speed while maintaining high precision, thereby effectively improving the efficiency and accuracy of stereo matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 is a method schematic diagram of an embodiment of a stereo matching method in stereo vision provided by the present invention;

[0045] Figure 2 is a flow chart of an embodiment of a stereo matching algorithm provided by the present invention;

[0046] Figure 3 It is a flow chart of the method for extracting structural features provided by the present invention;

[0047] Figure 4 It is a flow chart of constructing the sub-network cost space provided by the present invention;

[0048] Figure 5 It is a flow chart of the method for sub-network cost aggregation provided by the present invention;

[0049] Figure 6 It is an overall logic schematic diagram of the stereo matching provided by the present invention;

[0050] Figure 7 It is a logical schematic diagram of feature extraction in a subnetwork provided by the present invention;

[0051] Figure 8 It is a structural block diagram of an embodiment of a stereo matching device in stereo vision provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] In view of the problems of accuracy and low efficiency of existing stereo matching, in this example, it is considered that a priori network, that is, a sub-network can be introduced for processing. Specifically, a structural feature extraction module can be set up, through which rich structural information can be extracted from the input image. Combining the structural feature extraction module with local features and global features can improve the ability to capture image structural information. Especially when processing images with weak textures and complex structures, the structural feature extraction module can provide more accurate feature descriptions, thereby improving the accuracy and robustness of stereo matching.

[0055] Furthermore, by extracting deep prior information through a lightweight prior network, the network structure is more compact, occupies less memory, and has a faster computing speed. By combining deep prior information with multi-feature prior information, the algorithm's computing efficiency can be improved while maintaining high accuracy. The lightweight prior network can not only reduce computational complexity, but also improve the algorithm's adaptability in various complex scenarios.

[0056] The present invention provides a stereo matching method in stereo vision, which is particularly suitable for binocular stereo vision technology.

[0057] Figure 1 It is a method schematic diagram of an embodiment of a stereo matching method in stereo vision provided by the present invention.

[0058] like Figure 1 As shown, according to an embodiment of the present invention, the stereo matching method in stereo vision at least includes step S110, step S120, step S130 and step S140.

[0059] Step S110, obtaining a left view and a right view captured by a camera;

[0060] The camera preferably adopts a binocular camera, that is, images captured by the left and right cameras of the binocular camera are obtained respectively.

[0061] Step S120, extracting features of the left view and the right view through a sub-network, and constructing a fusion cost space of the sub-network;

[0062] Specifically, the left view and the right view can be convolved before feature extraction to obtain a left feature map and a right feature map. After obtaining the left feature map and the right feature map, after extracting the structural features according to the preset process, the structural feature space can be constructed, and the convolution feature space can be constructed according to the classical convolution method. Finally, the fusion feature space is constructed by the group correlation method. The feature space refers to putting features of different dimensions together according to a certain method, so that subsequent calculations can make full use of multi-dimensional information. In this example, the splicing method is used when constructing the structural feature and convolution feature space, that is, the features of different dimensions are spliced ​​according to the channel dimension. When constructing the fusion space, the group correlation method is used to group and map the features of different dimensions according to the channel dimension and then splice them.

[0063] In a specific implementation, extracting features from the left view and the right view through a sub-network and constructing a fusion cost space of the sub-network may include:

[0064] S1: extracting structural features of the left view and the right view through a feature extraction module in a sub-network;

[0065] S2: construct the structural cost space according to the extracted structural features, and construct the convolution cost space according to the convolution method;

[0066] S3: After the features of different dimensions in the structural cost space and the convolutional cost space are grouped and mapped according to the channel dimension, they are concatenated to obtain the fused cost space.

[0067] When the feature extraction is actually performed, it can be performed in the following manner, that is, the structural feature extraction of the left view and the right view is performed by the feature extraction module in the sub-network, which may include:

[0068] S1: taking the left view and the right view as target images;

[0069] S2: Obtain feature values ​​of feature points of the target image in a 3x3 window;

[0070] S3: taking the current feature point as the central feature point, and comparing the central feature point with the surrounding eight feature points to generate an 8-bit binary string corresponding to the current feature point;

[0071] S4: converting the binary string into unsigned 8-bit integer data, and using the 8-bit unsigned integer data as the structural feature value of the current feature point;

[0072] S5: Using the word embedding method, the structural feature values ​​of each feature point are converted into feature vectors that can be learned by the neural network, and the cosine value between the feature vectors is used to represent the similarity between two structural features.

[0073] For example, an 8-bit binary string corresponding to the current feature point can be generated according to the following formula:

[0074]

[0075] Among them, C(x,y) is a bit string, indicating the structural relationship between two pixels, I(x,y) indicates the eigenvalue of the central feature point (x,y), I(x+i,y+j) indicates the eigenvalue of the points around the central feature point, n′ is the largest integer not greater than half of n (window length), n is the length of the window, m′ is the largest integer not greater than half of m (window height), m is the height of the window, is the bit-by-bit concatenation operation, ξ thr is the learnable threshold.

[0076] Step S130, obtaining a sub-network disparity map through the fusion cost space;

[0077] Specifically, the fusion cost space and the structure cost space may be downsampled multiple times through a Hadamard product operation to achieve multiple filtering to obtain a sub-network disparity map.

[0078] Among them, the Hadamard product refers to the bitwise multiplication of each data of the matrix, which can be understood as a filtering. Because the purpose of the sub-network is to output the result simply and quickly, the convolution feature map in the feature extraction stage can be used as the filtering basis to perform multiple fast filtering on the cost space. In this way, a relatively ideal feature map can be obtained quickly and efficiently without the need to train the network multiple times.

[0079] Step S140, filtering the main network through the sub-network disparity map to obtain a target disparity map.

[0080] The above-mentioned main network is a deep convolutional stereo matching network, and the main network may include: feature extraction, cost space construction, cost aggregation, and generation of disparity maps. In a specific implementation, the deep convolutional stereo matching network can use a spatial pyramid module to extract multi-dimensional features in the feature extraction stage. At the same time, at this stage, the feature map extracted by the sub-network features can be fused. In this way, the connection between the two networks can be enhanced to prevent the network model from not fitting during training. Furthermore, the output of the sub-network can be used to filter the main network. The filtering method is the same as that in the sub-network aggregation stage and is also implemented through the Hadamard product, so that the sub-network can be used to guide the main network to output the disparity map. Because the cost space in the deep convolutional network is often constructed abnormally bloated and contains a large amount of redundant information, using a sub-network for filtering can reduce the impact of redundant information on the network.

[0081] In a specific embodiment, filtering the main network through the sub-network disparity map to obtain a target disparity map can include: in the feature extraction stage, the deep convolutional stereo matching network obtains the features extracted by the sub-network from the left view and the right view; the deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view through a spatial pyramid module in combination with the features extracted by the sub-network from the left view and the right view; in the cost aggregation stage, the deep convolutional stereo matching network filters the sub-network disparity map to obtain a target disparity map.

[0082] The above-mentioned deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view through a spatial pyramid module in combination with the features extracted from the left view and the right view by the sub-network, and may include: the deep convolutional stereo matching network uses an average pooling layer of four scales to compress the feature data of the left view and the right view into multiple dimensions to obtain context information of multiple levels; converts the context information of multiple levels to the same resolution through convolution and upsampling processing; and splices and aggregates the context information of multiple levels converted to the same resolution to obtain context information of multiple scales.

[0083] In the above example, based on the main network, a sub-network is introduced to optimize the stereo matching process, reducing the computational complexity of the network. Furthermore, through the use of multi-level feature fusion and deep prior information, the algorithm can achieve faster operation speed while maintaining high accuracy, thereby effectively improving the efficiency and accuracy of stereo matching. The above solution improves the accuracy and robustness of the stereo matching algorithm in various complex scenarios.

[0084] The above method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0085] like Figure 2 The figure shows the flow chart of the stereo matching algorithm provided in this example. The stereo matching algorithm may include: structural feature extraction, cost volume construction, cost aggregation and disparity calculation. Among them, the structural feature refers to the relationship between the pixel point (feature point) and its surrounding pixel points. For the image, these are pixel points; in the internal operation process of the neural network, they are called feature points.

[0086] Among them, structural feature extraction can be as follows Figure 3 As shown, the following steps are included:

[0087] S1: Get the feature values ​​of the feature points in the 3x3 window;

[0088] For example, a 3x3 window can be created that contains 9 feature points.

[0089] S2: Compare the current feature point (central feature point) with the surrounding 8 feature points in turn according to the rules, and generate an 8-bit binary string through this comparison;

[0090] S3: Convert the binary string into unsigned 8-bit integer data, and use the 8-bit unsigned integer data as the structural feature value of the current feature point;

[0091] That is, the structural features of the feature points are divided into 256 cases ranging from 0 to 255.

[0092] S4: Converted into a feature vector that can be automatically learned using the word embedding method, and the vector cosine value is used to represent the structural feature similarity between two points.

[0093] Specifically, the above structural feature values ​​can be converted into feature vectors learned by the neural network through the word embedding method, and then the cosine value between the feature vectors is used to represent the similarity between the two structural features for network calculation.

[0094] In this example, the word embedding method converts discrete data into vectors, so that the similarity between the data can be determined. For example, if you need to predict the similarity between the two words "he" and "man", you can first randomly assign a feature vector to each of the two words, and then adjust the two vectors through continuous training. After the training is completed, the cosine value between the two vectors represents the similarity between the two words. By applying this method to the image field, the similarity between discrete local features can be calculated.

[0095] like Figure 4 The figure shows the construction flow chart of the sub-network cost space. The left and right feature maps are the left and right images after convolution processing. Figure 3 After extracting the structural features in the process shown, the structural feature space can be constructed, and the convolution feature space can be constructed according to the classical convolution method. Finally, the fusion feature space can be constructed by the group correlation method. The feature space refers to putting features of different dimensions together in a certain way so that subsequent calculations can make full use of multi-dimensional information. In this example, the splicing method is used when constructing the structural features and convolution feature space, that is, the features of different dimensions are spliced ​​according to the channel dimension. When constructing the fusion space, the group correlation method is used to group and map the features of different dimensions according to the channel dimension and then splice them.

[0096] like Figure 5 The figure shows a flow chart of the method for sub-network cost aggregation. The Hadamard product is calculated for the fusion cost space and the structural cost space. The Hadamard product refers to the bitwise multiplication of each data in the matrix, which can be understood as a single filter. Because the purpose of the sub-network is to output the result simply and quickly, the convolution features 1 to 4 in the feature extraction stage can be used as the basis for filtering, and the cost space can be filtered quickly multiple times. In this way, a relatively ideal feature map can be obtained quickly and efficiently without the need to train the network multiple times. However, it is difficult to improve the accuracy of the final feature map after multiple trainings.

[0097] Among them, downsampling feature space refers to downsampling the cost space, which can be understood as downsampling a picture with a resolution of 1000*1000 to a picture of 500*500. When the picture is 2-dimensional, the cost space is 4-dimensional, and the dimensions of the feature space include C*D*H*W, where H and W are the height and width of the downsampling. Because the corresponding feature maps H and W are the same as the HW of the corresponding feature space, different feature spaces are filtered through convolutional features 1 to 4, which are suitable for Hadamard product operations. The final output feature map is the final output of the sub-network. The purpose of the output of the sub-network is to filter the main network.

[0098] Specifically, the main network is a deep convolutional stereo matching network, including: feature extraction, cost space construction, cost aggregation, and disparity map generation. Figure 6 As shown in the figure, in the feature extraction stage, the deep convolutional stereo matching network of this example uses a spatial pyramid module to extract multi-dimensional features. At the same time, at this stage, the feature map extracted by the sub-network feature is fused. In this way, the connection between the two networks can be enhanced to prevent the network model from not fitting during training. Furthermore, the output of the sub-network is used to filter the main network. The filtering method is the same as that in the sub-network aggregation stage and is also implemented through the Hadamard product, so that the sub-network can be used to guide the main network to output the disparity map. Because the cost space in the deep convolutional network is often constructed abnormally bloated and contains a large amount of redundant information, using the sub-network for filtering can reduce the impact of redundant information on the network.

[0099] like Figure 6 As shown, a pair of rectified images (images acquired by the left and right cameras, i.e., the left image and the right image) can be input into the multi-feature prior network to obtain the output of the multi-feature prior. In the feature extraction stage, feature extraction is performed through the pre-trained twin network model to obtain multiple features. When constructing the cost space, the structural cost space and the convolutional cost space are constructed separately. Then, the combined cost space is input into the cost space regularization stage of the multi-feature prior network. The cost space is filtered multiple times and at multiple scales using the multiple features extracted previously to generate a low-resolution feature map. Then, the generated low-resolution feature map is input into the disparity tuning network. In the background network, the fully relevant cost space is first constructed using the convolutional network, and then the cost space is filtered using the output of the multi-feature prior. Finally, after cost aggregation, the final disparity map is generated.

[0100] Specifically, when extracting structural features, a learnable structural feature extraction module is used to extract structural features using a window (3x3) with the same convolution size as that used by the neural network, wherein the extraction rule can be expressed as:

[0101]

[0102] Among them, C(x,y) is a bit string that represents the structural relationship between two pixels, I(x,y) represents the eigenvalue of the center point (x,y), and I(x+i,y+j) represents the eigenvalue of the points around the center point.

[0103] This formula is used to compare the feature values ​​of all points in a window of size 2nx2m with the center point of the window. If the feature value of a point is greater than the feature value of the center point × thr (threshold), the point is set to 1, otherwise it is set to 0. Finally, these 1s and 0s are concatenated in sequence into a string.

[0104] Where n' is the largest integer not greater than half of n (window length), m' is the largest integer not greater than half of m (window height), I is the grayscale of the pixel or the eigenvalue of the feature map, is the bit-by-bit concatenation operation, ξ thr is the learnable threshold, ξ thr It can be defined as follows:

[0105]

[0106] Among them, thr represents the set threshold, which can be learned by the network itself, and y×thr represents the characteristic value of the center point×threshold.

[0107] Specifically, in the feature extraction stage, the feature information of the feature points in a 3×3 window is extracted with the point to be extracted as the center. Then, the feature information of these local feature points is compared with the feature information of the center point. The correlation and relative position between each feature point in the window and the center point are determined by the transformation rules specified by the above formula. In this way, the spatial relationship between feature points and the interaction between features can be established. Then, the binary encoding form of these structural features is mapped to an eight-bit unsigned integer to represent the structural feature value of the center point of the window:

[0108]

[0109] In order to effectively represent and identify features in subsequent image processing and analysis, and to apply them in the cost space construction step, the structural feature extraction operator can customize the size like the convolution kernel and can implement the same method of expanding the receptive field as the "hole convolution", making it easy to integrate into the existing neural network.

[0110] For the sub-network, in the feature extraction module, the input image can be downsampled to 1 / 32 of the original size using MobileNetV2 pre-trained on ImageNet, and then, as Figure 7As shown in the figure, through multiple upsampling blocks with residual connections, the feature map is restored to 1 / 4 of its original size to obtain a multi-scale feature map. Then, the extracted convolution feature map is subjected to structural feature extraction to obtain richer local features. Then, the structural feature cost space is constructed based on this. At the same time, the convolution feature is used to construct the group correlation cost, and the features are divided into 8 groups along the channel dimension, and the correlation is calculated group by group.

[0111] In order to resolve the ambiguity caused by pathological areas, in this example, the prior disparity map is calculated by using the extremely lightweight 3DUNet regularized network aggregation and regularized cost space product. Furthermore, the cost space channel is stimulated by using the weights calculated from the left feature map to perform cost aggregation, thereby introducing more semantic and contextual information in the cost aggregation process, improving the deep learning model's ability to understand and express visual scenes.

[0112] For the main network, in order to obtain rich multi-scale information, a multi-scale feature extraction module including dilated convolution and spatial pyramid pooling (SPP) is introduced in this example. The feature map is compressed to different dimensions using four-scale average pooling layers to obtain context information at different levels, and then the same resolution is achieved through convolution and upsampling operations. Finally, the context information of different scales is spliced ​​and aggregated. In order to make full use of the results of the two network feature extraction modules, in this example, the cost space is constructed in series. The series cost space contains rich feature information, but it also brings a lot of redundant information. Therefore, when constructing the cost space, the output of the prior network is used to filter the cost space. The filtered cost space only needs simple cost aggregation to obtain a high-quality disparity map.

[0113] The comparison results of the stereo matching algorithm provided in this example and the existing stereo matching algorithm on the Scene Flow dataset are shown in Table 1 below. The evaluation index is the endpoint error (EPE) (the lower the index, the better).

[0114] Table 1

[0115]

[0116] The test results of the stereo matching algorithm provided in this example on the KITTI2015 dataset were submitted to the KITTI ranking website, and quantitatively compared with other methods (the lower the index, the better). The comparison results can be shown in Table 2 below:

[0117] Table 2

[0118]

[0119] The above method improves the problem of poor robustness of the stereo matching network based on convolutional neural network (CNN) when dealing with illumination changes. By introducing multi-feature prior information, especially depth prior and global features, the algorithm can better adapt to different lighting conditions, thereby improving matching accuracy. Furthermore, the ability to capture image structural information is enhanced. In view of the problem that the existing CNN-based stereo matching network has deficiencies in dealing with weak textures and complex structures, in this example, through multi-level feature fusion, it is possible to make fuller use of the structural information of the image and improve the robustness and accuracy of matching.

[0120] Furthermore, the processing method provided in this example solves the problems of bloated structure, large memory usage and slow operation speed of existing stereo matching network. By designing a more compact network structure and optimizing the calculation process, this example significantly improves the operation speed and memory efficiency while ensuring the accuracy of the algorithm. Specifically, in this example, the computational complexity of the network is reduced by introducing a lightweight feature extraction module and an efficient matching optimization algorithm. At the same time, through the use of multi-level feature fusion and deep prior information, the algorithm can achieve faster operation speed while maintaining high accuracy.

[0121] That is, a good balance is achieved between the speed and accuracy of the algorithm through the method of this example, which is suitable for stereo matching tasks in various complex scenes.

[0122] Figure 8 1 is a structural block diagram of an embodiment of a stereo matching device in stereo vision provided by the present invention. Figure 8 As shown, the stereo matching device 100 in stereo vision may include an acquisition unit 801 , an extraction unit 802 , a first generation unit 803 and a second generation unit 804 .

[0123] An acquisition unit 801 is used to acquire a left view and a right view captured by a camera;

[0124] An extraction unit 802, configured to extract features of the left view and the right view through a sub-network and construct a fusion cost space of the sub-network;

[0125] A first generating unit 803 is used to obtain a sub-network disparity map through the fusion cost space;

[0126] The second generating unit 804 is used to filter the main network through the sub-network disparity map to obtain a target disparity map.

[0127] In a specific implementation, the extraction unit 802 can specifically extract structural features of the left view and the right view through a feature extraction module in the sub-network; construct a structural cost space according to the extracted structural features, and construct a convolution cost space according to the convolution method; group and map features of different dimensions in the structural cost space and the convolution cost space according to the channel dimension, and then splice them to obtain a fused cost space.

[0128] In a specific implementation, the structural feature extraction of the left view and the right view is performed through the feature extraction module in the sub-network, which may include: taking the left view and the right view as target images; obtaining the feature values ​​of the feature points of the target image in a 3x3 window; taking the current feature point as the central feature point, and comparing the central feature point with the surrounding 8 feature points to generate an 8-bit binary string corresponding to the current feature point; converting the binary string into unsigned 8-bit integer data, and taking the 8-bit unsigned integer data as the structural feature value of the current feature point; using a word embedding method, converting the structural feature value of each feature point into a feature vector that can be learned by a neural network, and representing the similarity between two structural features by the cosine value between the feature vectors.

[0129] In a specific implementation, taking the current feature point as the central feature point and comparing the central feature point with the surrounding eight feature points to generate an 8-bit binary string corresponding to the current feature point may include: generating an 8-bit binary string corresponding to the current feature point according to the following formula:

[0130]

[0131] Among them, C(x,y) is a bit string, indicating the structural relationship between two pixels, I(x,y) indicates the eigenvalue of the central feature point (x,y), I(x+i,y+j) indicates the eigenvalue of the points around the central feature point, n′ is the largest integer not greater than half of n (window length), n is the length of the window, m′ is the largest integer not greater than half of m (window height), m is the height of the window, is the bit-by-bit concatenation operation, ξ thr is the learnable threshold.

[0132] In a specific implementation, obtaining the sub-network disparity map through the fused cost space may include: performing multiple downsampling processes on the fused cost space and the structural cost space through a Hadamard product operation to achieve multiple filtering to obtain the sub-network disparity map.

[0133] In a specific embodiment, the main network is a deep convolutional stereo matching network, and accordingly, the main network is filtered through the sub-network disparity map to obtain a target disparity map, which may include: in the feature extraction stage, the deep convolutional stereo matching network obtains the features extracted by the sub-network from the left view and the right view; the deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view through a spatial pyramid module in combination with the features extracted by the sub-network from the left view and the right view; in the cost aggregation stage, the deep convolutional stereo matching network filters the sub-network disparity map to obtain a target disparity map.

[0134] In a specific embodiment, the deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view through a spatial pyramid module in combination with the features extracted from the left view and the right view by a sub-network, which may include: the deep convolutional stereo matching network uses an average pooling layer of four scales to compress the feature data of the left view and the right view into multiple dimensions to obtain context information of multiple levels; converts the context information of multiple levels to the same resolution through convolution and upsampling processing; and splices and aggregates the context information of multiple levels converted to the same resolution to obtain context information of multiple scales.

[0135] The present invention also provides a storage medium corresponding to the stereo matching method in stereo vision, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0136] The present invention also provides a computer program product corresponding to a stereo matching method in stereo vision, comprising a computer program, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0137] Accordingly, the solution provided by the present invention, based on the main network, introduces a sub-network to optimize the stereo matching process, reducing the computational complexity of the network. Furthermore, through the use of multi-level feature fusion and deep prior information, the algorithm can achieve faster running speed while maintaining high precision, thereby effectively improving the efficiency and accuracy of stereo matching.

[0138] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0142] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A stereo matching method in stereo vision, characterized in that: include: Get the left view and right view captured by the camera; Extracting features of the left view and the right view through a sub-network, and constructing a fusion cost space of the sub-network; Obtaining a sub-network disparity map through the fusion cost space; The main network is filtered through the sub-network disparity map to obtain a target disparity map.

2. The method according to claim 1, characterized in that The features of the left view and the right view are extracted through a sub-network, and a fusion cost space of the sub-network is constructed, including: Extracting structural features of the left view and the right view through a feature extraction module in the sub-network; Construct a structural cost space according to the extracted structural features, and construct a convolution cost space according to the convolution method; The features of different dimensions in the structural cost space and the convolutional cost space are grouped and mapped according to the channel dimension, and then spliced ​​to obtain the fused cost space.

3. The method according to claim 2, characterized in that The feature extraction module in the sub-network extracts structural features of the left view and the right view, including: Taking the left view and the right view as target images; Obtain feature values ​​of feature points of the target image within a 3x3 window; Taking the current feature point as the central feature point, comparing the central feature point with the eight surrounding feature points to generate an 8-bit binary string corresponding to the current feature point; Convert the binary string into unsigned 8-bit integer data, and use the 8-bit unsigned integer data as the structural feature value of the current feature point; Using the word embedding method, the structural feature values ​​of each feature point are converted into feature vectors that can be learned by the neural network, and the similarity between two structural features is represented by the cosine value between the feature vectors.

4. The method according to claim 3, characterized in that The current feature point is taken as the central feature point, and the central feature point is compared with the eight surrounding feature points to generate an 8-bit binary string corresponding to the current feature point, including: An 8-bit binary string corresponding to the current feature point is generated according to the following formula: Among them, C(x,y) is a bit string, indicating the structural relationship between two pixels, I(x,y) indicates the eigenvalue of the central feature point (x,y), I(x+i,y+j) indicates the eigenvalue of the points around the central feature point, n′ is the largest integer not greater than half of n (window length), n is the length of the window, m′ is the largest integer not greater than half of m (window height), m is the height of the window, is the bit-by-bit concatenation operation, ξ thr is the learnable threshold.

5. The method according to claim 2, characterized in that: The sub-network disparity map is obtained by the fusion cost space, including: Through the Hadamard product operation, the fusion cost space and the structure cost space are downsampled multiple times to achieve multiple filtering to obtain a sub-network disparity map.

6. The method according to claim 1, characterized in that The main network is a deep convolutional stereo matching network. Accordingly, the main network is filtered through the sub-network disparity map to obtain a target disparity map, including: In the feature extraction stage, the deep convolutional stereo matching network obtains features extracted by the sub-network from the left view and the right view; The deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view by combining the features extracted from the left view and the right view by the sub-network through the spatial pyramid module; The deep convolutional stereo matching network performs filtering through the sub-network disparity map in the cost aggregation stage to obtain a target disparity map.

7. The method according to claim 6, characterized in that The deep convolutional stereo matching network extracts multi-dimensional features of the left view and the right view through a spatial pyramid module in combination with features extracted from the left view and the right view by a sub-network, including: The deep convolutional stereo matching network uses four-scale average pooling layers to compress the feature data of the left view and the right view into multiple dimensions to obtain multiple levels of context information; Through convolution and upsampling, multiple levels of context information are converted to the same resolution; The context information of multiple levels converted to the same resolution is concatenated and aggregated to obtain context information of multiple scales.

8. A stereo matching device in stereo vision, characterized in that: include: An acquisition unit, used for acquiring a left view and a right view captured by a camera; An extraction unit, used to extract features of the left view and the right view through a sub-network, and construct a fusion cost space of the sub-network; A first generating unit, configured to obtain a sub-network disparity map through the fusion cost space; The second generating unit is used to filter the main network through the sub-network disparity map to obtain a target disparity map.

9. A storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of any method described in claims 1-7 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of any one of the methods of claims 1 to 7 when being executed by a processor.

Images