Stereo matching method and device, medium and electronic equipment
By using global similarity matching and triangulation methods to process the depth information of non-key points in the stereo matching algorithm, the problem of insufficient matching robustness and real-time in the prior art is solved, and efficient stereo matching in complex scenarios is achieved.
Patent Information
- Application Number
- CN202411776345.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-28
AI Technical Summary
When existing stereo matching algorithms deal with complex scenes with repeated texture areas, they have poor matching robustness or large calculations and poor real-time performance.
By acquiring binocular images, sparse key points are extracted and global similarity matching is performed, the depth information of the key points is obtained, and the depth information of non-key points is processed through triangulation method, and the target depth map is finally generated.
It realizes stereo matching with good matching robustness and real-time performance in complex scenarios, reducing the amount of calculation and improving the accuracy and speed of matching.
Smart Images
Figure CN119723127A_ABST
Abstract
Description
Background Art
[0002] The stereo matching algorithm is the core technology in binocular stereo vision and is widely used in the fields of autonomous driving, augmented reality and human-computer interaction. It mainly finds matching point pairs from two or more images of the same scene, and calculates the depth information of the spatial physical points corresponding to the point pairs based on the principle of triangulation.
[0003] When existing stereo matching algorithms are applied to complex scenes with repeated texture areas, if an algorithm focusing on local features is used for matching, the matching robustness is poor. If an algorithm with global matching capabilities is used for matching, the calculation amount is large, the time consumption is long, and the real-time performance is poor.
[0004] Therefore, there is an urgent need for a stereo matching algorithm that has both good matching robustness and real-time performance. Summary of the invention
[0005] The embodiments of the present application provide a stereo matching method, device, medium and electronic device, which have good matching robustness and real-time performance.
[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.
[0007] According to a first aspect of an embodiment of the present application, a stereo matching method is provided, comprising:
[0008] Acquire a binocular image, where the binocular image is a first image and a second image;
[0009] According to the first image, obtain a corresponding first feature map and a first sparse key point thereof, and according to the second image, obtain a corresponding second feature map and a second sparse key point thereof;
[0010] Performing global matching of similarity on the first sparse key point and the second sparse key point to obtain the first sparse key point and the second sparse key point matched by similarity, and obtaining depth information of the first sparse key point according to the first sparse key point and the second sparse key point matched by similarity;
[0011] Obtaining depth information of non-key points according to the depth information of the first sparse key points, wherein the non-key points are feature points other than the first sparse key points in the first feature map;
[0012] A target depth map is obtained according to the depth information of the first sparse key points and the depth information of the non-key points.
[0013] In an embodiment of the present application, based on the aforementioned solution, obtaining the corresponding first feature map and its first sparse key points according to the first image, and obtaining the corresponding second feature map and its second sparse key points according to the second image, includes:
[0014] Performing feature extraction on the first image to obtain the first feature map, and performing key point detection on the first feature map using a key point detection network to obtain the first sparse key points;
[0015] Perform feature extraction on the second image to obtain the second feature map, and use a key point detection network to perform key point detection on the second feature map to obtain the second sparse key points.
[0016] In the embodiment of the present application, based on the above solution, extracting features from the first image to obtain the first feature map includes:
[0017] Downsampling the first image, extracting multi-scale features of the first image, and obtaining a first feature subgraph;
[0018] Upsampling or downsampling the first feature sub-image to obtain a first feature deformation image;
[0019] fusing the first feature deformation map to obtain the first feature map;
[0020] The step of extracting features from the second image to obtain the second feature map includes:
[0021] Downsampling the second image, extracting multi-scale features of the second image, and obtaining a second feature subgraph;
[0022] Upsampling or downsampling the second feature sub-image to obtain a second feature deformation image;
[0023] The second feature deformation map is fused to obtain the second feature map.
[0024] In an embodiment of the present application, based on the above-mentioned solution, the key point detection of the first feature map using a key point detection network to obtain the first sparse key point includes:
[0025] According to a preset step length, the first feature map is divided into a plurality of non-overlapping first feature regions, similarities between position features of the first feature point and other feature points in the first feature region are calculated to obtain a maximum similarity of the first feature points, and the maximum similarities of the first feature points are compared to select the first feature point with the smallest maximum similarity as the first sparse key point;
[0026] Using a key point detection network to perform key point detection on the second feature map to obtain the second sparse key points includes:
[0027] According to a preset step size, the second feature map is divided into multiple non-overlapping second feature areas, the similarity of the position features of the second feature points and other feature points in the second feature areas is calculated, the maximum similarity of the second feature points is obtained, and the maximum similarities of the second feature points are compared to select the second feature point with the smallest maximum similarity as the second sparse key point.
[0028] In an embodiment of the present application, based on the foregoing solution, the first sparse key point and the second sparse key point are globally matched in similarity to obtain the first sparse key point and the second sparse key point matched in similarity, and the first sparse key point and the second sparse key point matched in similarity are obtained according to the first sparse key point and the second sparse key point, The depth information of the first sparse key point is obtained, including:
[0029] According to the first sparse key point and the second sparse key point, obtain a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point;
[0030] Calculate the similarity between the first descriptor and all the second descriptors to obtain a target similarity;
[0031] Using the second descriptor corresponding to the largest target similarity as a matching descriptor of the first descriptor;
[0032] The first sparse key points and the second sparse key points with similarity matching are obtained according to the first descriptor and the matching descriptor.
[0033] In an embodiment of the present application, based on the above-mentioned solution, obtaining a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point according to the first sparse key point and the second sparse key point includes:
[0034] According to the first sparse key point, obtain a first coordinate and a first descriptor corresponding to the first sparse key point, and according to the first coordinate and the first descriptor, obtain a first input sequence;
[0035] According to the second sparse key point, obtain a second coordinate and a second descriptor corresponding to the second sparse key point, and according to the second coordinate and the second descriptor, obtain a second input sequence;
[0036] Mapping the first input sequence and the second input sequence to a feature vector space of the same dimension through an embedding layer;
[0037] The first input sequence and the second input sequence are calculated using a multi-head attention mechanism to obtain the first descriptor and the second descriptor with information enhancement.
[0038] In an embodiment of the present application, based on the above-mentioned solution, the multi-head attention mechanism is used to calculate the first input sequence and the second input sequence to obtain the information-enhanced first descriptor and the second descriptor, including:
[0039] A multi-head attention mechanism is used to perform self-attention calculation on the first input sequence, self-attention calculation on the second input sequence, and mutual attention calculation on the first input sequence and the second input sequence to obtain the first descriptor and the second descriptor with information enhancement.
[0040] In an embodiment of the present application, based on the above-mentioned solution, obtaining the first sparse key point and the second sparse key point with similarity matching according to the first descriptor and the matching descriptor includes:
[0041] Using a first sparse key point corresponding to the first descriptor and a second sparse key point corresponding to the matching descriptor as the first sparse key point and the second sparse key point for similarity matching;
[0042] The obtaining depth information of the first sparse key point according to the similarity matching of the first sparse key point and the second sparse key point includes:
[0043] The coordinates of the first sparse key point and the second sparse key point that are similarity matched are processed by a triangulation method to obtain depth information of the first sparse key point.
[0044] In the embodiment of the present application, based on the above solution, it also includes:
[0045] Determine whether the target similarity between the first descriptor and the matching descriptor is greater than or equal to a preset threshold;
[0046] If not, the first sparse key point corresponding to the first descriptor is discarded, and the step of obtaining the first sparse key point and the second sparse key point with similarity matching based on the first descriptor and the matching descriptor is not performed.
[0047] In an embodiment of the present application, based on the above solution, obtaining the depth information of the non-key point according to the depth information of the first sparse key point includes:
[0048] Calculate the similarity between the non-key point and all the first sparse key points to obtain a similarity value;
[0049] The similarity value is used as the weight of the depth information of the first sparse key point, and a weighted average is calculated to obtain the depth information of the non-key point.
[0050] According to a second aspect of the present application, a stereo matching device is provided, comprising:
[0051] An acquisition unit, which acquires a binocular image, wherein the binocular image is a first image and a second image;
[0052] A first obtaining unit, which obtains a corresponding first feature map and a first sparse key point thereof according to the first image, and obtains a corresponding second feature map and a second sparse key point thereof according to the second image;
[0053] A second obtaining unit performs global matching of similarity on the first sparse key point and the second sparse key point to obtain the first sparse key point and the second sparse key point with similarity matching, and obtains depth information of the first sparse key point according to the first sparse key point and the second sparse key point with similarity matching;
[0054] A third obtaining unit, which obtains depth information of non-key points according to the depth information of the first sparse key points, wherein the non-key points are other feature points other than the first sparse key points in the first feature map;
[0055] A fourth obtaining unit obtains a target depth map according to the depth information of the first sparse key points and the depth information of the non-key points.
[0056] According to a third aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by a processor, the method described in any embodiment of the first aspect above is implemented.
[0057] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions of the processors, wherein when the executable instructions are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the first aspect above.
[0058] The beneficial effects of this application are as follows:
[0059] The first sparse key point and the second sparse key point are globally matched for similarity. On the one hand, compared with the global matching of similarity between all feature points of the first feature map and all feature points of the second feature map, the number of key point matches is smaller, resulting in less calculation and better real-time performance. On the other hand, due to the use of global matching, the matching robustness is higher, and thus both real-time performance and matching robustness are better.
[0060] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0062] Figure 1 A flow chart of a stereo matching method in an embodiment of the present application is shown;
[0063] Figure 2 A schematic diagram showing a first feature map and a second feature map obtained by feature extraction in an embodiment of the present application is shown;
[0064] Figure 3 A schematic diagram showing detection of a first sparse key point and a second sparse key point by a key point detection network in an embodiment of the present application is shown;
[0065] Figure 4 A schematic diagram showing key point matching in an embodiment of the present application is shown;
[0066] Figure 5 A schematic diagram of obtaining a target depth map in an embodiment of the present application is shown;
[0067] Figure 6 A block diagram of a stereo matching device in an embodiment of the present application is shown;
[0068] Figure 7 A schematic diagram of a computer-readable storage medium according to an embodiment of the present application;
[0069] Figure 8 It is a schematic diagram of the system structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.
[0072] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0073] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0074] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0075] Figure 1 A flow chart of a stereo matching method in an embodiment of the present application is shown. Figure 2 A schematic diagram showing a first feature map and a second feature map obtained by feature extraction in an embodiment of the present application is shown, Figure 2 In the figure, the upper left image is the first image, and the lower left image is the second image. Figure 3 A schematic diagram showing detection of a first sparse key point and a second sparse key point by a key point detection network in an embodiment of the present application is shown, Figure 3In the figure, the feature points in the upper right drawing are the first sparse key points, and the feature points in the lower right drawing are the second sparse key points. Figure 4 A schematic diagram showing key point matching in an embodiment of the present application is shown. Figure 4 In the figure, the feature points in the upper left drawing are the first sparse key points, and the feature points in the lower left drawing are the second sparse key points. Figure 5 A schematic diagram of obtaining a target depth map in an embodiment of the present application is shown. Figure 5 The right side of the figure is the target depth map, see Figure 1 , the present application provides a stereo matching method, which includes at least S1 to S5, which are described in detail as follows:
[0076] In step S1, a binocular image is acquired, where the binocular image is a first image and a second image;
[0077] Specifically, the binocular image can be obtained by taking a binocular stereo camera, the binocular stereo camera includes a first camera and a second camera, the first image is an image taken by the first camera, and the second image is an image taken by the second camera.
[0078] In step S2, a corresponding first feature map and its first sparse key points are obtained according to the first image, and a corresponding second feature map and its second sparse key points are obtained according to the second image.
[0079] Specifically, the first sparse key points can be understood as a small number of representative key points in the first feature map, and the second sparse key points can be understood as a small number of representative key points in the second feature map.
[0080] In step S3, a global matching of similarity is performed on the first sparse key point and the second sparse key point to obtain the first sparse key point and the second sparse key point with similarity matching, and depth information of the first sparse key point is obtained according to the first sparse key point and the second sparse key point with similarity matching;
[0081] In step S4, depth information of non-key points is obtained according to the depth information of the first sparse key points, wherein the non-key points are other feature points other than the first sparse key points in the first feature map;
[0082] In step S5, a target depth map is obtained according to the depth information of the first sparse key points and the depth information of the non-key points.
[0083] Optionally, obtaining a corresponding first feature map and a first sparse key point thereof according to the first image, and obtaining a corresponding second feature map and a second sparse key point thereof according to the second image, includes:
[0084] Performing feature extraction on the first image to obtain the first feature map, and performing key point detection on the first feature map using a key point detection network to obtain the first sparse key points;
[0085] Perform feature extraction on the second image to obtain the second feature map, and use a key point detection network to perform key point detection on the second feature map to obtain the second sparse key points.
[0086] Optionally, extracting features from the first image to obtain the first feature map includes:
[0087] Downsampling the first image, extracting multi-scale features of the first image, and obtaining a first feature subgraph;
[0088] Upsampling or downsampling the first feature sub-image to obtain a first feature deformation image;
[0089] fusing the first feature deformation map to obtain the first feature map;
[0090] The step of extracting features from the second image to obtain the second feature map includes:
[0091] Downsampling the second image, extracting multi-scale features of the second image, and obtaining a second feature subgraph;
[0092] Upsampling or downsampling the second feature sub-image to obtain a second feature deformation image;
[0093] The second feature deformation map is fused to obtain the second feature map.
[0094] Specifically, by extracting multi-scale features of the first image, it is beneficial to capture key point information of different scales in the first image, and provide more comprehensive local features for subsequent key point matching. When upsampling the first feature sub-graph to obtain a first feature deformation graph, if the maximum size of the first feature sub-graph is a first maximum size, then the sizes of other first feature sub-graphs other than the first feature sub-graph with the first maximum size are changed to the first maximum size, thereby converting the first feature sub-graph into the first feature deformation graph. When downsampling the first feature sub-graph to obtain a first feature deformation graph, if the minimum size of the first feature sub-graph is a first minimum size, then the sizes of other first feature sub-graphs other than the first feature sub-graph with the first minimum size are changed to the first minimum size, thereby converting the first feature sub-graph into the first feature deformation graph. extracting multi-scale features of the second image, which is beneficial to capturing key point information of different scales in the second image, and providing more comprehensive local features for subsequent key point matching; upsampling the second feature sub-graph to obtain a second feature deformation graph, if the maximum size of the second feature sub-graph is the second maximum size, then the sizes of the other second feature sub-graphs other than the second feature sub-graph with the second maximum size are changed to the second maximum size, thereby converting the second feature sub-graph into the second feature deformation graph; downsampling the second feature sub-graph to obtain a second feature deformation graph, if the minimum size of the second feature sub-graph is the second minimum size, then the sizes of the other second feature sub-graphs other than the second feature sub-graph with the second minimum size are changed to the second minimum size, thereby converting the second feature sub-graph into the second feature deformation graph.
[0095] Optionally, the first feature deformation maps are fused by concatenation or addition to obtain the first feature map, and the second feature deformation maps are fused by concatenation or addition to obtain the second feature map.
[0096] Optionally, a convolutional neural network (CNN) model is used when downsampling the first image, extracting multi-scale features of the first image, and obtaining a first feature subgraph, or when downsampling the second image, extracting multi-scale features of the second image, and obtaining a second feature subgraph.
[0097] Optionally, the using a key point detection network to perform key point detection on the first feature map to obtain the first sparse key point includes:
[0098] According to a preset step length, the first feature map is divided into a plurality of non-overlapping first feature regions, the similarity of the position features of the first feature point and other feature points in the first feature region is calculated, and the maximum similarity of the first feature points is obtained, and the maximum similarities of the first feature points are compared to take the first feature point with the smallest maximum similarity as the first sparse key point. Among the feature points of the first feature region, the first sparse key point can be understood as the feature point that is least similar to other feature points, that is, the most representative and distinguishing;
[0099] Using a key point detection network to perform key point detection on the second feature map to obtain the second sparse key points includes:
[0100] According to a preset step size, the second feature map is divided into a plurality of non-overlapping second feature areas, and the similarity of the position features of the second feature point and other feature points in the second feature area is calculated to obtain the maximum similarity of the second feature point, and the maximum similarities of the second feature points are compared to take the second feature point with the smallest maximum similarity as the second sparse key point. Among the feature points in the second feature area, the second sparse key point can be understood as the feature point that is least similar to other feature points, that is, the most representative and distinctive.
[0101] Specifically, the key point detection network can automatically detect and locate key feature points in an image through a deep learning model. When the key point detection network performs key point detection on a feature map, it includes dividing the feature area, calculating the maximum similarity of the feature points in the feature area, and determining the feature point with the smallest maximum similarity as the coefficient key point. When calculating the similarity of the positional features of the first feature point and other feature points in the first feature area, the cosine similarity or the Euclidean distance can be calculated. When calculating the similarity of the positional features of the second feature point and other feature points in the second feature area, the cosine similarity or the Euclidean distance can be calculated.
[0102] It should be understood that when calculating the cosine similarity, the closer it is to 1, the more similar it is, and the closer it is to 0, the less similar it is.
[0103] For example, when the preset step size is 8 pixels, the first characteristic region and the second characteristic region are regions having 8*8 pixels.
[0104] Optionally, performing global matching of similarity on the first sparse key point and the second sparse key point to obtain the first sparse key point and the second sparse key point matched by similarity, and obtaining depth information of the first sparse key point according to the first sparse key point and the second sparse key point matched by similarity, comprises:
[0105] According to the first sparse key point and the second sparse key point, obtain a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point;
[0106] Calculate the similarity between the first descriptor and all the second descriptors to obtain a target similarity;
[0107] Using the second descriptor corresponding to the largest target similarity as a matching descriptor of the first descriptor;
[0108] The first sparse key points and the second sparse key points with similarity matching are obtained according to the first descriptor and the matching descriptor.
[0109] Optionally, obtaining a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point according to the first sparse key point and the second sparse key point includes:
[0110] According to the first sparse key point, obtain a first coordinate and a first descriptor corresponding to the first sparse key point, and according to the first coordinate and the first descriptor, obtain a first input sequence;
[0111] According to the second sparse key point, obtain a second coordinate and a second descriptor corresponding to the second sparse key point, and according to the second coordinate and the second descriptor, obtain a second input sequence;
[0112] Mapping the first input sequence and the second input sequence to a feature vector space of the same dimension through an embedding layer;
[0113] The first input sequence and the second input sequence are calculated using a multi-head attention mechanism to obtain the first descriptor and the second descriptor with information enhancement.
[0114] Specifically, according to the first sparse key point, a first coordinate and a first descriptor corresponding to the first sparse key point are obtained, or according to the second sparse key point, a second coordinate and a second descriptor corresponding to the second sparse key point are obtained, which can be obtained by a gather or sampling technology, that is, according to the gather technology, the position feature information of the first sparse key point and the second sparse key point is extracted by indexing, or according to the sampling technology, the position feature information of the first sparse key point and the second sparse key point is extracted by sampling. When the first sparse key point and the second sparse key point are not located at integer point positions, the position feature information of the first sparse key point and the second sparse key point is determined by the average value of the position feature information of the two adjacent points. For example, the coordinates of the first sparse key point are located at (2.5, 0), and the two adjacent points are (2, 0) and (3, 0). The position feature information of the first sparse key point is determined by the average value of the position feature information of (2, 0) and (3, 0). The first descriptor is used to uniquely identify the first sparse key point, and the second descriptor is used to uniquely identify the second sparse key point, providing accurate feature information for subsequent matching. The global correlation of the multi-head attention mechanism helps to achieve more comprehensive key point matching and improve matching accuracy.
[0115] Optionally, the adopting a multi-head attention mechanism to calculate the first input sequence and the second input sequence to obtain the information-enhanced first descriptor and the second descriptor includes:
[0116] A multi-head attention mechanism is used to perform self-attention calculation on the first input sequence, self-attention calculation on the second input sequence, and mutual attention calculation on the first input sequence and the second input sequence to obtain the first descriptor and the second descriptor with information enhancement.
[0117] Optionally, the encoder of the Transformer network is used to perform “obtaining, according to the first sparse key point and the second sparse key point, a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point”, which is specifically described as follows:
[0118] When obtaining a first input sequence according to the first coordinates and the first descriptor, the first coordinates and the first descriptor are concatenated, and the first coordinates and the first descriptor are position-encoded to form the first input sequence, wherein the first coordinates are used to encode the position information of the first sparse key point, and the first descriptor encodes the visual feature, and the position encoding of the first coordinates and the first descriptor is to retain the position information of the first coordinates and the first descriptor in the first image;
[0119] When a second input sequence is obtained according to the second coordinates and the second descriptor, the second coordinates and the second descriptor are concatenated, and the second coordinates and the second descriptor are position-encoded to form the second input sequence, the second coordinates are used to encode the position information of the second sparse key point, and the second descriptor encodes the visual feature. The position encoding of the second coordinates and the second descriptor is to retain the position information of the second coordinates and the second descriptor in the second image.
[0120] Linearly mapping the first input sequence and the second input sequence to a feature vector space of the same dimension through an embedding layer;
[0121] A multi-head attention mechanism is used to perform self-attention calculation on the first input sequence, self-attention calculation on the second input sequence, and mutual attention calculation on the first input sequence and the second input sequence to obtain the information-enhanced first descriptor and the second descriptor. Each attention head performs global correlation modeling on the first input sequence and the second input sequence to capture long-range dependencies. The multi-head attention mechanism is conducive to obtaining different attention information from different representation subspaces.
[0122] It should be understood that, compared with the original first descriptor and the original second descriptor, the information-enhanced first descriptor and the second descriptor have better distinguishing ability due to the fusion of global context information.
[0123] Optionally, obtaining the first sparse key point and the second sparse key point with similarity matching according to the first descriptor and the matching descriptor includes:
[0124] Using a first sparse key point corresponding to the first descriptor and a second sparse key point corresponding to the matching descriptor as the first sparse key point and the second sparse key point for similarity matching;
[0125] The obtaining depth information of the first sparse key point according to the similarity matching of the first sparse key point and the second sparse key point includes:
[0126] The coordinates of the first sparse key point and the second sparse key point that are similarity matched are processed by a triangulation method to obtain depth information of the first sparse key point.
[0127] Optionally, it also includes:
[0128] Determine whether the target similarity between the first descriptor and the matching descriptor is greater than or equal to a preset threshold;
[0129] If not, the first sparse key point corresponding to the first descriptor is discarded, and the step of obtaining the first sparse key point and the second sparse key point with similarity matching based on the first descriptor and the matching descriptor is not performed. If yes, the step of obtaining the first sparse key point and the second sparse key point with similarity matching based on the first descriptor and the matching descriptor is performed.
[0130] For example, the first descriptor is a i , where i is a natural number from 1 to n, and the second descriptor is b j , where j is a natural number from 1 to m, and cosine similarity is used to calculate a i and all b j The similarity of the target is obtained, a i The target similarity can be obtained by the similarity vector S i Indicates that, S i =[s im(a i ,b1),s im(a i ,b2),...,s im(a i ,b n )], if the maximum target similarity is s im(a i ,b k ), then judge s im(a i ,b k ) is greater than or equal to the preset threshold τ, if so, then b k for a i According to the first descriptor and the matching descriptor, the first sparse key point and the second sparse key point with similarity matching are obtained. If not, a i The corresponding first sparse key point is discarded, that is, a i Corresponding to the first sparse key point, there is no reliable second sparse key point in the second image that matches it.
[0131] Optionally, obtaining the depth information of the non-key point according to the depth information of the first sparse key point includes:
[0132] Calculate the similarity between the non-key point and all the first sparse key points to obtain a similarity value;
[0133] The similarity value is used as the weight of the depth information of the first sparse key point, and the weighted average is calculated to obtain the depth information of the non-key point, that is, the depth information of the non-key point is obtained according to the depth information of the first sparse key point by means of autocorrelation volume propagation, wherein the autocorrelation volume propagation can be understood as weighted averaging the depth information of the first sparse key point to obtain the depth information of the non-key point. Specifically, the similarity value is used as the weight of the depth information of the first sparse key point, and the weighted average is calculated to obtain the depth information of the non-key point, which can be performed using the following formula:
[0134] Among them, d p is the depth information of the pth non-key point, T i is the similarity value between the i-th sparse key point and the p-th non-key point, d i is the depth information of the i-th first sparse key point.
[0135] Figure 6 A block diagram of a stereo matching device in an embodiment of the present application is shown, see Figure 2 According to the second aspect of the present application, a stereo matching device 100 is provided, comprising:
[0136] An acquisition unit 101 acquires a binocular image, where the binocular image is a first image and a second image;
[0137] A first obtaining unit 102 obtains a first feature map and a first sparse key point corresponding to the first image, and obtains a second feature map and a second sparse key point corresponding to the second image;
[0138] The second obtaining unit 103 performs global matching of similarity on the first sparse key point and the second sparse key point to obtain the first sparse key point and the second sparse key point with similarity matching, and obtains depth information of the first sparse key point according to the first sparse key point and the second sparse key point with similarity matching;
[0139] A third obtaining unit 104 obtains depth information of non-key points according to the depth information of the first sparse key points, wherein the non-key points are other feature points other than the first sparse key points in the first feature map;
[0140] The fourth obtaining unit 105 obtains a target depth map according to the depth information of the first sparse key points and the depth information of the non-key points.
[0141] In the present application, a global match is performed on the similarity of the first sparse key points and the second sparse key points. On the one hand, compared with a global match of the similarity of all feature points of the first feature map and all feature points of the second feature map, the number of key point matches is smaller, resulting in less calculation and better real-time performance. On the other hand, due to the use of global matching, the matching robustness is higher, and thus both real-time performance and matching robustness are achieved, thereby improving the performance in areas with weaker textures and effectively improving the applicability of various scenarios.
[0142] Based on the same inventive concept, as a third aspect, the present application also provides a computer-readable storage medium on which is stored a program product capable of implementing the above-mentioned stereo matching method of the present specification. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present application described in the above-mentioned "Exemplary Method" section of the present specification.
[0143] refer to Figure 7 As shown, a program product 200 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0144] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0145] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0146] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0147] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0148] As another aspect, the present application also provides an electronic device capable of implementing the above method.
[0149] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.
[0150] Refer to the following Figure 8 The electronic device 300 according to this embodiment of the present application is described. Figure 8 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0151] like Figure 8As shown, the electronic device 300 is in the form of a general computing device. The components of the electronic device 300 may include but are not limited to: at least one processing unit 310, at least one storage unit 320, and a bus 330 connecting different system components (including the storage unit 320 and the processing unit 310).
[0152] The storage unit stores program codes, which can be executed by the processing unit 310, so that the processing unit 310 executes the steps described in the above “Example Method” section of this specification according to various exemplary implementations of the present application.
[0153] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 321 and / or a cache memory unit 322 , and may further include a read-only memory unit (ROM) 323 .
[0154] The storage unit 320 may also include a program / utility 324 having a set (at least one) of program modules 325, such program modules 325 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0155] Bus 330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0156] The electronic device 300 may also communicate with one or more external devices 400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 300, and / or communicate with any device that enables the electronic device 300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 350. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 360. As shown, the network adapter 360 communicates with other modules of the electronic device 300 via a bus 330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0157] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.
[0158] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0159] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly or all or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.
[0161] The above description is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
Claims
1. A stereo matching method, characterized in that: include: Acquire a binocular image, where the binocular image is a first image and a second image; According to the first image, obtain a corresponding first feature map and a first sparse key point thereof, and according to the second image, obtain a corresponding second feature map and a second sparse key point thereof; Performing global matching of similarity on the first sparse key point and the second sparse key point to obtain the first sparse key point and the second sparse key point with similarity matching, and obtaining depth information of the first sparse key point according to the first sparse key point and the second sparse key point with similarity matching; Obtaining depth information of non-key points according to the depth information of the first sparse key points, wherein the non-key points are feature points other than the first sparse key points in the first feature map; A target depth map is obtained according to the depth information of the first sparse key points and the depth information of the non-key points.
2. A stereo matching method according to claim 1, characterized in that: The step of obtaining a corresponding first feature map and a first sparse key point thereof according to the first image, and obtaining a corresponding second feature map and a second sparse key point thereof according to the second image includes: Performing feature extraction on the first image to obtain the first feature map, and performing key point detection on the first feature map using a key point detection network to obtain the first sparse key points; Perform feature extraction on the second image to obtain the second feature map, and use a key point detection network to perform key point detection on the second feature map to obtain the second sparse key points.
3. A stereo matching method according to claim 2, characterized in that: The extracting features from the first image to obtain the first feature map includes: Downsampling the first image, extracting multi-scale features of the first image, and obtaining a first feature subgraph; Upsampling or downsampling the first feature sub-image to obtain a first feature deformation image; fusing the first feature deformation map to obtain the first feature map; The step of extracting features from the second image to obtain the second feature map includes: Downsampling the second image, extracting multi-scale features of the second image, and obtaining a second feature subgraph; Upsampling or downsampling the second feature sub-image to obtain a second feature deformation image; The second feature deformation map is fused to obtain the second feature map.
4. A stereo matching method according to claim 2, characterized in that: The step of using a key point detection network to perform key point detection on the first feature map to obtain the first sparse key points includes: According to a preset step length, the first feature map is divided into a plurality of non-overlapping first feature regions, similarities between position features of the first feature point and other feature points in the first feature region are calculated to obtain a maximum similarity of the first feature points, and the maximum similarities of the first feature points are compared to select the first feature point with the smallest maximum similarity as the first sparse key point; Using a key point detection network to perform key point detection on the second feature map to obtain the second sparse key points includes: According to a preset step size, the second feature map is divided into multiple non-overlapping second feature areas, the similarity of the position features of the second feature points and other feature points in the second feature areas is calculated, the maximum similarity of the second feature points is obtained, and the maximum similarities of the second feature points are compared to select the second feature point with the smallest maximum similarity as the second sparse key point.
5. A stereo matching method according to claim 1, characterized in that: The globally matching the first sparse key point and the second sparse key point in similarity to obtain the first sparse key point and the second sparse key point with similarity matching, and obtaining the depth information of the first sparse key point according to the first sparse key point and the second sparse key point with similarity matching, comprises: According to the first sparse key point and the second sparse key point, obtain a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point; Calculate the similarity between the first descriptor and all the second descriptors to obtain a target similarity; Using the second descriptor corresponding to the largest target similarity as a matching descriptor of the first descriptor; The first sparse key points and the second sparse key points with similarity matching are obtained according to the first descriptor and the matching descriptor.
6. A stereo matching method according to claim 5, characterized in that: The obtaining, according to the first sparse key point and the second sparse key point, a first descriptor corresponding to the first sparse key point and a second descriptor corresponding to the second sparse key point, comprises: According to the first sparse key point, obtain a first coordinate and a first descriptor corresponding to the first sparse key point, and according to the first coordinate and the first descriptor, obtain a first input sequence; According to the second sparse key point, obtain a second coordinate and a second descriptor corresponding to the second sparse key point, and according to the second coordinate and the second descriptor, obtain a second input sequence; Mapping the first input sequence and the second input sequence to a feature vector space of the same dimension through an embedding layer; The first input sequence and the second input sequence are calculated using a multi-head attention mechanism to obtain the first descriptor and the second descriptor with information enhancement.
7. A stereo matching method according to claim 6, characterized in that: The method of using a multi-head attention mechanism to calculate the first input sequence and the second input sequence to obtain the first descriptor and the second descriptor with enhanced information includes: A multi-head attention mechanism is used to perform self-attention calculation on the first input sequence, self-attention calculation on the second input sequence, and mutual attention calculation on the first input sequence and the second input sequence to obtain the first descriptor and the second descriptor with information enhancement.
8. A stereo matching method according to claim 5, characterized in that: The obtaining, according to the first descriptor and the matching descriptor, the first sparse key point and the second sparse key point with similarity matching includes: Using a first sparse key point corresponding to the first descriptor and a second sparse key point corresponding to the matching descriptor as the first sparse key point and the second sparse key point for similarity matching; The obtaining depth information of the first sparse key point according to the similarity matching of the first sparse key point and the second sparse key point includes: The coordinates of the first sparse key point and the second sparse key point that are similarity matched are processed by a triangulation method to obtain depth information of the first sparse key point.
9. A stereo matching method according to claim 5, characterized in that: Also includes: Determine whether the target similarity between the first descriptor and the matching descriptor is greater than or equal to a preset threshold; If not, the first sparse key point corresponding to the first descriptor is discarded, and the step of obtaining the first sparse key point and the second sparse key point with similarity matching based on the first descriptor and the matching descriptor is not performed.
10. A stereo matching method according to claim 1, characterized in that: The obtaining the depth information of the non-key point according to the depth information of the first sparse key point includes: Calculate the similarity between the non-key point and all the first sparse key points to obtain a similarity value; The similarity value is used as the weight of the depth information of the first sparse key point, and a weighted average is calculated to obtain the depth information of the non-key point.