A stereo matching method, device and electronic equipment

Abstract

The application relates to the technical field of computer vision, in particular to a stereo matching method, a stereo matching device and electronic equipment. The stereo matching method comprises the following steps: acquiring a left image and a right image collected by left and right cameras, and performing down-sampling feature extraction on the left image and the right image respectively to obtain corresponding left image features and right image features; calculating the correlation of the left image features and the right image features at different disparities to obtain a cost space; upsampling the cost space to a target resolution; wherein the target resolution is a variable resolution; estimating a stereo matching disparity according to the upsampled cost space to obtain a target disparity at the target resolution. The application can output a disparity map of any resolution, thereby bringing great convenience to actual application.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a stereo matching method, apparatus and electronic device. Background Technology

[0002] Binocular stereo vision is a type of computer vision system that primarily involves acquiring images of the external environment using cameras (two cameras in a binocular stereo vision system), and then reconstructing the three-dimensional shape and coordinate information of the scene based on the principle of parallax. Specifically, assuming the input left and right images are IL and IR respectively, using the left image IL as the reference image, given a pixel (xL, yL) in the left image IL, the corresponding point (xR, yR) in the right image IR is calculated; this process is called stereo matching. Then, the parallax d = xL - xR is calculated using the pixel's coordinates in the left and right images. Finally, the scene depth Z = Bf / d is recovered based on the parallax d, where B is the distance between the binocular cameras and f is the focal length of the binocular cameras. Therefore, the core of binocular stereo vision lies in the stereo matching algorithm and the calculation of the parallax map.

[0003] In recent years, deep learning-based binocular stereo matching algorithms have shown significant advantages over traditional stereo matching methods in areas such as weak texture targets and occlusion handling. However, most existing deep learning-based binocular matching methods can only output disparity maps with a fixed resolution. For example, if the input image resolution is W×H, the network can only output a disparity map with a fixed resolution, such as W×H or W / 2×H / 2, and cannot output disparity maps with arbitrary resolutions. Moreover, deep learning-based binocular matching methods are easily affected by various interferences in real-world environments, such as noise, weak textures, and motion blur, meaning there is still room for improvement in the disparity estimation accuracy of these methods. Summary of the Invention

[0004] In view of this, embodiments of this application provide a stereo matching method, apparatus, and electronic device, which can solve at least one technical problem in the related art.

[0005] In a first aspect, one embodiment of this application provides a stereo matching method, comprising: acquiring a left image and a right image captured by a left and a right camera, and performing downsampling feature extraction on the left image and the right image respectively to obtain corresponding left image features and right image features; calculating the correlation between the left image features and the right image features at different disparities to obtain a cost space; upsampling the cost space to a target resolution; wherein the target resolution is a variable resolution; estimating the stereo matching disparity based on the upsampled cost space to obtain the target disparity at the target resolution.

[0006] This embodiment upsamples the cost space and then calculates the disparity based on the upsampled cost space, which can obtain a disparity map of arbitrary resolution, bringing great convenience to practical applications.

[0007] In some embodiments, estimating stereo matching disparity based on the upsampled cost space to obtain the target disparity at the target resolution includes: estimating stereo matching disparity based on the upsampled cost space, and performing multiple iterative calculations on the stereo matching disparity using the initial disparity and initial hidden features or the disparity and hidden features obtained in the previous iteration to obtain the target disparity at the target resolution; wherein the initial disparity and the initial hidden features are used in the first iteration calculation, the initial disparity is the disparity obtained in the first stereo matching, and the initial hidden features are any one of the left image features, the right image features, or a feature that is all zero.

[0008] This embodiment uses a cyclic iterative feedback structure to make the estimated disparity map more accurate and more robust.

[0009] Secondly, one embodiment of this application provides a stereo matching device, comprising: an acquisition module for acquiring a left image and a right image captured by a left and a right camera; a feature extraction module for performing downsampling feature extraction on the left image and the right image respectively to obtain corresponding left image features and right image features; a correlation module for calculating the correlation between the left image features and the right image features at different disparities to obtain a cost space; an upsampling module for upsampling the cost space to a target resolution; wherein the target resolution is a variable resolution; and a disparity calculation module for estimating stereo matching disparity based on the upsampled cost space to obtain the target disparity at the target resolution.

[0010] Thirdly, one embodiment of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stereo matching method as described in any embodiment of the first aspect.

[0011] Fourthly, one embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the stereo matching method as described in any embodiment of the first aspect.

[0012] Fifthly, one embodiment of this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the stereo matching method as described in any embodiment of the first aspect.

[0013] It should be understood that the beneficial effects of aspects two through five can be found in the relevant description of aspect one, and will not be repeated here. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram illustrating the implementation process of a stereo matching method provided in an embodiment of this application;

[0016] Figure 2 This is a schematic diagram of the structure of a stereo matching network provided in one embodiment of this application;

[0017] Figure 3 This is a schematic diagram of the structure of an iterative module in another stereo matching network provided in an embodiment of this application;

[0018] Figure 4 This is a schematic diagram illustrating the implementation process of another stereo matching method provided in an embodiment of this application;

[0019] Figure 5A This is a schematic diagram illustrating the implementation flow of an iterative calculation process provided in an embodiment of this application;

[0020] Figure 5B This is a schematic diagram of the structure of an iterative module in another stereo matching network provided in an embodiment of this application;

[0021] Figure 6 This is a schematic diagram illustrating the implementation flow of another iterative calculation process provided in one embodiment of this application;

[0022] Figure 7 This is a schematic diagram of the structure of an iterative module in another stereo matching network provided in an embodiment of this application;

[0023] Figure 8 This is a schematic diagram illustrating the implementation flow of another iterative calculation process provided in one embodiment of this application;

[0024] Figure 9 This is a schematic diagram of the structure of an iterative module in another stereo matching network provided in an embodiment of this application;

[0025] Figure 10 This is a schematic diagram illustrating the implementation flow of another iterative calculation process provided in one embodiment of this application;

[0026] Figure 11This is a schematic diagram of the structure of an iterative module in another stereo matching network provided in an embodiment of this application;

[0027] Figure 12 This is a schematic diagram of the structure of a stereo matching device provided in an embodiment of this application;

[0028] Figure 13 This is a schematic diagram of the structure of another stereo matching device provided in an embodiment of this application;

[0029] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0031] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0032] The terms "one embodiment" or "some embodiments" described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] Furthermore, in the description of this application, "a plurality of" means two or more. The terms "first" and "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0034] Classical binocular networks output a disparity map at a fixed resolution, such as the same resolution as the input. However, in practical applications, there are situations where it is desirable to output disparity maps at different resolutions.

[0035] To obtain a disparity map of arbitrary resolution, embodiments of this application provide a stereo matching method, apparatus, and electronic device. Figure 1 This is a schematic diagram illustrating the implementation process of a stereo matching method according to an embodiment of this application. The stereo matching method in this embodiment can be executed by an electronic device, including but not limited to computers, tablets, servers, mobile phones, wearable devices, cameras, robots, and vehicle terminals; wherein the server includes, but is not limited to, independent servers or cloud servers. Combined with... Figure 1 and Figure 2 As shown, the stereo matching method provided in this application is configured to correspond to each module in the stereo matching network. Specifically, the stereo matching method may include steps S110 to S150.

[0036] S110, acquire the left and right images captured by the binocular camera.

[0037] The electronic device acquires left and right images using a binocular camera; that is, it acquires left and right images using left and right cameras. The electronic device can operate independently of the binocular camera, or it can include the binocular camera. The electronic device acquires the left and right images captured by the binocular camera. The left image is denoted as P. L The figure on the right is denoted as P. R .

[0038] In one embodiment of this application, the stereo matching network includes an input module ( Figure 2 (Not shown), used to acquire the left and right images captured by the binocular camera.

[0039] S120: Extract the left image features of the left image and extract the right image features of the right image.

[0040] For the left image P L Feature extraction is performed to obtain the features of the left image, which are denoted as f. L For the right image P R Feature extraction is performed to obtain the features shown in the right figure, which are denoted as f. R .

[0041] In one embodiment of this application, such as Figure 2 As shown, the stereo matching network includes a feature extraction module, used to extract features from the left image P. L And the right image P R The input images are fed into a feature extraction module to extract abstract features. The feature extraction module can perform downsampling on the input images using a convolutional network to obtain the features of the left and right images. It should be noted that this application does not specify a particular implementation method for the feature extraction module.

[0042] S130, perform relevant calculations on the features of the left and right images to obtain the cost space.

[0043] According to the following formula, for the feature f in the left figure L and the feature f in the right figure R The cost space C is obtained by performing relevant calculations. corr ;

[0044]

[0045] Where <·, ·> represent inner product operations, and N c f is the number of channels for the feature. L and f R These are the features of the left image and the features of the right image, respectively. An inner product operation is required for each disparity d.

[0046] In one embodiment of this application, the stereo matching network further includes a feature correlation module. See also... Figure 2 As shown, the features of the left and right images are input into the feature correlation module to calculate the feature f of the left image. L and the feature f in the right figure R The correlation at different parallaxes yields the cost space C. corr .

[0047] S140, upsample the cost space to the target resolution.

[0048] The target resolution is variable. The cost space is upsampled to the target resolution. Upsampling methods can include bilinear interpolation, nearest neighbor interpolation, etc. It should be noted that the target resolution can be arbitrary, determined by the resolution of the output target disparity map, and is not limited here. Furthermore, this application does not specifically limit the upsampling method; any upsampling method that can realize the inventive concept of this application can be used in this application.

[0049] In one embodiment of this application, the stereo matching network further includes an upsampling module. See also: Figure 2 As shown, the cost space C corr The input upsampling module upsamples the data to the target resolution, resulting in the upsampled cost space C'. corr The upsampling module can perform upsampling operations on the cost space through convolutional networks. This application does not specify the implementation method of the upsampling module.

[0050] In some embodiments, the step of obtaining a target resolution may be included before step S140. In one embodiment, the step of obtaining a target resolution input by the user may be included before step S140.

[0051] S150, based on the cost space estimation after upsampling, the stereo matching disparity is obtained to obtain the target disparity at the target resolution.

[0052] Assuming the resolution of the left and right images is H×W, the dimension of the constructed cost space is... The target resolution of the desired output disparity map is H′×W′. After upsampling the cost space to the target resolution, the dimension of the upsampled cost space is H′×W′×C1. Based on the upsampled cost space, the stereo matching disparity is estimated, thus obtaining the target disparity that satisfies the target resolution H′×W′.

[0053] In some embodiments, a softargmin operation is performed on the upsampled cost space or a fully connected operation is performed on the channel dimension to estimate the stereo matching disparity. As a non-limiting example, the stereo matching disparity D is obtained by performing a softargmin operation according to the following formula. softargmin softargmin:

[0054]

[0055] Where σ(·) is the softmax operation (i.e., the softmax operation), and C′ corr Given the upsampled cost space, the probability of each disparity d is obtained from the upsampled cost space C′ through a softmax operation. corr The calculated stereo matching disparity D softargmin softargmin is obtained by weighting the probabilities of each disparity d. The disparity regression method described above has stronger robustness.

[0056] As another non-restrictive example, a fully connected operation is performed on the channel dimension of the upsampled cost space to obtain stereo matching parallax.

[0057] In one embodiment of this application, the stereo matching network further includes a disparity calculation module, such as... Figure 2 As shown, the upsampled cost space C' corr The disparity calculation module is used to obtain the stereo matching disparity. It should be noted that classic stereo matching networks generally include a feature extraction module, a feature correlation module, and a disparity calculation module, such as the Pyramid Stereo Matching Network (PSMNet) and StereoNet. Compared to classic stereo matching networks, this embodiment adds an upsampling module, thereby obtaining disparity maps with arbitrary target resolutions.

[0058] To better understand the technical solutions of the embodiments of this application, their working principles will be explained in more detail. If the resolution of the input left and right images is H×W, due to the excessive computational load, the cost space calculation will not be performed at the original resolution. Therefore, during feature extraction, the features are downsampled by a factor of n to... Next, calculate the cost space; at this point, the dimension of the cost space is... C is the number of channels in the cost space. If disparity is calculated at this point, a fixed disparity resolution is obtained. Therefore, in this embodiment, by upsampling the cost space before performing disparity calculation, a disparity map with any target resolution can be obtained. For example, if the dimension of the cost space is... target resolution To obtain the target resolution, the cost space is upsampled before the disparity is calculated, thus yielding the target resolution. The disparity map.

[0059] It should be noted that research has shown that directly calculating disparity using the upsampled cost space is not particularly accurate. Therefore, other embodiments of this application, based on the aforementioned embodiments, add a relevant feature filtering module (or iterative module) to the output side of the disparity calculation module in the stereo matching network to optimize the output disparity map and obtain a higher-precision disparity, thereby enhancing the robustness of the stereo matching network. The iterative module is another core inventive point of this application.

[0060] In this embodiment, the iterative module (i.e., the relevant feature filtering module) optimizes the result of the previous iteration using feedback signals. The iterations are performed multiple times, meaning there are multiple iterative modules. The number of iterations determines the accuracy of the output target disparity and the model inference time, and therefore can be designed according to actual conditions. It can be user-defined or set by system defaults. Specifically, for example... Figure 3 As shown, the iteration module is mainly used to obtain feedback information based on the initial disparity or the disparity obtained in the previous iteration and combined with the physical limitations of binocular stereo vision. Based on the feedback information, the initial disparity or the disparity obtained in the previous iteration is iterated to obtain the updated disparity. In the iteration process, the stereo matching disparity obtained in step S150 is used as the initial disparity.

[0061] As can be seen, in these improved embodiments, a feedback signal is obtained based on the initial disparity or the disparity obtained in the previous iteration and combined with the physical constraints of binocular stereo matching. The feedback signal is then input into the next iteration, thus realizing a feedback system and improving the disparity estimation accuracy through multiple feedbacks.

[0062] The following specific examples illustrate the implementation methods of the two iteration modules. It should be noted that the implementation methods of the iteration modules are not limited to those described below.

[0063] Figure 4 The diagram illustrates a stereo matching method provided in another embodiment of this application. It should be understood that... Figure 4 The illustrated embodiments and Figure 1 The similarities between the illustrated embodiments will not be repeated here; please refer to the foregoing.

[0064] like Figure 4 As shown, the stereo matching method may include steps S210 to S250.

[0065] S210: Acquire the left and right images captured by the binocular camera.

[0066] S220, extract the left image features of the left image and extract the right image features of the right image.

[0067] S230, perform relevant calculations on the features of the left and right images to obtain the cost space.

[0068] S240 upsamples the cost space to the target resolution.

[0069] S250 estimates the stereo matching disparity based on the cost space after upsampling, and performs multiple iterations of calculation on the stereo matching disparity to obtain the target disparity at the target resolution.

[0070] In one embodiment, such as Figure 5A As shown, the iterative calculation process includes more specifically:

[0071] S260, perform an affine transformation on the features of the right image based on the initial disparity or the disparity obtained in the previous iteration to obtain the ideal features of the left image.

[0072] S270, the ideal left image features and the left image features are fused to obtain the fused features.

[0073] S280: Calculate the hidden features and disparity for the current iteration based on the fused features, left image features, and initial hidden features or hidden features obtained from the previous iteration. The hidden features are feedback information obtained by combining the fused features, left image features, initial hidden features, and hidden features obtained from the previous iteration with the physical constraints of binocular stereo vision.

[0074] It should be noted that steps S260 to S280 are an iterative process provided in an embodiment of this application. During the iteration process, the initial disparity and initial hidden features are specific to the first iteration. Preferably, the stereo matching disparity obtained in step S250 can be used as the initial disparity, and the initial hidden feature preferably uses the feature f from the left image. LOr a feature that is all zeros. In addition, the physical limitations of binocular vision include the constraints, matching primitive selection, and other attribute issues that need to be considered when using stereo matching.

[0075] In some embodiments, such as Figure 5B As shown, the iterative module of the stereo matching network can include an image affine transformation (warp) module, a warp feature fusion module, and an update module for implementing iterative computation.

[0076] In one embodiment, the warp module is used to implement step S260. More specifically, in this application, the principle of the image affine transformation (warp) operation is as follows: based on the disparity estimated in the stereo matching network as the difference between corresponding pixels in the left and right images, the features of the left image are deduced back from the estimated disparity value and the features of the right image. If the disparity estimated by the stereo matching network is correct, then in an ideal state, except for the occluded area, the ideal features of the left image deduced back from the features of the right image (denoted as f) are obtained. L-warp ) and the feature f in the left figure L They should be exactly the same. Therefore, in one embodiment of this application, the warp operation performed by the warp module can be expressed as follows: assuming the disparity value estimated in the previous iteration is d, then for each pixel x = (u, v) in the left image, a corresponding pixel x′ = (ud(u, v), v) can be found in the right image. Thus, the index x′ can be used in the right image feature f. R Taking the value above, we obtain the ideal left-image feature f. L-warp .

[0077] In one embodiment, the warp feature fusion module is used to fused the ideal left-image features f obtained from the warp module. L-warp To implement step S270, specifically, the Warp feature fusion module will integrate the features f from the ideal left image. L-war and the feature f in the left figure L The fusion yields the fused features. It should be noted that the fusion methods may include, but are not limited to, fusion (concatenation) operations or subtraction operations.

[0078] In one embodiment, the update module is used to implement step S280 using the fusion features. Specifically, the fusion features and the left image features f are... L The initial hidden features or the hidden features from the previous iteration are input into the update module, which outputs the hidden features of the current iteration and the estimated disparity of the current iteration. The structure of the update module can use sequence models such as LSTM or GRU, and there are no restrictions here.

[0079] In other variant embodiments, the reference feature can be changed, selecting the feature that is traced back to the right figure. In this case, as... Figure 6 As shown, Figure 5ASteps S260 to S280 in the illustrated embodiment can be replaced as follows: S261, performing an affine transformation on the left image features based on the initial disparity or the disparity obtained in the previous iteration to obtain ideal right image features; S271, fusing the ideal right image features and the right image features to obtain fused features; S281, calculating the hidden features and disparity of the current iteration based on the fused features, the right image features, and the initial hidden features or the hidden features obtained in the previous iteration.

[0080] In one embodiment, such as Figure 7 As shown, the iterative module of the stereo matching network may include a warp module, a warp feature fusion module, and an update module. The warp module implements step S261, the warp feature fusion module implements step S271, and the update module implements step S281.

[0081] It should be understood that, Figure 6 and Figure 7 The variant embodiments shown are the same as those described above. Figure 5A and Figure 5B The working principle of the embodiments shown is the same, and those skilled in the art can implement this variant embodiment based on the foregoing embodiments, which will not be repeated here.

[0082] In another embodiment, such as Figure 8 As shown, the iterative calculation process includes more specifically:

[0083] S360 queries the cost parameters in the cost space corresponding to the initial disparity or the disparity obtained in the previous iteration, and uses the cost parameters as feedback features.

[0084] S370, calculate the hidden features and disparity of the current iteration based on the feedback features, left-image features, and initial hidden features or hidden features obtained from the previous iteration.

[0085] It should be noted that steps S360 to S370 are an iterative process provided in another embodiment of this application. During the iteration process, the initial disparity and initial hidden features are specific to the first iteration. Preferably, the stereo matching disparity obtained in step S350 can be used as the initial disparity for the first iteration calculation, and the initial hidden features can use the features f from the left image. L Or a feature that is all zeros.

[0086] In some embodiments, such as Figure 9As shown, the iterative module of the stereo matching network may include a cost space query module and an update module. In one embodiment, the cost space query module is used to implement step S360. Specifically, step S360 may include: calculating the abscissa of the corresponding pixel in the right image based on the initial disparity or the disparity obtained in the previous iteration and the abscissa of the pixel in the left image; constructing a query index set based on the abscissa of the corresponding pixel in the right image; determining the corresponding cost parameter in the cost space through the query index set and using the cost parameter as a feedback feature.

[0087] As a non-restrictive example, firstly, assuming the disparity value estimated in the previous iteration is d, then for each pixel x = (u, v) in the left image, a corresponding pixel x′ = (ud(u, v), v) can be found in the right image, with the corresponding x-coordinate in the right image being u′ = ud(u, v). Then, based on the x-coordinate in the right image u′ = ud(u, v), a query index set N(u') is constructed in the cost space:

[0088] N(u′)={u′+dx|dx∈Z,‖dx‖1≤r},

[0089] Where r represents the query range, Z represents the set of integers in the query index set N(u'), dx∈Z means dx belongs to integers, and ||dx||1≤r means that the L1 norm of dx is less than or equal to r. The corresponding cost parameters can be determined in the cost space using this query index set N(u') to obtain the feedback features.

[0090] In one embodiment, the update module is used to implement step S370. Specifically, it updates the fused features and the left image features f. L The hidden features, along with the initial hidden features or the hidden features from the previous iteration, are input into the update module, which outputs the hidden features for the current iteration and the estimated disparity for the current iteration. The update module can use a sequence model such as LSTM or GRU.

[0091] It should be noted that the iterative module in this embodiment does not use a warp operation. Instead, it queries the cost parameter corresponding to the disparity estimated in the previous iteration in the cost space as a feedback feature. Furthermore, the relevant module only needs to run once; that is, step S330 is executed only once, and the resulting cost space C... corr It can be reused in subsequent iterations.

[0092] In other variant embodiments, the reference image can be changed to the image on the right. In this case, as... Figure 10 As shown, Figure 8In the illustrated embodiment, step S370 can be replaced by: S371, calculating the hidden features and disparity of the current iteration based on the feedback features, the features in the right figure, and the initial hidden features or the hidden features obtained in the previous iteration.

[0093] In some embodiments, such as Figure 11 As shown, the iterative module of the stereo matching network may include a cost space query module and an update module. In one embodiment, the cost space query module is used to implement step S360. Specifically, step S360 may include: calculating the abscissa of the corresponding pixel in the left image based on the initial disparity or the disparity obtained in the previous iteration and the abscissa of the pixel in the right image; determining the query index set based on the abscissa of the corresponding pixel in the left image; and retrieving the corresponding cost parameter in the cost space through the query index set, using the cost parameter as a feedback feature.

[0094] In one embodiment, the update module is used to implement step S371. Specifically, step S371 includes: calculating the hidden features and disparity of the current iteration based on the feedback features, the right-side image features, and the initial hidden features or the hidden features obtained from the previous iteration. It should be noted that, for the first iteration, the hidden features of the previous iteration are the initial hidden features. Preferably, the right-side image features f can be used. R Alternatively, a feature with all zeros can be used as the initial hidden feature.

[0095] It should be understood that, Figure 10 and Figure 11 The variant embodiments shown are the same as those described above. Figure 8 and Figure 9 The working principle of the embodiments shown is the same, and those skilled in the art can implement this variant embodiment based on the foregoing embodiments, which will not be repeated here.

[0096] In summary, compared with existing technologies, the embodiments of this application can output disparities at different resolutions using only the same network, which brings great convenience to practical applications. Furthermore, the iterative structure allows the disparity map estimated by the stereo matching network to have higher accuracy and better robustness.

[0097] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0098] An embodiment of this application also provides a stereo matching device. For details not described in the preceding embodiments of the stereo matching method, please refer to the description of the stereo matching device.

[0099] Figure 12This is a schematic diagram of the structure of a stereo matching device according to an embodiment of this application. See also... Figure 12 The stereo matching device includes: an acquisition module 121, a feature extraction module 122, a correlation module 123, an upsampling module 124, and a disparity calculation module 125; wherein, the acquisition module 121 is used to acquire the left and right images captured by the binocular camera; the feature extraction module 122 is used to extract the left image features of the left image and the right image features of the right image; the correlation module 123 is used to perform correlation calculation on the left and right image features to obtain the cost space; the upsampling module 124 is used to upsample the cost space to the target resolution; and the disparity calculation module 125 is used to estimate the stereo matching disparity based on the upsampled cost space to obtain the target disparity at the target resolution.

[0100] Figure 13 This is a schematic diagram of a stereo matching device according to another embodiment of this application. See also... Figure 13 The stereo matching device includes, in addition to Figure 12 The shown acquisition module 121, feature extraction module 122, correlation module 123, upsampling module 124, and disparity calculation module 125 also include one or more iteration modules 126. Figure 13 Only one is shown in the image.

[0101] In some embodiments, the iteration module 126 is configured to perform an image affine transformation on the right image features based on the initial disparity or the disparity obtained in the previous iteration to obtain ideal left image features; fuse the ideal left image features and the left image features to obtain fused features; and calculate the hidden features and disparity of the current iteration based on the fused features, the left image features, and the initial hidden features or the hidden features obtained in the previous iteration.

[0102] In some other embodiments, the iteration module 126 is used to perform an image affine transformation on the left image features based on the initial disparity or the disparity obtained in the previous iteration to obtain ideal right image features; to fuse the ideal right image features and the right image features to obtain fused features; and to calculate the hidden features and disparity of the current iteration based on the fused features, the right image features, and the initial hidden features or the hidden features obtained in the previous iteration.

[0103] In some other embodiments, the iteration module 126 is used to calculate the abscissa of the corresponding pixel in the right image based on the initial disparity or the disparity obtained in the previous iteration and the abscissa of the pixel in the left image; determine the query index set based on the abscissa of the corresponding pixel in the right image; take the corresponding cost parameter in the cost space through the query index set, and use the cost parameter as the feedback feature; calculate the hidden feature and disparity of the current iteration based on the feedback feature, the left image feature and the initial hidden feature or the hidden feature obtained in the previous iteration.

[0104] In some other embodiments, the iteration module 126 is used to calculate the abscissa of the corresponding pixel in the left image based on the initial disparity or the disparity obtained in the previous iteration and the abscissa of the pixel in the right image; determine the query index set based on the abscissa of the corresponding pixel in the left image; take the corresponding cost parameter in the cost space through the query index set, and use the cost parameter as the feedback feature; calculate the hidden feature and disparity of the current iteration based on the feedback feature, the right image feature and the initial hidden feature or the hidden feature obtained in the previous iteration.

[0105] In one embodiment, the stereo matching device may further include left and right cameras for acquiring left and right images of the target area, respectively. Further, the stereo matching device may also include a projection module for projecting a patterned light beam onto the target area, with the left and right cameras acquiring the patterned light beam reflected back from the target area, wherein the patterned light beam may be speckle, stripes, or coded structured light, and is not limited thereto.

[0106] One embodiment of this application also provides an electronic device, such as... Figure 14 As shown, the electronic device may include one or more processors 100. Figure 14 (Only one is shown in the image), a memory 101, and a computer program 102 stored in the memory 101 and executable on one or more processors 100, such as a stereo matching program. When one or more processors 100 execute the computer program 102, they can implement the various steps in the stereo matching method embodiment. Alternatively, when one or more processors 100 execute the computer program 102, they can implement the functions of the various modules / units in the stereo matching apparatus embodiment, which is not limited here.

[0107] Those skilled in the art will understand that Figure 14 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. Electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different components. For example, electronic devices may also include input / output devices, network access devices, buses, etc.

[0108] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0109] In one embodiment, memory 101 may be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory 101 may also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 101 may include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the electronic device. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] An embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps in the stereo matching method embodiment.

[0112] One embodiment of this application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the various steps in the stereo matching method embodiment.

[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0118] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0119] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A stereo matching method, characterized in that, include: The left and right images captured by the left and right cameras are acquired, and the left and right images are downsampled by n times to extract the corresponding left and right image features. The correlation between the features of the left image and the features of the right image at different disparities is calculated to obtain the cost space; the spatial resolution of the cost space is H / n × W / n, where H and W are the resolutions of the left image and the right image, respectively; Obtain the target resolution H'×W' input by the user, where the target resolution is a variable resolution; The cost space is upsampled to the target resolution; wherein the target resolution is a variable resolution or an arbitrary resolution; Based on the cost space estimation after upsampling, the stereo matching disparity is estimated, and the stereo matching disparity is iteratively calculated multiple times using the initial disparity and initial hidden features or the disparity and hidden features obtained in the previous iteration to obtain the target disparity at the target resolution. Wherein, the initial disparity is the disparity obtained from the first stereo matching, the initial hidden feature is any one of the features of the left image, the features of the right image, or a feature that is all zero, the cost space is reused in the multiple iterations of calculation, and the correlation calculation is performed only once.

2. The stereo matching method as described in claim 1, characterized in that, The iterative calculation process includes: Based on the initial disparity or the disparity obtained in the previous iteration, perform an image affine transformation on the right image features to obtain the ideal left image features; The ideal left image features and the left image features are fused to obtain the fused features; The hidden features and disparity of the current iteration are calculated based on the fused features, the left image features, and the initial hidden features or the hidden features obtained from the previous iteration; wherein, the hidden features are feedback information obtained by combining the fused features, the left image features, and the initial hidden features or the hidden features obtained from the previous iteration with the physical constraints of binocular stereo vision.

3. The stereo matching method as described in claim 1, characterized in that, The iterative calculation process includes: Based on the initial disparity or the disparity obtained in the previous iteration, perform an affine transformation on the features of the left image to obtain the ideal features of the right image. The ideal right image features and the right image features are fused to obtain the fused features; The hidden features and disparity of the current iteration are calculated based on the fused features, the right image features, and the initial hidden features or the hidden features obtained from the previous iteration; wherein, the hidden features are feedback information obtained by combining the fused features, the left image features, and the initial hidden features or the hidden features obtained from the previous iteration with the physical constraints of binocular stereo vision.

4. The stereo matching method as described in claim 1, characterized in that, The iterative calculation process includes: Query the cost parameters corresponding to the initial disparity or the disparity obtained in the previous iteration in the cost space, and use the cost parameters as feedback features; The hidden features and disparity of the current iteration are calculated based on the feedback features, the left image features or the right image features, and the initial hidden features or the hidden features obtained from the previous iteration; the hidden features are feedback information obtained by combining the feedback features, the left image features or the right image features, and the initial hidden features or the hidden features obtained from the previous iteration with the physical constraints of binocular stereo vision.

5. The stereo matching method as described in claim 4, characterized in that, The cost parameter corresponding to the disparity obtained in the previous iteration in the cost space is used as a feedback feature, including: The x-coordinate of the corresponding pixel in the right image is calculated based on the initial disparity or the disparity obtained in the previous iteration and the x-coordinate of the pixel in the left image. Determine the query index set based on the x-coordinate of the corresponding pixel in the right image; The corresponding cost parameters are obtained from the cost space using the query index set, and these cost parameters are used as feedback features.

6. The stereo matching method as described in claim 4, characterized in that, The cost parameter corresponding to the disparity obtained in the previous iteration in the cost space is used as a feedback feature, including: The horizontal coordinates of the corresponding pixels in the left image are calculated based on the initial disparity or the disparity obtained in the previous iteration and the horizontal coordinates of the pixels in the right image. The query index set is determined based on the x-coordinate of the corresponding pixel in the left image; The corresponding cost parameters are obtained from the cost space using the query index set, and these cost parameters are used as feedback features.

7. The stereo matching method according to any one of claims 1 to 6, characterized in that, The estimation of stereo matching disparity based on the upsampled cost space includes: The upsampled cost space is subjected to a softargmin operation or a fully connected operation in the channel dimension to estimate the stereo matching disparity.

8. A stereo matching device, characterized in that, include: The acquisition module is used to acquire the left and right images captured by the left and right cameras; The feature extraction module is used to perform downsampling of the left image and the right image by n times to extract the corresponding left image features and right image features; The correlation module is used to calculate the correlation between the features of the left image and the features of the right image at different disparities to obtain the cost space; the spatial resolution of the cost space is H / n × W / n, where H and W are the height and width of the left image and the right image, respectively; An upsampling module is used to upsample the cost space to a target resolution; wherein the target resolution is a variable resolution or an arbitrary resolution; The disparity calculation module is used to estimate the stereo matching disparity based on the upsampled cost space, and to perform multiple iterative calculations on the stereo matching disparity using the initial disparity and initial hidden features or the disparity and hidden features obtained in the previous iteration to obtain the target disparity at the target resolution. Wherein, the initial disparity is the disparity obtained from the first stereo matching, the initial hidden feature is any one of the features of the left image, the features of the right image, or a feature that is all zero, the cost space is reused in the multiple iterations of calculation, and the correlation calculation is performed only once.

9. The stereo matching device as described in claim 8, characterized in that, It also includes left and right cameras, which are used to capture the left and right images of the target area; or, It also includes a projection module and left and right cameras. The projection module is used to project a patterned light beam onto the target area, and the left and right cameras are used to receive the patterned light beam reflected back from the target area and generate left and right images.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the stereo matching method as described in any one of claims 1 to 7.

11. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the stereo matching method as described in any one of claims 1 to 7.

Images