Stereo matching method and apparatus, binocular depth camera, and electronic device

Abstract

The present application relates to the field of image processing, and provides a stereo matching method and apparatus, a binocular depth camera, and an electronic device. The method comprises: obtaining a first image and a second image; determining a first window in the first image; obtaining N first parameter sets; on the basis of the first window and the N first parameter sets, determining N second windows in the second image; on the basis of the first window and the N second windows, determining N matching costs, and using a second window corresponding to a smallest matching cost among the N matching costs as a target matching window; and on the basis of an offset of a first parameter set corresponding to the target matching window, determining a first parallax. On the basis of the method, a point having a same name corresponding to a pixel within a first window range can be accurately found in the second image, and a parallax value between points having the same names corresponding to two window ranges is determined on the basis of the offset in the first parameter set, so as to determine a parallax map, so that the problem of large-area missing of the determined parallax map can be avoided.

Description

Stereo matching method, device, binocular depth camera and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 15, 2023, with application number 202311197456.7 and entitled “Stereo matching method, device, binocular depth camera and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing, and more specifically, to a method, device, binocular depth camera, and electronic device for stereo matching. Background Art

[0003] In binocular speckle stereo matching algorithms, it is usually necessary to set a rectangular window in the left image as a matching window. Through the matching operation, the target window in the right image that is most similar to the matching window is found. However, this method requires that the depth information of all pixels within the rectangular window be the same. However, this premise cannot be met in some specific scenarios. For example, when the left and right images include inclined surfaces with tilted angles, a rectangular window with specific geometric parameters is used to determine the matching window at the pixel to be matched in the left image. If a rectangular window with the same geometric parameters is still used in the right image to search and match for the same-name points of the pixel to be matched, it will be difficult or even impossible to match the same-name points in the right image, resulting in large areas of missing information in the disparity map obtained after matching.

[0004] In view of this, a binocular vision stereo matching method suitable for target scenes including inclined planes is urgently needed.

[0005] Summary of the Invention

[0006] The present application provides a stereo matching method, apparatus, binocular depth camera, and electronic device, which adjust the geometric parameters of the window used in the image to be matched and determine a new window to be applied to a specified position of the image to be matched, thereby enabling better stereo matching of binocular images. Even when the binocular image includes a slope, the same-name pixel points corresponding to the sloped portion in the binocular image can be matched more accurately, avoiding large-area omissions in the disparity map obtained by matching.

[0007] In a first aspect, a stereo matching method is provided, the method comprising: acquiring a first image and a second image, the first image and the second image being images taken by a binocular depth camera; determining a first window in the first image; acquiring N groups of first parameter sets, the first parameter sets comprising a skew angle, a scaling factor, and an offset, where N is an integer greater than 1; determining N second windows in the second image based on the first window and the N groups of first parameter sets; determining N matching costs based on the first window and the N second windows, and taking the second window corresponding to the smallest matching cost among the N matching costs as a target matching window; determining a first disparity based on the offset in the first parameter set corresponding to the target matching window, the first disparity being the disparity between each pixel point within the first window range and each corresponding point of the same name within the target matching window range, the first disparity being used to determine a first disparity map corresponding to the first image and the second image.

[0008] For example, the first image and the second image may be RGB images or infrared ray (IR) images, and the specific image format is related to the type of device that obtains the image.

[0009] For example, since the first image and the second image are images taken by a binocular depth camera, the first image and the second image should include the same target scene, which may include a slope, which refers to a surface with an inclined angle. The angle between the slope and the surface it intersects with can be any reasonable angle, such as 45 degrees, 90 degrees, 135 degrees, etc.

[0010] For example, the first window may be any polygon, but usually, the first window is a rectangle.

[0011] For example, each parameter in the above-mentioned first parameter set is a range of each parameter, such as a bevel angle range, a scaling factor range and an offset range, and after a discretization operation is performed, the determined multiple bevel angles, multiple scaling factors and multiple offsets are combined with each other to determine the above-mentioned N groups of first parameter sets.

[0012] For example, the bevel angle range, scaling factor range, and offset range can be discretized by evenly dividing each parameter range and taking the upper limit, lower limit, or middle value of the evenly divided sub-range as the discrete value. Alternatively, multiple discrete values ​​can be sampled within each parameter range based on Gaussian distribution sampling.

[0013] It should be understood that the purpose of determining the matching cost is to measure the correlation between a specified pixel to be matched (a pixel in the first image) and a specified candidate pixel (a pixel in the second image). Regardless of whether two pixels are homonymous points (also known as homonymous pixels), the matching cost function can be used to calculate the matching cost. The lower the cost, the greater the correlation and the greater the probability of being homonymous points. Therefore, using the second window corresponding to the smallest matching cost among N matching costs for stereo matching achieves the best results.

[0014] Based on the above technical solution, by adjusting the geometric parameters of the window used in the first image, multiple second windows are determined, and by calculating the matching cost, a target matching window is screened from the multiple second windows. Based on the target matching window, the same-name points corresponding to the pixel points within the first window range can be accurately found in the second image, and the offset in the first parameter set corresponding to the target matching window can be directly used as the disparity value between the corresponding same-name points within the two window ranges to determine the disparity map, which helps to avoid the problem of missing or poor accuracy of the determined disparity map.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned beveling angle is used to beveling the first window, the above-mentioned scaling factor is used to scale the first window, and the above-mentioned offset is used to indicate the offset between the position of the pixel point with the same name in the second image and the position in the first image.

[0016] In combination with the first aspect, in certain implementations of the first aspect, iterative propagation is performed for each pixel point in the first image and the second image, including: determining a propagation order of iterative propagation; determining a first pixel point and a second pixel point in the first image according to the propagation order, with the second pixel point being the one before the first pixel point in the propagation order; determining a first homonymous point corresponding to the first pixel point in the second image according to a first parameter set corresponding to the first pixel point, and determining a first matching cost between the first pixel point and the first homonymous point; determining a second homonymous point corresponding to the first pixel point in the second image according to the first parameter set corresponding to the second pixel point, and determining a second matching cost between the first pixel point and the second homonymous point; when the second matching cost is less than the first matching cost, replacing the first parameter set corresponding to the first pixel point with the first parameter set corresponding to the second pixel point.

[0017] For example, the iterative propagation refers to performing iterative propagation for each pixel in the first image and the second image respectively, and may include four ordered iterations, and the propagation direction of each iteration is different. For example, the propagation directions are from left to right, from top to bottom, from right to left, and from bottom to top, respectively. In the propagation process from left to right and from right to left, the pixels in each row of the first image and the second image may be iterated in sequence, or the pixels in multiple rows may be iterated in parallel; in the propagation process from top to bottom and from bottom to top, the pixels in each column of the first image and the second image may be iterated in sequence, or the pixels in multiple columns may be iterated in parallel.

[0018] For example, the number of iterations of the iterative propagation can be other numbers. In addition, the direction order of the iterative propagation can be any combination of other orders, but the number of directions of the subsequent iterative propagation needs to be different from the direction order of the previous iterative propagation.

[0019] Based on the above technical solution, the iterative parameters corresponding to each pixel in the first and second images can be updated in their entirety, further calibrating the first parameter set corresponding to each pixel. Because the offset in this first parameter set represents the offset between the position of the corresponding pixel in the second image and the position in the first image of the same name, the disparity map can be more accurately determined based on this updated offset.

[0020] In combination with the first aspect, in certain implementations of the first aspect, M disparity fine-tuning parameters are determined; the M disparity fine-tuning parameters are respectively added to the first disparity corresponding to the third pixel point to determine M calculation results, where the third pixel point is any pixel point in the first window; the M calculation results are respectively replaced with the offset in the first parameter set corresponding to the third pixel point to determine M second parameter sets; based on the M second parameter sets, M third homonymous points corresponding to the third pixel point in the second image are determined, and M third matching costs between the third pixel point and the M third homonymous points are determined; based on the offset in the scaling factor set corresponding to the smallest third matching cost among the M third matching costs, the second disparity is determined, and the first disparity is replaced with the second disparity.

[0021] For example, after performing the above-mentioned offset fine-tuning operation on all pixels in the binocular image, a first disparity map corresponding to the first image and the second image can be determined based on the current disparity value corresponding to each pixel, where each pixel in the first disparity map corresponds to a disparity value.

[0022] Based on the above technical solution, the accuracy of the offset in the first parameter set is further improved, which helps to improve the accuracy of determining the disparity map.

[0023] In combination with the first aspect, in certain implementations of the first aspect, after determining the first disparity map, a first adjacent pixel point adjacent to a fourth pixel point is determined, where the fourth pixel point is any pixel point in the first disparity map; the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point are obtained; when the difference between the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point is greater than a preset difference range, the first adjacent pixel point is removed from the first disparity map to determine a second disparity map.

[0024] The second disparity map is the final disparity map output.

[0025] Based on the above technical solution, the accuracy of the disparity map can be further improved, and the noise or speckle interference in the disparity map can be reduced.

[0026] In a second aspect, a stereo matching device is provided, which includes: an acquisition unit for acquiring a first image and a second image, the first image and the second image being images taken by a binocular depth camera; a processing unit for determining a first window in the first image; acquiring N groups of first parameter sets, the first parameter set including a skew angle, a scaling factor and an offset, where N is an integer greater than 1; determining N second windows in the second image based on the first window and the N groups of first parameter sets; determining N matching costs based on the first window and the N second windows, and taking the second window corresponding to the smallest matching cost among the N matching costs as a target matching window; determining a first disparity based on the offset in the first parameter set corresponding to the target matching window, the first disparity being the disparity between each pixel point within the first window range and each corresponding point of the same name within the target matching window range, and the first disparity being used to determine a first disparity map corresponding to the first image and the second image.

[0027] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned beveling angle is used to beveling the first window, the above-mentioned scaling factor is used to scale the first window, and the above-mentioned offset is used to indicate the offset between the position of the pixel point with the same name in the second image and the position in the first image.

[0028] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing unit is also used to perform iterative propagation for each pixel point in the first image and the second image, including: determining the propagation order of iterative propagation; determining the first pixel point and the second pixel point in the first image according to the propagation order, and the second pixel point is ranked before the first pixel point in the propagation order; determining the first homonymous point corresponding to the first pixel point in the second image according to the first parameter set corresponding to the first pixel point, and determining the first matching cost between the first pixel point and the first homonymous point; determining the second homonymous point corresponding to the first pixel point in the second image according to the first parameter set corresponding to the second pixel point, and determining the second matching cost between the first pixel point and the second homonymous point; when the second matching cost is less than the first matching cost, replacing the first parameter set corresponding to the first pixel point with the first parameter set corresponding to the second pixel point.

[0029] In combination with the second aspect, in some implementations of the second aspect, the above-mentioned processing unit is also used to determine M disparity fine-tuning parameters; add the M disparity fine-tuning parameters to the first disparity corresponding to the third pixel point respectively to determine M calculation results, where the third pixel point is any pixel point in the first window; replace the offset in the first parameter set corresponding to the third pixel point with the M calculation results respectively to determine M second parameter sets; determine M third homonymous points corresponding to the third pixel point in the second image based on the M second parameter sets, and determine M third matching costs between the third pixel point and the M third homonymous points respectively; determine the second disparity based on the offset in the scaling factor set corresponding to the smallest third matching cost among the M third matching costs, and replace the first disparity with the second disparity.

[0030] In combination with the second aspect, in certain implementations of the second aspect, after determining the first disparity map, the above-mentioned processing unit is also used to determine a first adjacent pixel point adjacent to a fourth pixel point, where the fourth pixel point is any pixel point in the first disparity map; obtain the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point; when the difference between the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point is greater than a preset difference range, remove the first adjacent pixel point from the first disparity map to determine the second disparity map.

[0031] In a third aspect, a stereo matching device is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation of the method design of the first aspect above.

[0032] In a fourth aspect, a binocular depth camera is provided, including: a first receiving module for acquiring a first image; a second receiving module for acquiring a second image; a processor for determining a first window in the first image; acquiring N groups of first parameter sets, the first parameter set including a skew angle, a scaling factor and an offset, where N is an integer greater than 1; determining N second windows in the second image based on the first window and the N groups of first parameter sets; determining N matching costs based on the first window and the N second windows, and taking the second window corresponding to the smallest matching cost among the N matching costs as the target matching window; determining a first disparity based on the offset in the first parameter set corresponding to the target matching window, the first disparity being the disparity between each pixel point within the first window range and each corresponding point of the same name within the target matching window range, and the first disparity being used to determine a first disparity map corresponding to the first image and the second image.

[0033] In combination with the fourth aspect, in certain implementations of the fourth aspect, the above-mentioned beveling angle is used to beveling the first window, the above-mentioned scaling factor is used to scale the first window, and the above-mentioned offset is used to indicate the offset between the position of the pixel point with the same name in the second image and the position in the first image.

[0034] In combination with the fourth aspect, in certain implementations of the fourth aspect, the above-mentioned processor is also used to perform iterative propagation for each pixel point in the first image and the second image, including: determining the propagation order of iterative propagation; determining the first pixel point and the second pixel point in the first image according to the propagation order, and the second pixel point is ranked before the first pixel point in the propagation order; determining the first homonymous point corresponding to the first pixel point in the second image according to the first parameter set corresponding to the first pixel point, and determining the first matching cost between the first pixel point and the first homonymous point; determining the second homonymous point corresponding to the first pixel point in the second image according to the first parameter set corresponding to the second pixel point, and determining the second matching cost between the first pixel point and the second homonymous point; when the second matching cost is less than the first matching cost, replacing the first parameter set corresponding to the first pixel point with the first parameter set corresponding to the second pixel point.

[0035] In combination with the fourth aspect, in certain implementations of the fourth aspect, the above-mentioned processor is further used to determine M disparity fine-tuning parameters; add the M disparity fine-tuning parameters to the first disparity corresponding to the third pixel point respectively to determine M calculation results, where the third pixel point is any pixel point in the first window; replace the offset in the first parameter set corresponding to the third pixel point with the M calculation results respectively to determine M second parameter sets; determine M third homonymous points corresponding to the third pixel point in the second image based on the M second parameter sets, and determine M third matching costs between the third pixel point and the M third homonymous points respectively; determine the second disparity based on the offset in the scaling factor set corresponding to the smallest third matching cost among the M third matching costs, and replace the first disparity with the second disparity.

[0036] In combination with the fourth aspect, in certain implementations of the fourth aspect, after determining the first disparity map, the above-mentioned processor is also used to determine a first adjacent pixel point adjacent to a fourth pixel point, where the fourth pixel point is any pixel point in the first disparity map; obtain the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point; when the difference between the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point is greater than a preset difference range, remove the first adjacent pixel point from the first disparity map to determine the second disparity map.

[0037] In a fifth aspect, an electronic device is provided, comprising a device in any possible implementation of the device design of the second aspect above, or a binocular depth camera in any possible implementation of the device design of the fourth aspect above.

[0038] In the sixth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to echo a signal from the memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method in any possible implementation of the method design of the first aspect above.

[0039] In a seventh aspect, a computer-readable storage medium is provided, storing a computer program or instruction, which is used to implement the method in any possible implementation manner in the method design of the first aspect.

[0040] In an eighth aspect, a computer program product is provided. When the computer program code or instructions are executed on a computer, the computer executes a method in any possible implementation of the method design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a schematic diagram of a stereo matching method commonly used at this stage;

[0042] FIG2 is a schematic diagram of a binocular depth camera 200 proposed in an embodiment of the present application;

[0043] FIG3 is a schematic diagram of an electronic device 300 proposed in an embodiment of the present application;

[0044] FIG4 is a flow chart of a stereo matching method 400 proposed in an embodiment of the present application;

[0045] FIG5 is a schematic diagram of determining multiple groups of first parameter sets proposed in an embodiment of the present application;

[0046] FIG6 is a schematic diagram of determining a second window according to an embodiment of the present application;

[0047] FIG7 is a diagram showing the expected effect of a stereo matching method proposed in an embodiment of the present application;

[0048] FIG8 is a flow chart of an iterative propagation method 800 proposed in an embodiment of the present application;

[0049] FIG9 is a flow chart of a method 900 for fine-tuning an offset according to an embodiment of the present application;

[0050] FIG10 is a flow chart of a method 1000 for denoising a disparity map according to an embodiment of the present application;

[0051] FIG11 is a schematic block diagram of a stereo matching apparatus 1100 provided in an embodiment of the present application;

[0052] FIG12 is a schematic block diagram of a computer device 1200 provided in an embodiment of the present application;

[0053] FIG13 is a schematic block diagram of a computer-readable storage medium 1300 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is a kind of association relationship that describes associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0055] In the embodiments of this application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity, or content of the described objects. The use of prefixes such as ordinal numbers in the embodiments of this application to distinguish description objects does not constitute a limitation on the described objects. For a statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0056] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0057] Stereo matching algorithms, also known as disparity algorithms, aim to determine the disparity between two images. This involves finding corresponding points of the same name in images from different viewpoints. Stereo matching between images from different viewpoints is the primary method for obtaining disparity maps and is commonly used in object recognition scenarios using binocular vision devices.

[0058] Figure 1 is a schematic diagram of a currently used stereo matching method. Figure 1 includes a left image and a right image, both of which correspond to the same target space. This example uses a pixel to be matched in the left image as a reference to determine a first window. Based on this first window, stereo matching is performed on pixels in both images to find a point in the right image that corresponds to the pixel to be matched.

[0059] Currently used binocular-based stereo matching methods typically use a rectangular window to match corresponding points in the left and right images. For example, if any point M is taken in the target space, then the corresponding point m1 in the left image and point m2 in the right image are considered to be points of the same name.

[0060] However, referring to Figure 1, it can be seen that the left image and the right image include a slope, and the above-mentioned pixel points to be matched selected in the left image are pixels corresponding to the slope. Correspondingly, the above-mentioned first window is also located at the slope position of the left image. Based on the imaging principle of the binocular device, it can be seen that the information of the pixels included in the first window in the left image and the information of the pixels included in the first window in the right image are obviously different. Then in this scenario, if the matching operation between the left image and the right image is still performed through the first window, it is impossible to find the same-name point m2 corresponding to the point m1 in the left image. Normally, the above-mentioned matching operation refers to sliding the first window in the horizontal direction in the right image with the position of the first window in the left image as the reference to find the same-name point m2 as the point m1. However, because point m2 in the right image is located outside the range covered by the horizontal sliding of the first window, even if all the pixel points within the sliding range of the first window in the right image are matched with point m1, the point m2 with the same name cannot be found, or other pixel points in the first window that slides to a certain position in the right image are considered to be the corresponding points with the same name of point m1, resulting in the disparity map obtained based on the above stereo matching process being partially missing or having poor accuracy.

[0061] In view of this, for target scenes that are common in real scenes, including inclined surfaces, an embodiment of the present application proposes a stereo matching method. By adjusting the geometric parameters of the first window determined based on the pixel points to be matched in the first image, it is ensured that the second window obtained after adjustment can find the same-name points corresponding to the pixel points to be matched in the second image as accurately as possible, so as to determine the disparity between the same-name points in the two windows.

[0062] 2 is a schematic diagram of a binocular depth camera 200 proposed in an embodiment of the present application. The binocular depth camera 200 may include: a first receiving module, a second receiving module, and a depth calculation chip.

[0063] The first receiving module and the second receiving module can be RGB cameras, each used to capture an image of the target space and output an RGB image corresponding to the target space. The image output by the first receiving module can be called the left RGB image, and the image output by the second receiving module can be called the right RGB image. The RGB images are then input into the depth calculation chip.

[0064] In some possible embodiments, the first receiving module and the second receiving module may be modules of the same type or modules of different types.

[0065] The above-mentioned depth calculation chip is used to execute the stereo matching method proposed in the embodiment of the present application, thereby obtaining a disparity image of the target space, and further, a depth image can be obtained based on the disparity image.

[0066] In some possible embodiments, the binocular depth camera 200 may further include a memory, which is used to store intermediate parameters or output results determined by the depth calculation chip during the execution of the stereo matching method.

[0067] In some possible embodiments, the first receiving module and the second receiving module may also be infrared cameras. Correspondingly, the image output by the first receiving module may be referred to as the left IR image, and the image output by the second receiving module may be referred to as the right IR image. Based on this, to improve image acquisition accuracy, the binocular depth camera may further include a projection module configured to project a light beam comprising an encoded structured light pattern toward the target space. The structured light pattern may be an infrared speckle pattern, a stripe pattern, or a two-dimensional pattern.

[0068] 3 is a schematic diagram of an electronic device 300 according to an embodiment of the present application. The electronic device 300 may be a mobile phone, tablet, computer, or television, and the electronic device 300 integrates the above-mentioned binocular depth camera, thus having a depth calculation function.

[0069] In some possible embodiments, the depth calculation chip included in the binocular depth camera can be integrated into the processor of the electronic device 300. Under this architecture, the binocular depth camera can output left and right target RGB images or IR images. After receiving the images sent by the binocular depth camera, the processor of the electronic device 300 can perform depth calculation in addition to executing the stereo matching method proposed in the embodiment of the present application, thereby obtaining a depth image of the target space.

[0070] Based on the above-mentioned apparatus 200 or electronic device 300 , an embodiment of the present application proposes a stereo matching method 400 .

[0071] FIG4 is a flow chart of a stereo matching method 400 proposed in an embodiment of the present application.

[0072] S410: Acquire a first image and a second image.

[0073] Among them, the first image and the second image are images taken by a binocular depth camera, so the first image and the second image should include the same target scene, which may include a slope. The slope refers to a surface with an inclined angle, and the angle between the slope and the surface it intersects with can be any reasonable angle, such as 45 degrees, 90 degrees, 135 degrees, etc.

[0074] In some possible embodiments, if the first image is the left image described in the above embodiment, then the second image is the right image described in the above embodiment, and vice versa.

[0075] In some possible embodiments, the first image and the second image may be RGB images or IR images. The specific image format is related to the type of device that obtains the image, and the embodiments of the present application do not limit this.

[0076] S420: Determine a first window in the first image.

[0077] It should be understood that the first image mentioned above can be used as a reference image for stereo matching.

[0078] In some possible embodiments, the first window can be determined as follows: select the pixel points to be matched in the first image, and determine the coordinate information of the pixel points to be matched in the first image; determine the first window information based on the coordinate information, and the first window information includes the geometric parameters and position information of the first window; determine the first window based on the first window information.

[0079] In some possible embodiments, the first window may be a rectangular window, and the geometric parameters of the rectangular window may be represented by the lengths of the four sides of the rectangular window, thereby determining the coordinates of the four corner points of the rectangular window.

[0080] In some possible embodiments, the pixel to be matched can be the center point of the first window. For example, when the first window is a rectangular window, the pixel to be matched can be the intersection of the diagonals of the rectangular window. In this scenario, the pixel to be matched should be a non-edge pixel in the first image; or the pixel to be matched can be a corner point of the first window. In this scenario, the pixel to be matched can be any pixel in the first image. It should be understood that after determining the coordinate information of the pixel to be matched in the first image, it is determined whether the center point or corner point of the first window of the pixel to be matched can determine the geometric parameters and position information of the first window. For example, a coordinate system is established with the lower left corner of the first image as the origin, the coordinates of the pixel to be matched are (x1, y1), and the first window is determined with the pixel to be matched as the center point of the window. Then, the coordinates of the pixel to be matched can be used to represent the position information of the first window. Correspondingly, the geometric parameters of the first window can be as follows: the first window is a rectangle with a length of x2 and a width of y2.

[0081] In some possible embodiments, the first window may be any polygon, but typically, the first window is a rectangle. Furthermore, the first window has specific geometric parameters. For example, if the first window is a quadrilateral, the geometric parameters may be the lengths of the sides of the quadrilateral and the angles of the interior angles of the quadrilateral.

[0082] In some possible embodiments, the geometric parameters of the first window may be determined based on factors such as the detection accuracy of the binocular vision device and the operating speed of the processor.

[0083] S430: Acquire N groups of first parameter sets, where the first parameter sets include a bevel angle, a scaling factor, and an offset.

[0084] Among them, the bevel angle is used to bevel the first window, the scaling factor is used to scale the first window, the offset is used to indicate the offset between the position of the pixel point with the same name in the second image and the position in the first image, and N is an integer greater than 1.

[0085] In some possible embodiments, taking the first window as a rectangular window as an example, the first window can be beveled based on the above bevel angles, with both sides of the first window being tilted accordingly. That is, both sides of the first window are tilted as a whole horizontally, thereby converting the rectangular window into a parallelogram window by tilting and stretching both sides while maintaining the same width. The area of ​​the converted parallelogram window should be equal to that of the rectangular window before the conversion.

[0086] In some possible embodiments, the first window can be scaled based on the above-mentioned scaling factor, and the scaling direction can be the horizontal direction of the image, and the first window can be stretched or compressed based on the scaling factor. It can be seen that the area of ​​the first window will change before and after scaling.

[0087] It should be understood that the first image and the second image are set in a reference coordinate system, and the origin positions of the first image and the second image are the same. Assuming that the coordinates of the pixel point p1 to be matched in the first image are: (x1, y1), then the coordinates of the same-name point p2 of point p1 in the second image correspond to: (x1+dx, y1), where dx is the pixel offset between the first image and the second image. Since the first window is determined based on the pixel point p1 to be matched, the offset between all the same-name pixels within the first window and the target matching window is dx, that is, the coordinate relationship between all the same-name pixels satisfies the above-mentioned coordinate relationship between p1 and p2. The offset corresponding to each pixel in the window can be used to calculate the disparity between the first image and the second image.

[0088] In summary, the above-mentioned skew angle and the above-mentioned scaling factor are parameters for controlling the deformation degree of the above-mentioned first window, and the above-mentioned offset is a parameter for adjusting the position of the deformed first window in the second image.

[0089] In some possible embodiments, the N groups of first parameter sets may be determined by the following method:

[0090] It should be understood that the specific values ​​of the bevel angle, scaling factor and offset respectively included in the N groups of first parameter sets may be pre-set in the following manner.

[0091] Before determining the first parameter set, it is necessary to determine the bevel angle range, the scaling factor range, and the offset range.

[0092] Regarding the bevel angle range: The bevel angle range can be determined based on the target scene information, which may include the degree of inclination of the slope. For example, the bevel angle range can be [20, 90)°. The specific range depends on the specific situation and is not limited in this embodiment of the application.

[0093] Regarding the zoom factor range: the zoom factor range can be determined based on the target scene information, which may include the degree of inclination of the slope. For example, the zoom factor range can be [0.5, 2]. The specific range depends on the specific situation and is not limited in this embodiment of the application.

[0094] For the offset range: the offset range can be determined according to the baseline and measurement range of the binocular depth camera, for example (0, 2).

[0095] Then, the above-mentioned bevel angle range, scaling coefficient range, and offset range are discretized to determine i bevel angles, j scaling coefficients, and k offsets. Finally, these parameters are combined. Here, i, j, and k can be the same or different, and the specific values ​​of i, j, and k can be determined based on the processing power of the processor.

[0096] In some possible embodiments, the bevel angle range, scaling factor range, and offset range described above may be discretized by equally dividing each parameter range and taking the upper limit, lower limit, or middle value of the averaged sub-range as the discrete value. Alternatively, multiple discrete values ​​may be sampled within each parameter range based on Gaussian distribution sampling.

[0097] FIG5 is a schematic diagram of determining multiple groups of first parameter sets proposed in an embodiment of the present application.

[0098] Referring to FIG5 , in this example, the discretized bevel angles, scaling factors, and offsets are all the same, which is 3. The multiple bevel angles can be obtained by θ n To express, multiple scaling factors can be expressed by α n To indicate that multiple offsets can be expressed by dx n To represent, wherein n is greater than or equal to 1 and less than or equal to 3. After combining these parameters, a total of (3×3×3) first parameter sets are obtained.

[0099] Generally, when there are i bevel angles, j scaling coefficients, and k offsets determined through discretization, after combining these three parameters, a (i×j×k) first parameter set can be obtained.

[0100] In some possible embodiments, the discretized bevel angles may be recorded in a first discrete table, the discretized scaling factors may be recorded in a second discrete table, and the discretized offsets may be recorded in a third discrete table. Furthermore, multiple sets of first parameters obtained by combining these parameters may also be recorded in a matching window parameter table. The three discrete tables and the matching window parameter table may all be stored in a memory within the apparatus 200 or electronic device 300.

[0101] S440: Determine N second windows in the second image according to the first window and the N groups of first parameter sets.

[0102] Based on the above N groups of first parameter sets, the geometric parameters of the above first window are adjusted to achieve various deformation modes of the first window, such as compression, elongation and beveling, so as to obtain N second windows. Then, it is necessary to select a suitable window from the N second windows for stereo matching.

[0103] After the first parameter set is determined, the first window may be deformed to determine the second window.

[0104] Figure 6 is a schematic diagram of determining a second window according to an embodiment of the present application. In this example, the top-right corner point P of the first window is a rectangular window. Since the four corner points of the first window in this embodiment are converted in the same manner, the four corner points of the second window can be determined after determining the top-right corner point P. Based on this, all pixels within the first window can find corresponding points of the same name within the second window based on this conversion method.

[0105] It should be understood that in the example of FIG6 , the first parameter set should have been determined, that is, the above-mentioned bevel angle, scaling factor, and offset are known, and then the first window can be deformed based on these three parameters:

[0106] First, determine the coordinates of point P. Assuming the length of the first window is △x and the width is △y, the coordinates of point P are: (△x, △y). Usually, the parallax direction between the first image and the second image is along the x-axis. Based on this scenario, the process of determining the second window is as follows:

[0107] The bevel angle is known to be θ. Since the bevel angle can tilt the first window laterally, the abscissa of point P needs to be adjusted based on θ. Referring to FIG. 7 , the abscissa of point P1 is: Δx + (Δy / tanθ).

[0108] The scaling factor is known to be α. Since the scaling factor can horizontally scale the first window, the abscissa of point P needs to be adjusted based on α. ​​Referring to FIG7 , the abscissa of point P2 is: α△x+(△y / tanθ);

[0109] The offset is known as dx. Since the offset represents the offset between the first image and the second image, the horizontal coordinate of point P needs to be adjusted based on dx. Referring to Figure 7, the horizontal coordinate of point P3 is: α△x+(△y / tanθ)+dx;

[0110] To summarize, the top-right corner point P' of the second window corresponds to the top-right corner point P of the first window, and the coordinates of point P' are (α△x+(△y / tanθ)+dx,△y). The conversion method for the other three corner points of the first window is similar and will not be repeated here. By converting the other three corner points of the first window using the above method, the first window can be transformed into the second window.

[0111] It should be understood that, in the case where the pixel point to be matched for determining the first window is the center point of the first window, the conversion method described in the above embodiment is also applicable to determining the second window.

[0112] If the first image does not include a slope, or the first window does not include pixels constituting a slope, the bevel angle in the first parameter set finally selected may be 0°, the scaling factor may be 1, and the offset may be a non-zero value.

[0113] In some possible embodiments, the first window may be transformed in a purely radial deformation manner to determine the second window, that is, the second window may be determined by designing transformation operations such as scaling, rotation, translation, and shearing.

[0114] S450: Determine N matching costs according to the first window and the N second windows, and use the second window corresponding to the smallest matching cost among the N matching costs as a target matching window.

[0115] The purpose of determining the matching cost is to measure the correlation between the pixel to be matched (the pixel in the first image) and the candidate pixel (the pixel in the second image). Regardless of whether two pixels are homonymous, the matching cost function can be used to calculate the matching cost. The lower the cost, the greater the correlation and the greater the probability of being homonymous. Therefore, using the second window corresponding to the smallest matching cost among N matching costs for stereo matching is the most effective.

[0116] Furthermore, because the stereo matching method proposed in the embodiments of the present application requires discretization of the bevel angle range, scaling coefficient range, and offset range, multiple sets of first parameter sets can be obtained in the embodiments of the present application. Accordingly, multiple second windows can be determined. However, the effects of these second windows on stereo matching vary, with some having good stereo matching effects while others have relatively poor results. Therefore, it is necessary to perform the above-described matching process based on the pixels within the multiple second windows and the pixels within the first window, i.e., to calculate the matching costs corresponding to the multiple second windows and select the second window with the lowest matching cost as the target matching window for stereo matching.

[0117] In some possible embodiments, the matching cost may also be represented by a normalized cross-correlation coefficient (NCC), which is expressed as the following formula (1):

[0118] Where f(x, y) and t(x, y) represent the pixel samples in the first window and the second window, respectively. n is the total number of pixels in the matching window. σ represents the standard deviation of the pixel grayscale in the window. μ represents the average grayscale of the pixels in the window.

[0119] It should be understood that a larger NCC value indicates a stronger window correlation. Therefore, in practice, the matching cost is typically represented by 1-NCC. That is, a lower matching cost indicates a stronger correlation between multiple pairs of identical points within the first and second windows. The maximum NCC value is 1.

[0120] In some possible embodiments, the matching cost may also be represented by a grayscale difference or a square sum of grayscale values ​​between each pair of like-named points within the first window and the second window. The present application does not limit the specific method for determining the matching cost.

[0121] S460: Determine a first disparity according to the offset in the first parameter set corresponding to the target matching window.

[0122] The first disparity is the disparity between each pixel within the first window and each corresponding point of the same name within the target matching window, and the first disparity is used to determine a first disparity map corresponding to the first image. It should be understood that when determining the target matching window in the first image based on the second image, a second disparity view corresponding to the second image can be determined based on the disparity.

[0123] In some possible embodiments, after executing the above steps S410 to S460, it means that the same-name points corresponding to each pixel point within the first window have been found in the second image. Assuming that the first window is determined based on the first pixel point to be matched, it is necessary to subsequently select the second pixel point to be matched in the first image, repeat the above steps S410 to S460, redefine a first window, and then redefine a target matching window to obtain the offset between the corresponding same-name points within the redetermined first window and the target matching window, and then determine the disparity values ​​corresponding to these same-name points. This process is repeated. After completing the matching between all pixels in the first image and the second image, multiple disparity values ​​are determined, and then the corresponding first disparity map can be determined based on the multiple disparity values.

[0124] Figure 7 is a diagram showing the expected effect of a stereo matching method proposed in an embodiment of the present application. The target scene shown in Figure 7 is the same as the target scene described in Figure 1 above.

[0125] Referring to FIG7 , similar to the scene shown in FIG1 , the target space in the first image and the second image includes a slope, and a first window is determined with a pixel to be matched in the first image as the center. The first window is used to slide in the second image, and at least the same-name point m4 of point m3 within the first window of the first image cannot be matched. However, based on the stereo matching method 400 proposed in the embodiment of the present application, the first window is correspondingly cropped and scaled, and then the window is set at a corresponding position in the second image, and then the window is offset to determine the target matching window in the right image corresponding to the first window of the left image. The target matching window can include point m3, so stereo matching based on the second window for the second image will at least not cause matching failure or matching error of point m4, so points m3 and m4, which are points of the same name, will not be missing in the disparity map.

[0126] Based on the above technical solution, by adjusting the geometric parameters of the window used in the first image, multiple second windows are determined, and by calculating the matching cost, a target matching window is screened from the multiple second windows. Based on the target matching window, the same-name points corresponding to the pixel points within the first window range can be accurately found in the second image, and the offset in the first parameter set corresponding to the target matching window can be directly used as the disparity value between the corresponding same-name points within the two window ranges to determine the disparity map, which helps to avoid the problem of missing or poor accuracy of the determined disparity map.

[0127] It should be understood that after completing stereo matching of some pixels of the first image and the second image based on the first window and the target matching window, the deviation value in the first parameter set corresponding to the target matching window can be used as the disparity value and assigned to all pixels within the window range.

[0128] FIG8 is a flow chart of an iterative propagation method 800 proposed in an embodiment of the present application.

[0129] In some possible embodiments, iterative propagation is performed for each pixel in the first image and the second image.

[0130] Among them, the above-mentioned iterative propagation refers to iterative propagation for each pixel in the above-mentioned first image and second image respectively, and may include four ordered iterations, and the propagation direction of each iteration is different. For example, the propagation directions are from left to right, from top to bottom, from right to left, and from bottom to top, respectively. Among them, in the propagation process from left to right and from right to left, the pixels in each row of the first image and the second image can be iterated in sequence, or the pixels in multiple rows can be iterated in parallel; in the propagation process from top to bottom and from bottom to top, the pixels in each column of the first image and the second image can be iterated in sequence, or the pixels in multiple columns can be iterated in parallel.

[0131] Taking two random adjacent pairs of pixels with the same name in a binocular image as an example, the iterative propagation includes:

[0132] S810: Determine the propagation order of iterative propagation;

[0133] S820: Determine a first pixel point and a second pixel point in the first image according to the propagation order, where the second pixel point precedes the first pixel point in the propagation order;

[0134] S830: Determine, based on the first parameter set corresponding to the first pixel, a first like-named point corresponding to the first pixel in the second image, and determine a first matching cost between the first pixel and the first like-named point;

[0135] S840: Determine, based on the first parameter set corresponding to the second pixel, a second point of the same name in the second image corresponding to the first pixel, and determine a second matching cost between the first pixel and the second point of the same name;

[0136] S850: When the second matching cost is less than the first matching cost, replace the first parameter set corresponding to the first pixel point with the first parameter set corresponding to the second pixel point.

[0137] It should be understood that the first parameter set corresponding to all target matching windows determined by the method of the above embodiment can be used as the initial iteration parameters for iterative propagation of the binocular image. Therefore, the above method 800 corresponds to the first iteration of the iterative propagation. However, in subsequent iterations, the iteration parameters corresponding to each pixel can be the result parameters determined after the previous iterative calculation.

[0138] In some possible embodiments, the number of iterations of the iterative propagation can be other numbers. In addition, the direction order of the iterative propagation can be any combination of other orders, but the number of directions of the subsequent iterative propagation needs to be different from the direction order of the previous iterative propagation.

[0139] Based on the above technical solution, the iterative parameters corresponding to each pixel in the first and second images can be updated in their entirety, further calibrating the first parameter set corresponding to each pixel. Because the offset in this first parameter set represents the offset between the position of the corresponding pixel in the second image and the position in the first image of the same name, the disparity map can be more accurately determined based on this updated offset.

[0140] FIG9 is a flow chart of a method 900 for fine-tuning an offset proposed in an embodiment of the present application.

[0141] In some possible embodiments, in order to further improve the accuracy of the offset in the first parameter set, the following offset fine-tuning operation may be performed:

[0142] S910: Determine M parallax fine-tuning parameters.

[0143] In some possible embodiments, before determining the M parallax fine-tuning parameters, it is necessary to determine the offset fine-tuning interval and discretize the interval into M discrete values ​​a x , where x is an integer greater than 1 and less than or equal to M, and the offset fine-tuning interval can be a preset reasonable interval, which can include a positive interval and a negative interval.

[0144] S920: Add the M disparity fine-tuning parameters to the first disparity corresponding to the third pixel point respectively to determine M calculation results.

[0145] The third pixel point is any pixel point in the first window.

[0146] S930: Replace the offset in the first parameter set corresponding to the third pixel point with the M calculation results respectively to determine M second parameter sets.

[0147] For example, when x = 1, based on the operations in S920 and S930 above, the offset dx in the first parameter set is replaced with (dx + a1). Then, when x = 2, the above operations are repeated until x = M. M second parameter sets are thus determined. It should be understood that the second parameter set differs from the first parameter set only in the specific values ​​of the offsets.

[0148] S940: Determine, based on the M second parameter sets, M third like-named points corresponding to the third pixel in the second image, and determine M third matching costs between the third pixel and the M third like-named points, respectively.

[0149] S950: Determine a second disparity according to an offset in the scaling coefficient set corresponding to the smallest third matching cost among the M third matching costs, and replace the first disparity with the second disparity.

[0150] In some possible embodiments, after performing the above-mentioned offset fine-tuning operation on all pixels in the binocular image, a first disparity map corresponding to the first image can be determined based on the current disparity value corresponding to each pixel, where each pixel in the first disparity map corresponds to a disparity value.

[0151] Based on the above technical solution, the accuracy of the offset in the first parameter set is further improved, which helps to improve the accuracy of determining the disparity map.

[0152] FIG10 is a flow chart of a method 1000 for denoising a disparity map according to an embodiment of the present application.

[0153] In some possible embodiments, after determining the first disparity map, the disparity values ​​of each pixel in the first disparity map may be compared with those of adjacent pixels, thereby denoising the first disparity map:

[0154] S1010: Determine a first adjacent pixel point adjacent to a fourth pixel point, where the fourth pixel point is any pixel point in the first disparity map.

[0155] S1020: Obtain a disparity value corresponding to the fourth pixel point and a disparity value corresponding to the first adjacent pixel point.

[0156] S1030: When the difference between the disparity value corresponding to the fourth pixel and the disparity value corresponding to the first adjacent pixel is greater than a preset difference range, remove the first adjacent pixel from the first disparity map to determine a second disparity map.

[0157] The second disparity map is the disparity map finally outputted. The preset difference range is any reasonable difference range, and the present embodiment does not limit this.

[0158] Based on the above technical solution, the accuracy of the disparity map can be further improved, and the noise or speckle interference in the disparity map can be reduced.

[0159] It should be noted that, since the first pixel point, the third pixel point or the fourth pixel point mentioned in the embodiment of the present application, these three pixel points can be the same pixel point or different pixel points, and no specific pixel point is specified. They are all random pixel points in the image, or random pixel points within a certain range.

[0160] In addition, embodiments of the present application also provide a device for implementing any of the above methods. For example, a stereo matching device is provided, which includes a unit (or means) for implementing any of the above stereo matching methods.

[0161] FIG11 is a schematic block diagram of a stereo matching apparatus 1100 provided in an embodiment of the present application. The apparatus 1100 includes:

[0162] An acquiring unit 1110 is configured to acquire a first image and a second image, where the first image and the second image are images captured by a binocular depth camera;

[0163] Processing unit 1120 is used to determine a first window in a first image; obtain N groups of first parameter sets, the first parameter sets including a skew angle, a scaling factor, and an offset, where N is an integer greater than 1; determine N second windows in a second image based on the first window and the N groups of first parameter sets; determine N matching costs based on the first window and the N second windows, and use the second window corresponding to the smallest matching cost among the N matching costs as a target matching window; determine a first disparity based on the offset in the first parameter set corresponding to the target matching window, the first disparity being the disparity between each pixel point within the first window range and each corresponding point of the same name within the target matching window range, and the first disparity being used to determine a first disparity map corresponding to the first image and the second image.

[0164] In some possible embodiments, the above-mentioned beveling angle is used to beveling the first window, the above-mentioned scaling factor is used to scale the first window, and the above-mentioned offset is used to indicate the offset between the position of the pixel point with the same name in the second image and the position in the first image.

[0165] In some possible embodiments, the processing unit 1120 is further configured to perform iterative propagation for each pixel in the first image and the second image, including: determining an order of iterative propagation; determining a first pixel and a second pixel in the first image according to the order of propagation, with the second pixel being ranked before the first pixel in the order of propagation; determining a first point of the same name corresponding to the first pixel in the second image according to a first parameter set corresponding to the first pixel, and determining a first matching cost between the first pixel and the first point of the same name; determining a second point of the same name corresponding to the first pixel in the second image according to the first parameter set corresponding to the second pixel, and determining a second matching cost between the first pixel and the second point of the same name; and replacing the first parameter set corresponding to the first pixel with the first parameter set corresponding to the second pixel when the second matching cost is less than the first matching cost.

[0166] In some possible embodiments, the processing unit 1120 is further used to determine M disparity fine-tuning parameters; add the M disparity fine-tuning parameters to the first disparity corresponding to the third pixel point respectively to determine M calculation results, where the third pixel point is any pixel point in the first window; replace the offset in the first parameter set corresponding to the third pixel point with the M calculation results respectively to determine M second parameter sets; determine M third homonymous points corresponding to the third pixel point in the second image based on the M second parameter sets, and determine M third matching costs between the third pixel point and the M third homonymous points respectively; determine the second disparity based on the offset in the scaling factor set corresponding to the smallest third matching cost among the M third matching costs, and replace the first disparity with the second disparity.

[0167] In some possible embodiments, after determining the first disparity map, the above-mentioned processing unit 1120 is further used to determine a first adjacent pixel point adjacent to a fourth pixel point, where the fourth pixel point is any pixel point in the first disparity map; obtain the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point; when the difference between the disparity value corresponding to the fourth pixel point and the disparity value corresponding to the first adjacent pixel point is greater than a preset difference range, remove the first adjacent pixel point from the first disparity map to determine the second disparity map.

[0168] In some possible embodiments, the binocular depth camera 200 proposed in the embodiments of the present application may include the device 1100 described in any of the above embodiments. For example, the first receiving module and the second receiving module in the binocular depth camera 200 may include the acquisition unit 1110 under any embodiment of the device 1100, and the depth calculation chip in the binocular depth camera 200 may include the processing unit 1120 under any embodiment of the device 1100.

[0169] In some possible embodiments, the electronic device 300 proposed in the embodiments of the present application may also include the device 1100 described in any of the above embodiments, wherein the processor in the electronic device 300 may include the processing unit 1120 under any implementation mode of the above device 1100.

[0170] Figure 12 is a schematic block diagram of a computer device 1200 provided in an embodiment of the present application. The computer device 1200 shown in Figure 12 includes a memory 1210, a processor 1220, and a bus 1240. Optionally, the computer device 1200 also includes a communication interface 1230. The memory 1210, processor 1220, and communication interface 1230 are interconnected via bus 1240.

[0171] The memory 1210 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1210 may store programs. When the programs stored in the memory 1210 are executed by the processor 1220, the processor 1220 is configured to perform the various steps of the method according to the embodiments of the present application.

[0172] The processor 1220 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method proposed in the embodiment of the present application.

[0173] The processor 1220 may also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the method described in the embodiments of the present application may be completed through hardware integrated logic circuits in the processor 1220 or instructions in software form.

[0174] The processor 1220 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods involved in the embodiments of this application may be directly implemented as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1210. The processor 1220 reads the information in the memory 1210 and, in combination with its hardware, completes the functions required to be executed by the units included in the apparatus shown in FIG. 10 , or executes the methods described in the embodiments of the method of this application.

[0175] The communication interface 1230 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 1200 and other devices or a communication network.

[0176] The bus 1240 may include a path for transmitting information between the various components of the device 1200 (eg, the memory 1210 , the processor 1220 , and the communication interface 1230 ).

[0177] It should be understood that although the above-mentioned device 1200 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art will understand that the device 1200 may also include other components necessary for normal operation. At the same time, according to specific needs, those skilled in the art will understand that the device 1200 may also include hardware components that implement other additional functions. In addition, those skilled in the art will understand that the device 1200 may also include only the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in Figure 12.

[0178] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0179] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0180] Figure 13 is a schematic block diagram of a computer-readable storage medium 1300 provided in an embodiment of the present application. The computer-readable storage medium 1300 shown in Figure 13 stores computer instructions 1310. When executed by a processor, the computer instructions 1310 can implement the method corresponding to the above embodiment.

[0181] In some possible embodiments, the computer-readable storage medium 1300 can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes a collection of one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0182] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0187] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0188] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A stereo matching method, characterized in that: The method comprises: Acquire a first image and a second image, where the first image and the second image are images taken by a binocular depth camera; determining a first window in the first image; Acquire N groups of first parameter sets, where the first parameter sets include a bevel angle, a scaling factor, and an offset, and N is an integer greater than 1; Determine N second windows in the second image according to the first window and the N groups of first parameter sets; Determine N matching costs according to the first window and the N second windows, and use the second window corresponding to the smallest matching cost among the N matching costs as a target matching window; A first disparity is determined according to an offset in a first parameter set corresponding to the target matching window, wherein the first disparity is a disparity between each pixel point within the first window range and each corresponding point of the same name within the target matching window range, and the first disparity is used to determine a first disparity map corresponding to the first image.

2. The method according to claim 1, characterized in that The bevel angle is used to bevel the first window, the scaling factor is used to scale the first window, and the offset is used to indicate the offset between the position of the pixel point with the same name in the second image and the position in the first image.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Performing iterative propagation on each pixel point in the first image and the second image includes: Determining a propagation order of the iterative propagation; Determine, according to the propagation order, a first pixel point and a second pixel point in the first image, wherein the second pixel point is one before the first pixel point in the propagation order; Determine, according to a first parameter set corresponding to the first pixel point, a first point of the same name in the second image corresponding to the first pixel point, and determine a first matching cost between the first pixel point and the first point of the same name; Determine, according to the first parameter set corresponding to the second pixel point, a second point of the same name in the second image corresponding to the first pixel point, and determine a second matching cost between the first pixel point and the second point of the same name; When the second matching cost is less than the first matching cost, the first parameter set corresponding to the first pixel point is replaced by the first parameter set corresponding to the second pixel point.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Determine M parallax fine-tuning parameters; Adding the M disparity fine-tuning parameters to the first disparity corresponding to a third pixel point respectively to determine M calculation results, where the third pixel point is any pixel point in the first window; The M calculation results are used to replace the offsets in the first parameter set corresponding to the third pixel point respectively, to determine M second parameter sets; Determine, according to the M second parameter sets, M third points of the same name corresponding to the third pixel in the second image, and determine M third matching costs between the third pixel and the M third points of the same name respectively; A second disparity is determined according to an offset in a scaling factor set corresponding to a smallest third matching cost among the M third matching costs, and the first disparity is replaced by the second disparity.

5. The method according to claim 1 or 2, characterized in that: After determining the first disparity map, the method further includes: Determine a first adjacent pixel point adjacent to a fourth pixel point, where the fourth pixel point is any pixel point in the first disparity map; Obtaining a disparity value corresponding to the fourth pixel and a disparity value corresponding to the first adjacent pixel; When the difference between the disparity value corresponding to the fourth pixel and the disparity value corresponding to the first adjacent pixel is greater than a preset difference range, the first adjacent pixel is removed from the first disparity map to determine a second disparity map.

6. A stereo matching device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 5.

7. A binocular depth camera, characterized in that: include: A first receiving module, used for acquiring a first image; A second receiving module, used for acquiring a second image; a processor, configured to determine a first window in the first image; Acquire N groups of first parameter sets, the first parameter sets including a bevel angle, a scaling factor, and an offset, wherein the bevel angle is used to bevel the first window, the scaling factor is used to scale the first window, the offset is used to indicate the offset between the position of the same-name pixel point corresponding to the second image and the position in the first image, and N is an integer greater than 1; determine N second windows in the second image according to the first window and the N groups of first parameter sets; determine N matching costs according to the first window and the N second windows, and use the second window corresponding to the smallest matching cost among the N matching costs as the target matching window; According to the offset in the first parameter set corresponding to the target matching window, a first disparity is determined, where the first disparity is the disparity between each pixel point within the first window range and each corresponding point of the same name within the target matching window range, and the first disparity is used to determine a first disparity map corresponding to the first image and the second image.

8. An electronic device, characterized in that: Includes the device as claimed in claim 6, or includes the binocular depth camera as claimed in claim 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that When the computer program code or instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 5.

Images