Stereo matching method and device, terminal and readable storage medium

Abstract

The present application provides a stereo matching method and device, a terminal and a storage medium. The stereo matching method includes: performing straight line detection on a first image, obtaining a plurality of straight lines in the first image, and dividing the plurality of straight lines into a plurality of first parallel straight line groups according to different straight line directions, wherein the first parallel straight line group includes at least two parallel straight lines. A first parallelogram is obtained based on two first parallel straight line groups with different directions. Performing straight line detection on a second image, obtaining a plurality of straight lines in the second image, and dividing the plurality of straight lines into a plurality of second parallel straight line groups according to different straight line directions, wherein the second parallel straight line group includes at least two parallel straight lines; and obtaining a second parallelogram based on two second parallel straight line groups with different directions. Matching the second parallelogram with the first parallelogram to obtain a target parallelogram, wherein the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a parallelogram-based stereo matching method, a stereo matching device, a terminal, and a non-volatile computer-readable storage medium. Background Art

[0002] The basic principle of stereo matching is to find the matching point that is most similar to a given point in one image, by calculating the similarity based on the matching cost within a certain range in another image. Stereo matching schemes can generally be divided into local matching methods and global matching methods. Local matching methods have difficulty obtaining accurate results in areas with weak texture or repeated texture, and rely on subsequent aggregation and disparity optimization steps to optimize the disparity based on the cost confidence. Global stereo matching algorithms primarily use global optimization theory to estimate disparity, establish a global energy function, and obtain the optimal disparity value by minimizing the global energy function. The results obtained by the global matching algorithm are relatively accurate, but the runtime is relatively long, making it unsuitable for real-time operation. Among them, the disadvantage of the local matching method is that it cannot effectively solve the problem of mismatching in large areas without texture or with repeated texture. This leads to a lack of reliable disparity estimates in these areas, making the subsequent optimization module unable to effectively solve the mismatching problem. Summary of the Invention

[0003] Embodiments of the present application provide a parallelogram-based stereo matching method, a stereo matching device, a terminal, and a non-volatile computer-readable storage medium.

[0004] The parallelogram-based stereo matching method of the embodiment of the present application includes: performing line detection on a first image, acquiring multiple straight lines in the first image, and dividing the multiple straight lines into multiple first parallel straight line groups according to different straight line directions, each of which includes at least two parallel straight lines. A first parallelogram is acquired based on two first parallel straight line groups with different directions. Line detection is performed on a second image, acquiring multiple straight lines in the second image, and dividing the multiple straight lines into multiple second parallel straight line groups according to different straight line directions, each of which includes at least two parallel straight lines, and the second image and the first image have different shooting angles. A second parallelogram is acquired based on the two second parallel line groups with different directions. The second parallelogram is matched with the first parallelogram to acquire a target parallelogram in the second image, and the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0005] The stereo matching device according to an embodiment of the present application includes a first detection module, a first acquisition module, a second detection module, a second acquisition module, and a third acquisition module. The first detection module is configured to perform line detection on a first image, acquire multiple lines from the first image, and divide the multiple lines into multiple first parallel line groups based on different line directions, each of which includes at least two parallel lines. The first acquisition module is configured to acquire a first parallelogram based on two first parallel line groups with different directions. The second detection module is configured to perform line detection on a second image, acquire multiple lines from the second image, and divide the multiple lines into multiple second parallel line groups based on different line directions, each of which includes at least two parallel lines. The second image and the first image are shot from different perspectives. The second acquisition module is configured to acquire a second parallelogram based on the two second parallel line groups with different directions. The third acquisition module is configured to match the second parallelogram with the first parallelogram to acquire a target parallelogram in the second image, wherein the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0006] A terminal according to an embodiment of the present application includes one or more processors, a memory, and one or more programs. One or more of the programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the stereo matching method described in the embodiment of the present application. The stereo matching method includes: performing line detection on a first image to obtain multiple lines in the first image, and dividing the multiple lines into multiple first parallel line groups based on different line directions, each of which includes at least two parallel lines. A first parallelogram is obtained based on two first parallel line groups with different directions. A second parallelogram is obtained based on the two second parallel line groups with different directions. A second parallelogram is obtained based on the two second parallel line groups with different directions. The second parallelogram is matched with the first parallelogram to obtain a target parallelogram in the second image, wherein the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0007] The non-volatile computer-readable storage medium of an embodiment of the present application includes a computer program. When executed by one or more processors, the computer program causes the processors to perform the following stereo matching method: performing line detection on a first image, acquiring multiple lines in the first image, and dividing the multiple lines into multiple first parallel line groups based on different line directions, each of which includes at least two parallel lines. A first parallelogram is obtained based on two first parallel line groups with different directions. Line detection is performed on a second image, acquiring multiple lines in the second image, and dividing the multiple lines into multiple second parallel line groups based on different line directions, each of which includes at least two parallel lines, wherein the second image is shot from a different perspective than the first image. A second parallelogram is obtained based on the two second parallel line groups with different directions. Matching the second parallelogram with the first parallelogram to obtain a target parallelogram in the second image, wherein the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0008] In the parallelogram-based stereo matching method, stereo matching device, terminal, and non-volatile computer-readable storage medium of the present application, line detection is performed on a first image and a second image taken from different perspectives. A first parallel line group in the first image is obtained based on a plurality of straight lines obtained from the line detection results in the first image, and a first parallelogram in the first image is obtained based on the plurality of first parallel line groups. Similarly, for the second image, a second parallel line group in the second image is obtained based on a plurality of straight lines obtained from the line detection results in the second image, and a second parallelogram in the second image is obtained based on the plurality of second parallel line groups. The second parallelogram is then matched based on the first parallelogram to obtain a target parallelogram in the second image, where the difference between the edge features of the target parallelogram and the edge features of the first parallelogram in the first image is within a preset range. Based on parallelogram matching, the first image and the second image are matched to obtain an accurate correspondence between the parallelograms. The disparity result of the area where the parallelogram is located is obtained according to the edge features of the parallelogram's outline, thereby providing richer and more reliable prior data for disparity optimization and improving the accuracy of disparity calculation within the first parallelogram and the target parallelogram.

[0009] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0011] Figure 1 is a schematic flow chart of a stereo matching method according to certain embodiments of the present application;

[0012] Figure 2 is a schematic structural diagram of a stereo matching device according to certain embodiments of the present application;

[0013] Figure 3 is a schematic structural diagram of a terminal in some embodiments of the present application;

[0014] Figure 4 and Figure 5 is a schematic flow chart of a stereo matching method according to certain embodiments of the present application;

[0015] Figure 6 is a schematic diagram of a first group of parallel straight lines in a first image in a stereo matching method according to certain embodiments of the present application;

[0016] Figure 7 is a schematic flow chart of a stereo matching method according to certain embodiments of the present application;

[0017] Figure 8 is a schematic diagram of a first group of parallel straight lines in a first image and a second group of parallel straight lines in a second image in a stereo matching method according to certain embodiments of the present application;

[0018] Figure 9 and Figure 10 is a schematic flow chart of a stereo matching method according to certain embodiments of the present application;

[0019] Figure 11 is a schematic diagram of a first matching descriptor in a stereo matching method according to certain embodiments of the present application;

[0020] Figure 12 and Figure 13 is a schematic flow chart of a stereo matching method according to certain embodiments of the present application;

[0021] Figure 14 This is a schematic diagram of the connection between a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and should not be understood as limiting the embodiments of the present application.

[0023] See also Figure 1, an embodiment of the present application provides a parallelogram-based stereo matching method, the stereo matching method comprising:

[0024] 01: Perform line detection on the first image to obtain multiple straight lines in the first image, and divide the multiple straight lines into multiple first parallel straight line groups according to different straight line directions, where each first parallel straight line group includes at least two parallel straight lines;

[0025] 02: Obtain the first parallelogram based on two first parallel straight line groups with different directions;

[0026] 03: Perform line detection on the second image to obtain multiple straight lines in the second image, and divide the multiple straight lines into multiple second parallel straight line groups according to different straight line directions. Each second parallel straight line group includes at least two parallel straight lines. The second image is shot at a different viewing angle than the first image.

[0027] 04: Obtain the second parallelogram based on two second parallel straight line groups with different directions; and

[0028] 05: Matching the second parallelogram according to the first parallelogram to obtain a target parallelogram in the second image, where the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0029] Please combine Figure 2The embodiment of the present application also provides a stereo matching device 10, which includes a first detection module 11, a first acquisition module 13, a second detection module 15, a second acquisition module 17, and a third acquisition module 19. The stereo matching method of the embodiment of the present application can be applied to the stereo matching device 10. Among them, the first detection module 11 is used to execute the method in 01. The first acquisition module 13 is used to execute the method in 02. The second detection module 15 is used to execute the method in 03. The second acquisition module 17 is used to execute the method in 04. The third acquisition module 19 is used to execute the method in 05. That is, the first detection module 11 is used to: perform straight line detection on the first image, obtain multiple straight lines in the first image, and divide the multiple straight lines into multiple first parallel straight line groups according to different straight line directions, each first parallel straight line group includes at least two parallel straight lines. The first acquisition module 13 is used to: obtain a first parallelogram based on two first parallel straight line groups with different directions. The second detection module 15 is configured to perform line detection on the second image, obtain multiple lines from the second image, and divide the multiple lines into multiple second parallel line groups based on their different directions. Each second parallel line group includes at least two parallel lines, and the second image is captured at a different viewing angle than the first image. The second acquisition module 17 is configured to obtain a second parallelogram based on the two second parallel line groups with different directions. The third acquisition module 19 is configured to match the second parallelogram with the first parallelogram to obtain a target parallelogram in the second image, wherein the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0030] Please combine Figure 3The present application also provides a terminal 100, which includes one or more processors 30, a memory 50, and one or more programs. The one or more programs are stored in the memory 50 and executed by the one or more processors 30. The programs include instructions for executing the stereo matching method of the present application. That is, when the one or more processors 30 execute the programs, the processors 30 can implement the methods 01, 02, 03, 04, and 05. That is, one or more processors 30 are used to: perform straight line detection on the first image, obtain multiple straight lines in the first image, and divide the multiple straight lines into multiple first parallel straight line groups according to different straight line directions, each first parallel straight line group includes at least two parallel straight lines; obtain a first parallelogram according to the two first parallel straight line groups with different directions; perform straight line detection on the second image, obtain multiple straight lines in the second image, and divide the multiple straight lines into multiple second parallel straight line groups according to different straight line directions, each second parallel straight line group includes at least two parallel straight lines, and the shooting perspective of the second image is different from that of the first image; obtain a second parallelogram according to the two second parallel straight line groups with different directions; and match the second parallelogram with the first parallelogram to obtain a target parallelogram in the second image, and the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0031] Specifically, the terminal 100 may include, but is not limited to, a mobile phone, a laptop computer, a smart TV, a tablet computer, a smartwatch, a drone, a digital camera, a digital camcorder, or a computer. The stereo matching device 10 may be a collection of functional modules integrated into the terminal 100. This application only uses the mobile phone as an example for description. The situations where the terminal 100 is other types of devices are similar to those of the mobile phone and will not be described in detail.

[0032] The basic principle of stereo matching is to find the matching point that is most similar to a given point in one image, by calculating the similarity based on the matching cost within a certain range in another image. Stereo matching schemes can generally be divided into local matching methods and global matching methods. Local matching methods have difficulty obtaining accurate results in areas with weak texture or repeated texture, and rely on subsequent aggregation and disparity optimization steps to optimize the disparity based on the cost confidence. Global stereo matching algorithms primarily use global optimization theory to estimate disparity, establish a global energy function, and obtain the optimal disparity value by minimizing the global energy function. The results obtained by the global matching algorithm are relatively accurate, but the runtime is relatively long, making it unsuitable for real-time operation. Among them, the disadvantage of the local matching method is that it cannot effectively solve the problem of mismatching in large areas without texture or with repeated texture. This leads to a lack of reliable disparity estimates in these areas, making the subsequent optimization module unable to effectively solve the mismatching problem.

[0033] In the stereo matching method, stereo matching device 10, and terminal 100 of the present application, line detection is performed on a first image and a second image taken from different perspectives. A first parallel line group in the first image is obtained based on the multiple lines obtained from the line detection results in the first image, and a first parallelogram in the first image is obtained based on the multiple first parallel line groups. Similarly, for the second image, a second parallel line group in the second image is obtained based on the multiple lines obtained from the line detection results in the second image, and a second parallelogram in the second image is obtained based on the multiple second parallel line groups. The second parallelogram is then matched with the first parallelogram to obtain a target parallelogram in the second image, where the difference between the edge features of the target parallelogram and the edge features of the first parallelogram in the first image is within a preset range. By matching the first and second images based on parallelogram matching, an accurate correspondence between the parallelograms is obtained. The disparity result of the area containing the parallelogram is obtained based on the edge features of the parallelogram's outline, thereby providing richer and more reliable prior data for disparity optimization and improving the accuracy of disparity calculation within the first and target parallelograms.

[0034] In the present application, the first image and the second image may be obtained based on binocular cameras, and the binocular cameras are located on the same horizontal line. When the terminal 100 obtains the disparity between the first image and the second image, it may be obtained based on the horizontal coordinates of each pixel in the first image and the horizontal coordinates of each pixel in the second image.

[0035] The implementation process of method 01 is the same as that of method 03, and the implementation process of method 02 is the same as that of method 04. This application only describes in detail the specific process of performing straight line detection on the first image and obtaining the first parallelogram, and does not discuss in detail the implementation process of performing straight line detection on the second image and obtaining the second parallelogram.

[0036] In methods 01 and 02, the first detection module 11 or the processor 30 performs line detection on the first image. The line detection result includes multiple lines in the first image. The lines detected in different directions are parallel or intersecting. Therefore, the multiple lines detected by the first detection module 11 or the processor 30 can be divided into multiple first parallel line groups according to the extension direction of the lines, and each first parallel line group includes at least two parallel lines. For example, Figure 6As shown, a first parallel line group includes two parallel lines. To ensure that all lines in the first image can form a first parallel line group, the parallel lines forming the first parallel line group may not be completely parallel. For example, when the angle between the two lines is within the range of [1.0°, 9.0°], such as when the angle between the two lines is any one of 1.0°, 1.8°, 2.6°, 3.5°, 4.4°, 5.7°, 6.5°, 7.3°, 8.4°, or 9.0°, the two lines can be considered to form a first parallel line group. This avoids the situation where nearly parallel lines in the first image cannot form a first parallelogram, and ensures that the regions containing the nearly parallel lines can be matched to corresponding regions in the second image, ensuring that reliable disparity estimates can be obtained in these regions.

[0037] Two first parallel straight line groups in different directions can form at least one first parallelogram. In method 05, the third acquisition module 19 or the processor 30 matches the second image based on the first parallelogram. Compared with matching the second image based on the straight line, the third acquisition module 19 or the processor 30 can more accurately find the target parallelogram corresponding to the first parallelogram in the second image. Specifically, the third acquisition module 19 matches multiple second parallelograms based on the edge features of the first parallelogram to obtain the target parallelogram corresponding to the first parallelogram in the second image, and the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a predetermined range, ensuring that all parallelograms in the first image can be matched to the corresponding target parallelogram, and ensuring that each area in the first image has a reliable disparity estimate.

[0038] It should be noted that when there is no reliable parallelogram in the first image and the second image, it is possible to consider adding detection of other graphics such as straight lines, circles or triangles or using a combination of them, and adopt a shape matching strategy similar to that of parallelograms to calculate the contour disparity information of special areas, so as to improve the optimization effect on textureless areas or repeated texture areas.

[0039] See also Figure 4 In some embodiments, 01: performing straight line detection on the first image to obtain a plurality of straight lines in the first image may include:

[0040] 011: Acquire multiple straight lines in the first image based on the Hough line detection algorithm or the straight line segment detection algorithm.

[0041] Please combine Figure 2 The first detection module 11 is further configured to execute the method in step 011. That is, the first detection module 11 is further configured to: acquire a plurality of straight lines in the first image based on the Hough line detection algorithm or the straight line segment detection algorithm.

[0042] Please combine Figure 3 The processor 30 is further configured to execute the method in step 011. That is, the processor 30 is further configured to: acquire a plurality of straight lines in the first image based on the Hough line detection algorithm or the straight line segment detection algorithm.

[0043] In one embodiment, when first detection module 11 or processor 30 acquires multiple lines in the first image based on the Hough line detection algorithm, it converts the coordinate space of the first image into a parameter space to perform line-to-curve fitting, thereby extracting the lines in the first image. Line detection in the first image based on the Hough line detection algorithm is insensitive to incomplete lines, noise, and other coexisting non-linear structures in the first image, and can tolerate gaps in feature boundary descriptions.

[0044] In another embodiment, when the first detection module 11 or the processor 30 obtains multiple straight lines in the first image based on a straight line segment detection algorithm, the first detection module 11 or the processor 30 calculates the gradient magnitude and direction of each pixel in the first image, and considers adjacent pixels with small gradient direction changes as a connected domain. Based on the rectangularity of each connected domain, it is determined whether it needs to be disconnected according to a rule to form multiple connected domains with larger rectangularities. Finally, all generated connected domains are improved and screened to obtain multiple straight lines in the first image. The straight line segment detection algorithm can quickly extract multiple straight lines from the first image, and the straight line segment detection algorithm does not require parameter adjustment, which can effectively improve the accuracy of line detection.

[0045] Please continue reading Figure 4 , Similarly, 03: perform straight line detection on the second image to obtain multiple straight lines in the second image, including:

[0046] 031: Acquire multiple straight lines in the second image based on the Hough line detection algorithm or the straight line segment detection algorithm.

[0047] Please combine Figure 2 The second detection module 15 is further configured to execute the method in step 031. That is, the second detection module 15 is further configured to: acquire a plurality of straight lines in the second image based on the Hough line detection algorithm or the straight line segment detection algorithm.

[0048] Please combine Figure 3 The processor 30 is further configured to execute the method in step 031. That is, the processor 30 is further configured to: acquire a plurality of straight lines in the second image based on the Hough line detection algorithm or the straight line segment detection algorithm.

[0049] The implementation process of the second detection module 15 performing straight line detection on the second image is the same as the implementation process of the first detection module 11 performing straight line detection on the first image, and will not be further described.

[0050] See also Figure 5 In some embodiments, 02: obtaining a first parallelogram according to two first parallel straight line groups with different directions includes:

[0051] 021: Get two adjacent parallel lines in the first parallel line group in the first direction, and use them as the first parallel lines respectively;

[0052] 023: Obtain two adjacent parallel lines in the first parallel line group in the second direction as second parallel lines, each of which intersects with the first parallel line, and the four intersection points of the first parallelogram are all located in the first image, and the second direction is different from the first direction; and

[0053] 025: Get the first parallelogram based on the intersection of two first parallel lines and two second parallel lines.

[0054] Please combine Figure 2 , the first acquisition module 13 is further configured to execute the methods in 021, 023, and 025. That is, the first acquisition module 13 is further configured to: acquire two adjacent parallel lines in the first parallel line group in the first direction as first parallel lines; acquire two adjacent parallel lines in the first parallel line group in the second direction as second parallel lines, wherein the second parallel lines intersect with the first parallel lines, and the four intersection points of the first parallelogram are all located in the first image, and the second direction is different from the first direction; and acquire the first parallelogram according to the intersection points of the two first parallel lines and the two second parallel lines.

[0055] Please combine Figure 3 , the processor 30 is further configured to execute the methods in 021, 023, and 025. That is, the processor 30 is further configured to: obtain two adjacent parallel lines in the first parallel line group in the first direction as first parallel lines; obtain two adjacent parallel lines in the first parallel line group in the second direction as second parallel lines, wherein the second parallel lines intersect with the first parallel lines, and the four intersection points of the first parallelogram are all located in the first image, and the second direction is different from the first direction; and obtain the first parallelogram according to the intersection points of the two first parallel lines and the two second parallel lines.

[0056] Please combine Figure 6, where two adjacent parallel lines in the first group of parallel lines are considered to be adjacent if no other parallel lines in the first group of parallel lines exist between them. If the first group of parallel lines includes mutually parallel lines A1D1, K1L1, M1N1, E1H1, and I1J1, for example, if line K1L1 exists between lines A1B1 and M1N1, then lines A1B1 and M1N1 are not adjacent parallel lines. However, if lines B1C1 and F1H1 do not have any other lines in the first group of parallel lines between them, then lines B1C1 and F1H1 are considered adjacent parallel lines.

[0057] In one example, in method 021, when the first obtaining module 13 or the processor 30 obtains the first parallelogram according to two first parallel straight line groups in different directions, if the first parallel straight line group in the first direction and the first parallel straight line group in the second direction are used to obtain the first parallelogram, such as Figure 6 As shown, the first parallel straight line group in the first direction includes mutually parallel straight lines A1B1, E1F1, I1J1, K1L1, and M1N1, and the first parallel straight line group in the second direction includes mutually parallel straight lines A1I1, B1J1, K1M1, and L1N1. The first acquisition module 13 or the processor 30 acquires two adjacent parallel straight lines in the first parallel straight line group in the first direction (e.g., straight lines A1B1 and K1L1; or straight lines K1L1 and M1N1; or straight lines M1N1 and E1F1; or straight lines E1F1 and I1J1) as first parallel straight lines. Similarly, the first acquisition module 13 or the processor 30 acquires two adjacent parallel straight lines in the first parallel straight line group in the second direction (e.g., straight lines A1I1 and K1M1; or straight lines K1M1 and L1N1; or straight lines L1N1 and B1J1; or straight lines E1I1 and F1J1) as second parallel straight lines. To avoid the situation where one first parallelogram contains other first parallelograms when obtaining a first parallelogram based on two first parallel lines and two second parallel lines, and to try to eliminate the interference of complex internal situations of the first parallelograms with the matching results, for example, when the two first parallel lines are non-adjacent lines A1B1 and I1J1, and the two second parallel lines are A1I1 and B1J1, the obtained first parallelogram A1B1J1I1 contains parallelograms A1B1F1E1 and E1F1J1I1. For another example, when the two first parallel lines are non-adjacent lines A1B1 and E1F1, and the two second parallel lines are A1I1 and B1J1, the obtained first parallelogram A1B1F1E1 contains parallelograms K1L1N1M1.

[0058] If the first parallel straight line intersects with the second parallel straight line (can form a first parallelogram), and the four intersection points of the two first parallel straight lines and the two second parallel straight lines are all located in the first image, the first acquisition module 11 post-processor 30 obtains the first parallelogram (such as the first parallelogram K1L1N1M1 and the first parallelogram E1F1J1I1) based on the four intersection points of the two first parallel straight lines and the two second parallel straight lines.

[0059] In another example, if the first parallel straight line group in the first direction includes mutually parallel straight lines A1D1, E1F1, I1J1, K1L1, and M1N1, and the first parallel straight line group in the second direction includes mutually parallel straight lines C1G1 and D1H1, assuming that the first parallel straight lines acquired by the first acquisition module 11 or the processor 30 are respectively straight lines C1D1 and G1H1, and the second parallel straight lines acquired are respectively straight lines C1G1 and D1H1, and since, among the four intersection points of the two first parallel straight lines and the two second parallel straight lines, intersection point D1 is located outside the first image, and since there is no edge feature at intersection point D1 in the first image, the first acquisition module 11 or the processor 30 excludes the first parallelogram C1D1H1G1, and the third acquisition module 19 or the processor 30 does not match the first parallelogram C1D1H1G1.

[0060] The first acquisition module 11 or the processor 30 sequentially acquires multiple first parallelograms in the first image according to the methods in 021, 023, and 025. Based on the same implementation process, the second acquisition module 17 or the processor 30 sequentially acquires multiple second parallelograms in the second image, which will not be further described.

[0061] See also Figure 7 In some embodiments, 05: the second parallelogram includes a plurality of parallelograms, and matching the second parallelogram according to the first parallelogram to obtain a target parallelogram in the second image includes:

[0062] 051: Obtain four intersection points of the first parallelogram in sequence along a predetermined direction as a first intersection point set, and screen multiple second parallelograms according to the ordinate values ​​of the intersection points in the first intersection point set to obtain an intermediate parallelogram;

[0063] 053: Obtain four intersection points of the middle parallelogram in sequence according to a predetermined direction as a second intersection point set of the middle parallelogram;

[0064] 055: Obtaining first matching descriptors corresponding to each intersection point in the first intersection point set according to pixel features of each intersection point in the first image;

[0065] 057: Obtaining second matching descriptors corresponding to each intersection point in the second intersection point set according to pixel features of each intersection point in the second image; and

[0066] 059: Obtain a target parallelogram according to multiple first matching descriptors and multiple second matching descriptors.

[0067] Please combine Figure 2 , the third acquisition module 19 is further configured to execute the methods in 051, 053, 055, 057, and 059. That is, the third acquisition module 19 is further configured to: sequentially acquire the four intersection points of the first parallelogram according to a predetermined direction as a first intersection point set, screen the plurality of second parallelograms according to the ordinate values ​​of the intersection points in the first intersection point set to acquire an intermediate parallelogram; sequentially acquire the four intersection points of the intermediate parallelogram according to a predetermined direction as a second intersection point set of the intermediate parallelogram; acquire first matching descriptors corresponding to the intersection points in the first intersection point set according to the pixel features of the intersection points in the first image; acquire second matching descriptors corresponding to the intersection points in the second intersection point set according to the pixel features of the intersection points in the second image; and acquire a target parallelogram according to the plurality of first matching descriptors and the plurality of second matching descriptors.

[0068] Please combine Figure 3 , the processor 30 is further configured to execute the methods in 051, 053, 055, 057, and 059. That is, the processor 30 is further configured to: sequentially obtain the four intersection points of the first parallelogram in a predetermined direction as a first intersection point set, screen the plurality of second parallelograms according to the ordinate values ​​of the intersection points in the first intersection point set to obtain an intermediate parallelogram; sequentially obtain the four intersection points of the intermediate parallelogram in a predetermined direction as a second intersection point set of the intermediate parallelogram; obtain first matching descriptors corresponding to the intersection points in the first intersection point set according to the pixel features of the intersection points in the first image; obtain second matching descriptors corresponding to the intersection points in the second intersection point set according to the pixel features of the intersection points in the second image; and obtain a target parallelogram according to the plurality of first matching descriptors and the plurality of second matching descriptors.

[0069] When the third acquisition module 19 or the processor 30 matches one of the first parallelograms to obtain a corresponding target parallelogram from the plurality of second parallelograms, matching is performed based on pixel features at four intersection points of the first parallelogram.

[0070] The third acquisition module 19 or processor 30 selects a target parallelogram corresponding to the first parallelogram from multiple second parallelograms based on the edge features of the four intersection points of the first parallelogram, that is, selects a second parallelogram whose edge features are within a preset range with the edge features of the first parallelogram.

[0071] Specifically, the third acquisition module 19 or the processor 30 executes the method 051 to obtain the four intersection points of the first parallelogram as the first intersection point set. The predetermined direction can be clockwise or counterclockwise. In one example, the first parallelogram and the second parallelogram are as follows: Figure 8 If the predetermined direction is counterclockwise, the first intersection point set of the first parallelogram A1B1C1D1 is {A1, B1, C1, D1}, and multiple second parallelograms are screened according to the ordinates of each intersection point in the first intersection point set to obtain an intermediate parallelogram, wherein the ordinates of each intersection point of the intermediate parallelogram are the same as the ordinates of the corresponding intersection points in the first intersection point set, thereby reducing the amount of calculation in the matching process of the third acquisition module 19 or the processor 30, and effectively improving the matching efficiency. For example, Figure 8 As shown, in the process of obtaining the intermediate parallelogram by the third acquisition module 19 or the processor 30, since the ordinates of the four points G2, H2, I2, and J2 are different from the ordinates of the four points in the first intersection set {A1, B1, C1, D1}, the second parallelogram G2H2I2J2 can be excluded, and only the second parallelograms located on the ordinates of the intersection points in the first intersection set (as shown by the dotted lines in the figure) are screened, and the following is obtained: Figure 8 The intermediate parallelogram A2E2F2D2 and the intermediate parallelogram A2B2C2D2 are shown.

[0072] Similarly, in method 053, the four intersection points of the intermediate parallelogram are sequentially obtained in a predetermined direction as the second intersection point set of the intermediate parallelogram. The predetermined direction is the same as the predetermined direction in method 051. For example, if the predetermined direction in method 051 is counterclockwise, the second intersection point set obtained by the third acquisition module 19 or the processor 30 for the intermediate parallelogram A2E2F2D2 is {A2, E2, F2, D2}, and the second intersection point set obtained for the intermediate parallelogram A2B2C2D2 is {A2, B2, C2, D2}.

[0073] In method 055, the third acquisition module 19 or the processor 30 obtains the first matching descriptor corresponding to each intersection point in the first intersection point set based on the pixel features of each intersection point in the first image. In method 057, the third acquisition module 19 or the processor 30 obtains the second matching descriptor corresponding to each intersection point in the second intersection point set based on the pixel features of each intersection point in the first image. The first matching descriptor of the intersection point is a feature set of the intersection point in the first image, and the feature set includes translation, scaling, and rotation invariance, so that the intersection point has a high uniqueness, so as to increase the probability of correct intersection matching. The first matching descriptor may include but is not limited to SIFT feature descriptors, SURF feature descriptors, or ORB feature descriptors. Similarly, the second matching descriptor of the intersection point is a feature set of the intersection point in the second image, and the feature set includes translation, scaling, and rotation invariance. Both the first matching descriptor and the second matching descriptor are presented in matrix form.

[0074] In method 059, the matching rate between each intersection in the first intersection set and the corresponding intersection in the second intersection set is determined based on the first matching descriptor obtained in method 055 and the second matching descriptor obtained in method 057, and the second parallelogram corresponding to the second intersection set with a high matching rate is used as the target parallelogram. Based on the first parallelogram and the target parallelogram, a high-precision disparity estimation of the edge part of the first parallelogram (i.e., the part where the side of the first parallelogram is located) can be obtained.

[0075] See also Figure 9 In some embodiments, 051: screening multiple second parallelograms according to the ordinate value of each intersection point in the first intersection point set to obtain an intermediate parallelogram includes:

[0076] 0511: sorting the intersection points in the first intersection point set in a predetermined order according to the ordinates of the intersection points in the first intersection point set;

[0077] 0513: For each second parallelogram execute:

[0078] 05131: Obtain the four intersection points of the second parallelogram in sequence according to the predetermined direction as the third intersection point set;

[0079] 05133: sorting the intersection points in the third intersection point set in a predetermined order according to the ordinates of the intersection points in the third intersection point set, wherein the intersection points in the sorted third intersection point set correspond to the intersection points in the sorted first intersection point set respectively;

[0080] 05135: Obtain the absolute value of the difference between the ordinate of each intersection point in the sorted third intersection point set and the ordinate of each intersection point in the sorted first intersection point set;

[0081] 05137: When the absolute values ​​of multiple differences are all less than the first preset threshold, the second parallelogram is used as the intermediate parallelogram; when the absolute value of one of the differences is greater than or equal to the first preset threshold, the process returns to the step of obtaining the four intersection points of the second parallelogram in sequence in a predetermined direction as the third intersection point set for the next second parallelogram.

[0082] Please combine Figure 2 The third acquisition module 19 is also used to execute the methods in 0511, 0513, 05131, 05133, 05135 and 05137. That is, the third acquisition module 19 is also used to: sort the intersections in the first intersection set in a predetermined order according to the longitudinal coordinates of the intersections in the first intersection set; execute for each second parallelogram: sequentially obtain the four intersections of the second parallelogram in a predetermined direction as a third intersection set; sort the intersections in the third intersection set in a predetermined order according to the longitudinal coordinates of the intersections in the third intersection set, and the intersections in the sorted third intersection set correspond to the intersections in the sorted first intersection set respectively; sequentially obtain the absolute value of the difference between the longitudinal coordinates of the intersections in the sorted third intersection set and the longitudinal coordinates of the intersections in the sorted first intersection set; when the absolute values ​​of multiple differences are all less than the first preset threshold, the second parallelogram is used as the intermediate parallelogram; when the absolute value of one of the differences is greater than or equal to the first preset threshold, return to the step of sequentially obtaining the four intersections of the second parallelogram in a predetermined direction as the third intersection set and starting to execute for the next second parallelogram.

[0083] Please combine Figure 3 , the processor 30 is also used to execute the methods in 0511, 0513, 05131, 05133, 05135 and 05137. That is, the processor 30 is also used to: sort the intersection points in the first intersection point set in a predetermined order according to the longitudinal coordinates of the intersection points in the first intersection point set; execute for each second parallelogram: sequentially obtain the four intersection points of the second parallelogram in a predetermined direction as a third intersection point set; sort the intersection points in the third intersection point set in a predetermined order according to the longitudinal coordinates of the intersection points in the third intersection point set, and the intersection points in the sorted third intersection point set correspond to the intersection points in the sorted first intersection point set respectively; sequentially obtain the absolute value of the difference between the longitudinal coordinates of each intersection point in the sorted third intersection point set and the longitudinal coordinates of each intersection point in the sorted first intersection point set; when the absolute values ​​of multiple difference values ​​are all less than the first preset threshold, the second parallelogram is used as the intermediate parallelogram; when the absolute value of one of the difference values ​​is greater than or equal to the first preset threshold, return to the step of sequentially obtaining the four intersection points of the second parallelogram in a predetermined direction as the third intersection point set and starting to execute for the next second parallelogram.

[0084] When the third acquisition module 19 or the processor 30 selects one or more intermediate parallelograms corresponding to the first parallelogram from multiple second parallelograms, methods 05131, 05133, 05135 and 05137 are executed on each second parallelogram in the second image until all second parallelograms are judged to obtain at least one intermediate parallelogram.

[0085] First, when the third acquisition module 19 or the processor 30 sorts the intersections in the first intersection set according to a predetermined order, the predetermined order can be sorted in descending order or ascending order according to the size of the ordinate of each intersection. In one example, the third acquisition module 19 or the processor 30 sorts the intersections in the first intersection set in descending order according to the size of the ordinate, and first obtains the ordinate of each intersection in the first intersection set. For example, if the first intersection set of the first parallelogram A1B1C1D1 is {A1, B1, C1, D1}, and the ordinate of A1>the ordinate of B1>the ordinate of D1>the ordinate of C1, then after sorting, the first intersection set of the first parallelogram A1B1C1D1 is {A1, B1, D1, C1}.

[0086] For each second parallelogram in the second image, the third acquisition module 19 or the processor 30 executes the methods in 05131 and 01533 to sort the intersection points in the third intersection point set of the second parallelogram, and the predetermined order is the same as the predetermined order in method 0511, so that the intersection points in the sorted third intersection point set correspond to the intersection points in the sorted first intersection point set. For example, for the second parallelogram A2E2F2D2 in the second image, the third acquisition module 19 or the processor 30 sorts the intersection points in the third intersection point set (i.e., {A2, E2, F2, D2}) of the second parallelogram A2E2F2D2 in descending order according to the size of the vertical coordinate, and the sorted third intersection point set of the second parallelogram A2E2F2D2 is {A2, E2, D2, F2}. For the second parallelogram in the second image, the third acquisition module 19 or the processor 30 sorts the intersection points of the third intersection point set of the second parallelogram A2B2C2D2 (i.e., {A2, B2, C2, D2}) in descending order according to the size of the ordinate, and obtains the sorted third intersection point set of the second parallelogram A2E2F2D2 as {A2, B2, D2, C2}. For the second parallelogram G2H2I2J2 in the second image, the third acquisition module 19 or the processor 30 sorts the intersection points of the third intersection point set of the second parallelogram G2H2I2J2 (i.e., {G2, H2, I2, J2}) in descending order according to the size of the ordinate, and obtains the sorted third intersection point set of the second parallelogram A2E2F2D2 as {G2, J2, H2, I2}.

[0087] Please combine Figure 8, the third acquisition module 19 or the processor 30 obtains an intermediate parallelogram corresponding to the first parallelogram based on the absolute value of the difference between the vertical coordinates of each intersection point in the sorted third intersection set and the vertical coordinates of each intersection point in the sorted first intersection set. In method 01537, when the absolute values ​​of multiple differences are all less than the first preset threshold, the second parallelogram is used as the intermediate parallelogram. Among them, the first preset threshold is related to the accuracy of the horizontal consistency between the first image and the second image, and the horizontal consistency between the first image and the second image is related to the height difference between the left and right cameras in the binocular camera. In theory, when the vertical coordinates of the same intersection in the first image and the vertical coordinates in the second image are the same, it is considered that the accuracy of the horizontal consistency between the first image and the second image is the highest.

[0088] For the first parallelogram A1B1C1D1 (the first intersection point set after sorting is {A1, B1, D1, C1}), when judging the second parallelogram A2E2F2D2 (the third intersection point set after sorting is {A2, E2, D2, F2}), if the vertical coordinates of the first image and the second image are as follows Figure 8 As shown in the xy coordinate direction, abs(A2, A1), abs(E2, B1), abs(D2, D1) and abs(F2, C1) are all less than the first preset threshold, then the second parallelogram A2E2F2D2 is considered to be the middle parallelogram, where abs(A2, A1) represents the absolute value of the difference between the ordinate of the intersection A2 and the ordinate of the intersection A1, and the meaning of other abs is the same. Similarly, for the second parallelogram A2B2C2D2 (the third intersection set after sorting is {A2, B2, D2, C2}), abs(A2, A1), abs(B2, B1), abs(D2, D1) and abs(C2, C1) are all less than the first preset threshold, then the second parallelogram A2B2C2D2 is considered to be the middle parallelogram. For the second parallelogram G2H2I2J2 (the third intersection set after sorting is {G2, J2, H2, I2}), since abs(G2, A1), abs(J2, B1), abs(H2, D1) and abs(I2, C1) are all greater than the first preset threshold, the second parallelogram G2H2I2J2 is excluded. If a second parallelogram exists, when the absolute value of one of the differences of the second parallelogram is greater than or equal to the first preset threshold, the second parallelogram is excluded. Finally, the intermediate parallelograms corresponding to the first parallelogram A1B1C1D1 obtained by the third acquisition module 19 or the processor 30 are the second parallelogram A2E2F2D2 and the second parallelogram A2B2C2D2 respectively.

[0089] Please combine Figure 10In some embodiments, 055: obtaining first matching descriptors corresponding to respective intersection points in the first intersection point set according to pixel features of respective intersection points in the first image, includes:

[0090] 0551: Obtain pixel features of each intersection point in the first intersection point set within a predetermined window in the first image, wherein each intersection point in the first intersection point set is located at the center of the predetermined window; and

[0091] 0553: Obtain first matching descriptors corresponding to each intersection point in the first intersection point set according to pixel features within the predetermined window.

[0092] Please combine Figure 2 The third acquisition module 19 is further configured to execute the methods in 0551 and 0553. That is, the third acquisition module 19 is further configured to: obtain pixel features of each intersection point in the first intersection point set within a predetermined window in the first image, wherein each intersection point in the first intersection point set is located at the center of the predetermined window; and obtain a first matching descriptor corresponding to each intersection point in the first intersection point set based on the pixel features within the predetermined window.

[0093] Please combine Figure 3 The processor 30 is further configured to execute the methods in 0551 and 0553. That is, the processor 30 is further configured to: obtain pixel features of each intersection point in the first intersection point set within a predetermined window in the first image, where each intersection point in the first intersection point set is located at the center of the predetermined window; and obtain a first matching descriptor corresponding to each intersection point in the first intersection point set based on the pixel features within the predetermined window.

[0094] Please combine Figure 8 In one example, for the first parallelogram A1B1C1D1, when the third acquisition module 19 or the processor 30 acquires the first matching descriptor of the intersection A1 in the first intersection set, the pixel features of the intersection A1 within the predetermined window in the first image are acquired, and the intersection A1 is located at the center of the predetermined window. For example, the predetermined window size is 3×3, and the pixel features include the grayscale value, gradient value, and gradient direction of the pixel. Then, Gaussian weighted processing is performed on the grayscale value, gradient value, and gradient direction corresponding to all pixels in the predetermined window. If the first matching descriptor is as follows: Figure 11 As shown in the figure, the eight arrow directions in each square represent the eight gradient directions of the pixel points, and the length of the arrow represents the gradient value.

[0095] Similarly, the implementation process of the third acquisition module 19 or the processor 30 acquiring the second matching descriptor of each intersection point in the second intersection point set is the same as the implementation process of methods 0551 and 0553, which will not be described in detail.

[0096] Please continue reading Figure 10In some embodiments, 059: obtaining a target parallelogram according to a plurality of first matching descriptors and a plurality of second matching descriptors includes:

[0097] 0591: Obtain the Hamming distance between the first matching descriptor of each intersection point in the first intersection point set and the second matching descriptor of the corresponding intersection point in the second intersection point set; and

[0098] 0593: When the multiple Hamming distances are all smaller than the second preset threshold, the middle parallelogram corresponding to the second intersection point set is used as the target parallelogram.

[0099] Please combine Figure 2 The third acquisition module 19 is further configured to execute the methods in 0591 and 0593. That is, the third acquisition module 19 is further configured to: obtain the Hamming distance between the first matching descriptor of each intersection point in the first intersection point set and the second matching descriptor of the corresponding intersection point in the second intersection point set; and when multiple Hamming distances are all less than a second preset threshold, select the middle parallelogram corresponding to the second intersection point set as the target parallelogram.

[0100] Please combine Figure 3 The processor 30 is further configured to execute the methods in 0591 and 0593. That is, the processor 30 is further configured to: obtain the Hamming distance between the first matching descriptor of each intersection point in the first intersection point set and the second matching descriptor of the corresponding intersection point in the second intersection point set; and when multiple Hamming distances are all less than a second preset threshold, use the middle parallelogram corresponding to the second intersection point set as the target parallelogram.

[0101] The second preset threshold is related to the selection of the first and second matching descriptors. The Hamming distance refers to the number of differences in the values ​​at corresponding positions in two matching descriptors. For example, for the first parallelogram A1B1C1D1 and the middle parallelogram A2B2C2D2, if the second preset threshold is 3, the values ​​in the first matching descriptor of intersection A1 in the first intersection set of the first parallelogram A1B1C1D1 are "1, 3, 3, 2, 4, 1, 1, 6, 7" respectively, and the values ​​in the second matching descriptor of A2 in the second intersection set of the middle parallelogram A2B2C2D2 are "1, 3, 3, 2, 4, 1, 2, 5, 7" respectively. Because the seventh value "1" in the first matching descriptor differs from the seventh value "2" in the second matching descriptor, and the eighth value "6" in the first matching descriptor differs from the eighth value "5" in the second matching descriptor, the Hamming distance between the first matching descriptor of intersection A1 and the second matching descriptor of intersection A2 is 2. Based on the above implementation process, the Hamming distances corresponding to the first matching descriptors of the other intersections in the first intersection set and the second matching descriptors of the corresponding intersections in the second intersection set are sequentially obtained. If multiple Hamming distances are all less than a second preset threshold, the middle parallelogram A2B2C2D2 is selected as the target parallelogram corresponding to the first parallelogram A1B1C1D1.

[0102] See also Figure 12 In some embodiments, the stereo matching method may further include:

[0103] 06: Obtain a disparity result between a first area of ​​the first image and a second area of ​​the second image based on the first parallelogram and the target parallelogram, where the first area is an area in the first image corresponding to the first parallelogram, and the second area is an area in the second image corresponding to the target parallelogram.

[0104] Please combine Figure 2 The stereo matching device 10 may further include a fourth acquisition module 18, which is configured to execute the method in step 06. That is, the fourth acquisition module 18 is configured to obtain a disparity result between a first region of the first image and a second region of the second image based on the first parallelogram and the target parallelogram, where the first region is a region in the first image corresponding to the first parallelogram, and the second region is a region in the second image corresponding to the target parallelogram.

[0105] Please combine Figure 3The processor 30 is further configured to execute the method in 06. That is, the processor 30 is further configured to obtain a disparity result between a first area of ​​the first image and a second area of ​​the second image according to the first parallelogram and the target parallelogram, where the first area is an area in the first image corresponding to the first parallelogram, and the second area is an area in the second image corresponding to the target parallelogram.

[0106] The first region may be the region containing the area of ​​the first parallelogram in the first image, and the second region may be the region containing the area of ​​the target parallelogram in the second image; or the first region may be the region containing the four sides of the first parallelogram in the first image, and the second region may be the region containing the four sides of the target parallelogram in the second image. The fourth acquisition module 18 or the processor 30 may obtain the disparity result between the first region and the second region based on the edge features of the first parallelogram (i.e., pixel information near the four sides of the first parallelogram) and the edge features of the target parallelogram (i.e., pixel information near the four sides of the target parallelogram), thereby resolving the mismatching problem of large areas of textureless or repeated textured areas in the first image.

[0107] See also Figure 13 In some embodiments, 06: obtaining a disparity result between a first region of the first image and a second region of the second image according to the first parallelogram and the target parallelogram includes:

[0108] 061: Obtain the disparity result according to the difference between the horizontal coordinates of each intersection point of the first parallelogram and the horizontal coordinates of the corresponding intersection points in the target parallelogram.

[0109] Please combine Figure 2 The fourth acquisition module 18 is further configured to execute the method in 061. That is, the fourth acquisition module 18 is further configured to obtain a disparity result according to the difference between the horizontal coordinates of the intersection points of the first parallelogram and the horizontal coordinates of the corresponding intersection points in the target parallelogram.

[0110] Please combine Figure 3 The processor 30 is further configured to execute the method in 061. That is, the processor 30 is further configured to obtain a disparity result according to the difference between the horizontal coordinates of each intersection point of the first parallelogram and the horizontal coordinates of the corresponding intersection point in the target parallelogram.

[0111] Specifically, the fourth acquisition module 18 or the processor 30 can obtain the coordinates of the four intersection points in the first parallelogram and obtain the straight line equations of the four sides of the first parallelogram. Since the target parallelogram corresponds to the first parallelogram, that is, the longitudinal coordinates of each intersection point in the first parallelogram are the same as the longitudinal coordinates of the corresponding intersection points in the target parallelogram, the coordinate information of the corresponding intersection points in the target parallelogram can be obtained based on the coordinate information of each intersection point in the first parallelogram. According to the coordinate information of the two intersection points, the straight line equation of one side of the target parallelogram can be obtained, and the coordinate information of each point on each side of the target parallelogram can be obtained.

[0112] Please combine Figure 8 Assuming that the target parallelogram corresponding to the first parallelogram A1B1C1D1 is the second parallelogram A2B2C2D2, and the linear equation of line A1B1 in the first parallelogram is a0X+b0Y+c0=0, and the coordinates of intersection A1 are labeled (x0, y0), the linear equation of line A2B2 in the target parallelogram can be set to a1X+b1Y+c1=0, and the ordinate of intersection A2 is y0. Then, the coordinates of intersection A2 are (-(c1+b1y0) / a1, y0), and the disparity between intersections A1 and A2 is (-(c1+b1y0) / a1)-x0. Similarly, the coordinate information of multiple points on line A2B2 is obtained, and the disparity between the area where line A1B1 and line A2B2 are located is obtained based on the coordinate information of multiple points on line A1B1 and the coordinate information of corresponding points on line A2B2. Finally, the fourth acquisition module 18 or the processor 30 can obtain the disparity results between the first area and the second area based on the disparity results of the four edges, such as the sum of the disparity results of the straight line A2B2, the disparity results of the straight line B2C2, the disparity results of the straight line C2D2 and the disparity results of the straight line A2D2 is recorded as S1, and the sum of the disparity results of the straight line A1B1, the disparity results of the straight line B1C1, the disparity results of the straight line C1D1 and the disparity results of the straight line A1D1 is recorded as S2, then the disparity result between the first area and the second area is S1-S2.

[0113] See also Figure 14 The present application also provides a non-volatile computer-readable storage medium 200 including a computer program 201. When the computer program 201 is executed by one or more processors 30, the processors 30 execute the stereo matching methods in 01, 02, 03, 04, 05, 011, 021, 023, 025, 031, 051, 053, 055, 057, 059, 0511, 0513, 05131, 05133, 05135, 05137, 0551, 0553, 0591, 0593, 06, and 061.

[0114] For example, when the computer program 201 is executed by one or more processors 30, the processor 30 is caused to perform the following method:

[0115] 01: Perform line detection on the first image to obtain multiple straight lines in the first image, and divide the multiple straight lines into multiple first parallel straight line groups according to different straight line directions. Each first parallel straight line group includes at least two parallel straight lines.

[0116] 02: Obtain the first parallelogram based on two first parallel straight line groups with different directions;

[0117] 03: Perform line detection on the second image to obtain multiple straight lines in the second image, and divide the multiple straight lines into multiple second parallel straight line groups according to different straight line directions. Each second parallel straight line group includes at least two parallel straight lines. The second image and the first image are shot at different viewing angles.

[0118] 04: Obtain the second parallelogram based on two second parallel straight line groups with different directions; and

[0119] 05: Matching the second parallelogram according to the first parallelogram to obtain a target parallelogram in the second image, where the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range.

[0120] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually inconsistent.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0122] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A parallelogram-based stereo matching method, characterized in that: include: Performing line detection on a first image to obtain a plurality of straight lines in the first image, and dividing the plurality of straight lines into a plurality of first parallel straight line groups according to different straight line directions, wherein each first parallel straight line group includes at least two parallel straight lines; and no other parallel straight lines in the first parallel straight line group extend between two adjacent parallel straight lines in the first parallel straight line group; Obtaining a first parallelogram according to the two first parallel straight line groups with different directions; performing line detection on the second image to obtain a plurality of straight lines in the second image, and dividing the plurality of straight lines into a plurality of second parallel straight line groups according to different straight line directions, each of the second parallel straight line groups including at least two parallel straight lines, wherein the second image is shot at a different viewing angle than the first image; Obtaining a second parallelogram according to the two second parallel straight line groups with different directions; and Match the second parallelogram according to the first parallelogram to obtain a target parallelogram in the second image, wherein the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range; the second parallelogram includes an intermediate parallelogram, and the longitudinal coordinates of each intersection point of the intermediate parallelogram are the same as the longitudinal coordinates of the corresponding intersection points in the first intersection point set, and the first intersection point set is the four intersection points of the first parallelogram.

2. The stereo matching method according to claim 1, wherein: The performing straight line detection on the first image to obtain a plurality of straight lines in the first image includes: A plurality of straight lines in the first image are acquired based on a Hough line detection algorithm or a straight line segment detection algorithm.

3. The stereo matching method according to claim 1, wherein: The step of obtaining a first parallelogram according to the two first parallel straight line groups with different directions includes: Obtaining two adjacent parallel straight lines in the first parallel straight line group in a first direction, each as a first parallel straight line; Obtaining two adjacent parallel lines in the first group of parallel lines in a second direction as second parallel lines, wherein the second parallel lines intersect with the first parallel lines, and the four intersection points of the first parallelogram are all located in the first image, and the second direction is different from the first direction; and The first parallelogram is obtained according to the intersection of the two first parallel straight lines and the two second parallel straight lines.

4. The stereo matching method according to claim 1, wherein: The second parallelogram includes a plurality of parallelograms, and the step of matching the second parallelograms according to the first parallelogram to obtain a target parallelogram in the second image includes: Sequentially acquiring four intersection points of the first parallelogram along a predetermined direction as a first intersection point set, screening a plurality of second parallelograms according to the ordinate value of each intersection point in the first intersection point set to acquire an intermediate parallelogram, and sequentially acquiring four intersection points of the intermediate parallelogram along a predetermined direction as a second intersection point set of the intermediate parallelogram; Obtaining first matching descriptors corresponding to each intersection point in the first intersection point set according to pixel features of each intersection point in the first image; Obtaining second matching descriptors corresponding to each intersection point in the second intersection point set according to pixel features of each intersection point in the second image; and The target parallelogram is obtained according to a plurality of the first matching descriptors and a plurality of the second matching descriptors.

5. The stereo matching method according to claim 4, wherein: The step of screening the plurality of second parallelograms according to the ordinate value of each intersection point in the first intersection point set to obtain an intermediate parallelogram includes: sorting the intersection points in the first intersection point set in a predetermined order according to the ordinates of the intersection points in the first intersection point set; For each of the second parallelograms, execute: sequentially acquiring four intersection points of the second parallelogram along a predetermined direction as a third intersection point set; sorting the intersection points in the third intersection point set in a predetermined order according to the ordinates of the intersection points in the third intersection point set, wherein the sorted intersection points in the third intersection point set correspond to the sorted intersection points in the first intersection point set; sequentially obtaining the absolute value of the difference between the ordinate of each intersection point in the sorted third intersection point set and the ordinate of each intersection point in the sorted first intersection point set; When the absolute values ​​of multiple differences are all less than the first preset threshold, the second parallelogram is used as the intermediate parallelogram; when the absolute value of one of the differences is greater than or equal to the first preset threshold, the step of returning to the next second parallelogram and starting to execute the step of obtaining the four intersection points of the second parallelogram in sequence in a predetermined direction as the third intersection point set.

6. The stereo matching method according to claim 4, wherein: The obtaining, according to the pixel features of each intersection point in the first intersection point set in the first image, first matching descriptors corresponding to each intersection point in the first intersection point set respectively includes: Obtaining pixel features of each intersection point in the first intersection point set within a predetermined window in the first image, wherein each intersection point in the first intersection point set is located at a center of the predetermined window; and A first matching descriptor corresponding to each intersection point in the first intersection point set is obtained according to the pixel features in the predetermined window.

7. The stereo matching method according to claim 4, wherein: The step of obtaining the target parallelogram according to the plurality of first matching descriptors and the plurality of second matching descriptors includes: Obtaining the Hamming distance between the first matching descriptor of each intersection point in the first intersection point set and the second matching descriptor of the corresponding intersection point in the second intersection point set; and When the plurality of Hamming distances are all smaller than a second preset threshold, the intermediate parallelogram corresponding to the second intersection point set is used as the target parallelogram.

8. A stereo matching device, characterized in that: include: a first detection module configured to perform line detection on a first image, obtain a plurality of lines in the first image, and divide the plurality of lines into a plurality of first parallel line groups according to different line directions, wherein each first parallel line group includes at least two parallel lines; and no other parallel lines in the first parallel line group exist between the extension lengths of two adjacent parallel lines in the first parallel line group; A first acquisition module, configured to acquire a first parallelogram according to the two first parallel straight line groups with different directions; a second detection module, configured to perform line detection on a second image, obtain a plurality of straight lines in the second image, and divide the plurality of straight lines into a plurality of second parallel straight line groups according to different straight line directions, each second parallel straight line group including at least two parallel straight lines, wherein the second image is shot at a different viewing angle than the first image; A second acquisition module, configured to acquire a second parallelogram according to the two second parallel straight line groups with different directions; and A third acquisition module is used to match the second parallelogram according to the first parallelogram to obtain a target parallelogram in the second image, and the difference between the edge features of the target parallelogram and the edge features of the first parallelogram is within a preset range; the second parallelogram includes an intermediate parallelogram, and the longitudinal coordinates of each intersection point of the intermediate parallelogram are the same as the longitudinal coordinates of the corresponding intersection points in the first intersection point set, and the first intersection point set is the four intersection points of the first parallelogram.

9. A terminal, characterized in that: include: One or more processors, memory; and One or more programs, wherein one or more of the programs are stored in the memory and executed by one or more of the processors, and the programs include instructions for executing the stereo matching method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium storing a computer program, wherein when the computer program is executed by one or more processors, the stereo matching method according to any one of claims 1 to 7 is implemented.

Images