A multi-line structured light binocular vision stereo matching method and system

Abstract

A multi-line structured light binocular vision stereo matching method and system, comprising: left and right camera binocular vision sensor construction, the left camera and the multi-line laser projector constitute the left structured light sensor; the multi-line laser projector projects multiple light bars to the measured object, and the binocular stereo vision sensor simultaneously collects the left and right images of the object; based on the Hessian matrix method, the light bar center point coordinates of the left and right images are extracted, and the left and right light bar center point sets are constructed; the light bar set with corresponding relationship in the left and right images is obtained by using the polar line constraint, light bar point pre-positioning method and light bar point accurate positioning method; on the light bar with corresponding relationship, the matching point pair is obtained based on the polar line constraint; and the matching point pair is reconstructed based on the binocular stereo vision measurement model, so that the three-dimensional coordinates of the light bar center feature points are obtained. The application can more quickly and accurately identify the corresponding light bars in the binocular images, and when the distribution of the light bars on the binocular images appears deformation, fracture and disappearance, etc., the correct light bar corresponding relationship can still be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional measurement technology, and in particular to a multi-line structured light binocular vision stereo matching method and system. BACKGROUND

[0002] In recent years, with the rapid development of computer and manufacturing technology, in some industry fields, such as reverse engineering, defect detection, biological medicine and other industries, two-dimensional information has far failed to meet people's requirements. Under the background of such demand, various three-dimensional measurement technologies have developed rapidly. Modern three-dimensional measurement technology can be divided into contact measurement and non-contact measurement.

[0003] In the contact measurement method, the three-coordinate measuring machine is a typical contact measuring instrument, but due to its property of contacting the measured object with the probe, it may cause damage to the measured object during the measurement process. And its measurement speed is slow, and the measurement efficiency is low. Non-contact measurement includes methods based on optics, acoustics, electromagnetism, etc. Among them, the optical vision measurement method based on optics has the advantages of non-contact, high resolution, high speed, etc.

[0004] Among them, the active three-dimensional measurement technology based on line structured light has become one of the most widely used three-dimensional shape measurement methods because it does not depend on the texture characteristics of the measured object surface and is not limited by the lighting conditions of the measured environment. The optical projector projects a laser plane, and the laser plane intersects with the object surface to form a light bar. According to the structured light vision model, the three-dimensional coordinates of the light bar points can be reconstructed. According to the number of light planes projected by the structured light projector, the existing line structured light measurement method can be divided into single-line structured light and multi-line structured light measurement method. The single-line structured light system is simple in structure and easy to implement, but its main disadvantage is also very obvious: the measurement range is limited, the measurement efficiency is low, and there is a measurement blind area for complex objects. Therefore, the single-line structured light measurement system is only suitable for measuring objects with simple structure and no occlusion. Compared with single-line structured light, multi-line structured light can reconstruct the three-dimensional information of multiple light bars at a time, and the measurement speed is faster and the efficiency is higher, so it is especially suitable for three-dimensional reconstruction of complex object surfaces.

[0005] Although multi-line structured light brings improvement in measurement efficiency, it also introduces some problems. When the multi-line structured light is projected onto the measured object, due to the modulation of the object, the projection distribution on the two-dimensional image may appear bending, breaking and deformation, etc. This brings many problems to the matching between the center points of the light bars in the binocular images. In the multi-line structured light three-dimensional measurement, if the left and right feature points are matched incorrectly, the three-dimensional coordinates reconstructed will be wrong, thereby affecting the measurement result. For the multi-line structured light binocular vision three-dimensional measurement system, the difficulty lies in how to quickly obtain a unique matching point pair.

[0006] The prior art, such as Xie Guanghui et al. in the paper "a grating binocular stereo vision sensor light strip matching method" proposes a method of regarding the binocular camera as two grating structured light sensors, judging whether two light strips match according to the distance between the measurement results of the two grating structured light sensors. The essential idea of the algorithm is enumeration based on distance constraint. Although Wei Zhunzhong et al. in "stereo matching method of grating binocular stereo vision sensor" and "fast matching algorithm of grating binocular stereo vision sensor" optimized the search algorithm, excluded redundant calculation and greatly reduced the amount of calculation, the algorithm idea did not change in essence, and the extracted all light strip center feature points were still traversed, and whether it is a correct matching point is judged by the last distance similarity. Moreover, the method of analyzing the light strip combination mode is based on the premise that the light strip is continuous and clearly visible, and when the surface of the measured object has a region with depth mutation or large fluctuation, there will be a large number of mismatching cases.

[0007] In summary, when the measured object has a complex surface, the projection distribution of the multi-line structured light on the binocular image will appear deformation, fracture and disappearance, etc. The existing multi-line structured light binocular stereo matching method has poor robustness, a large amount of mismatching and a large amount of calculation data required, which leads to that the three-dimensional reconstruction cannot meet the engineering practice requirements. SUMMARY

[0008] In order to overcome the shortcomings of the prior art, the present application provides a multi-line structured light binocular vision stereo matching method and system. The method of the present application firstly uses a light strip point pre-positioning method to roughly find the corresponding light strip, and then uses a light strip point precise positioning method and an epipolar constraint to refine the results, and performs stereo matching on the light strip with unique corresponding relationship. A coarse-to-fine spatial geometric constraint is used to gradually refine the matching results, which improves the efficiency and accuracy of stereo matching. At the same time, when the light strip appears deformation, fracture, disappearance and the like under the binocular image, correct matching results can still be obtained.

[0009] The technical scheme of the present application is as follows:

[0010] The first aspect of the present application provides a multi-line structured light binocular vision stereo matching method, which comprises the following steps:

[0011] Step S100, left and right cameras form a binocular stereo vision sensor, and the left camera and the multi-line structured light projector form a left structured light sensor;

[0012] Step S200, calibrate the binocular stereo vision sensor and the left structured light sensor respectively, obtain the left and right camera intrinsic parameters, extrinsic parameters and light plane parameter equation, establish the relationship between the structured light in space and the light strip in the image, and obtain a pre-positioning line segment set;

[0013] Step S300, based on the Hessian matrix, light bar center extraction is performed on the images captured by the left and right cameras, and left and right light bar center point sets are obtained respectively;

[0014] Step S400, connected domain analysis is performed on the left and right images, and the connected domain corresponding to each light bar center point is obtained. Based on the predetermined set of line segments, the light bar center point extracted on the left image is searched for its corresponding point position on the right image, and the candidate light bar connected domain and the candidate light plane on the right image corresponding to the connected domain where the light bar point is located are obtained.

[0015] Step S500, based on the results obtained in step S400, based on the minimum distance constraint of the spatial point to the light plane, the light bar matching result is further refined, and finally the light bar pair with unique corresponding relationship and the light plane to which it belongs are obtained.

[0016] Step S600, based on the results obtained in step S500, based on the epipolar constraint, the matching points on the light bar pair with unique corresponding relationship are calculated, and the three-dimensional coordinates of the object surface are calculated.

[0017] Further, step S200 includes:

[0018] Step S210, calibrate the binocular stereo vision sensor and the left structured light sensor respectively, and obtain the left and right camera intrinsic parameters, extrinsic parameters and light plane parameter equation;

[0019] Step S220, establish the relationship between the structured light in space and the light bar in the image, and find the nearest and farthest spatial positions that make the end points of the multi-line structured light projection light bar in the image clear and visible within the depth of field of the left and right cameras; record the pixel coordinates of the center points of each projection light bar on the image when the nearest and farthest positions are recorded, connect the corresponding center points of the corresponding light bars, and establish a predetermined set of line segments based on the lookup table.

[0020] Further, the step S400 includes:

[0021] Step S410, connected domain analysis is performed on the left and right images, and the connected domain to which each light bar center point belongs is obtained.

[0022] Step S420, in a light strip connected domain of the left image, uniformly select a light strip center point, for each selected light strip center point, based on a predetermined set of line segments, calculate and store a left image predetermined line segment that meets the conditions, and the corresponding right image predetermined line segment and the light plane to which the predetermined line segment belongs are also obtained; calculate the epipolar line corresponding to the light strip center point, calculate the intersection of the epipolar line and the predetermined line segment, analyze the right image light strip connected domain to which the intersection belongs, if the intersection is located in a connected domain of the right image, record: the corresponding left image light strip center point, the right image intersection, the light strip connected domain to which the intersection belongs, and the light plane; traverse all light strip connected domains of the left image, and thus obtain all right image light strips that the left image light strip can correspond to and the corresponding light planes.

[0023] Further, the step S500 includes:

[0024] Step S510, based on the results obtained in step S420, in a light strip connected domain of the left image, based on the principle of triangulation, calculate the three-dimensional space point coordinates of different right image intersections corresponding thereto respectively, the three-dimensional points obtained by the intersections under the same light strip connected domain of the right image belong to the same set, the three-dimensional points obtained by the intersections under different right image light strip connected domains belong to different sets, traverse all left image light strip connected domains, and thus obtain all three-dimensional space point sets of the left image light strip and the right image light strips that can correspond thereto;

[0025] Step S520, based on the three-dimensional space point sets corresponding to all light strips of the left image obtained in step S510, calculate the average distance between all points in the set and the corresponding light plane, if the multiple right image light strip connected domains corresponding to the left image light strip belong to the same light plane, merge them into the same set, take the set with the smallest average distance as the candidate set, if the average distance is less than a certain threshold value at this time, consider the light strip connected domains corresponding to the left and right images at this time as the correct light strip pair, and the corresponding light plane is also obtained.

[0026] Further, the step S600 includes:

[0027] Step S610, based on the left and right image corresponding light strip results obtained in step S520, traverse all light strip center points of the left image, calculate the corresponding epipolar line on the right image, search for a group of light strip center points within a certain threshold distance from the epipolar line in the corresponding right image light strip connected domain, perform straight line fitting on the searched light strip center points, calculate the intersection of the fitted straight line and the epipolar line, and the intersection is the matching point;

[0028] Step S620, based on the results obtained in S610, traverse all matching point pairs, based on the principle of triangulation, bring the matching point pairs into the binocular stereo vision measurement model, calculate the corresponding three-dimensional points, and realize three-dimensional reconstruction.

[0029] The application also provides a multi-line structured light binocular vision stereo matching system, comprising:

[0030] A construction module, the left and right cameras form a binocular stereo vision sensor, and the left camera and the multi-line structured light projector form a left structured light sensor;

[0031] A calibration module, which respectively calibrates the binocular stereo vision sensor and the left structured light sensor, obtains the intrinsic parameters, extrinsic parameters and light plane parameter equation of the left and right cameras, establishes the relationship between the structured light in space and the light bar in the image, and obtains a predetermined line segment set;

[0032] A light bar center point extraction module, which extracts the light bar center points based on the Hessian matrix and the images captured by the left and right cameras, and respectively obtains the left and right light bar center point sets;

[0033] A light bar point pre-positioning module, which performs connected domain analysis on the left and right images, obtains the connected domain corresponding to each light bar center point, and based on the predetermined line segment set, finds the corresponding point position of the light bar center point extracted on the left image on the right image, obtains the candidate light bar connected domain and the candidate light plane on the right image corresponding to the connected domain where the light bar point is located;

[0034] A light bar point accurate positioning module, which further refines the light bar matching result based on the result obtained in step S400 and the minimum distance constraint from the space point to the light plane, and finally obtains the light bar pair with unique corresponding relationship and the light plane to which it belongs;

[0035] A light bar feature point stereo matching algorithm module, which, based on the result obtained in step S500, calculates the matching points based on the epipolar constraint on the light bar pair with unique corresponding relationship, and calculates the three-dimensional coordinates of the object surface.

[0036] The application also provides an electronic device, which comprises:

[0037] At least one processor; and

[0038] A memory in communication connection with the at least one processor; wherein

[0039] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0040] The application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the method described above.

[0041] Compared with the prior art, the present application has the following beneficial technical effects:

[0042] 1. The existing binocular stereo matching algorithm based on multi-line structured light has large calculation amount and a large amount of redundant calculation. The present application is based on a structured light perspective projection model and increases a light strip point pre-positioning method. Compared with the traditional method of traversing all light planes and light strip connected domains, the calculation amount of the pre-positioning method is reduced by more than half, the ambiguity of multi-line structured light binocular vision stereo matching is reduced, and the efficiency and accuracy of finding corresponding point matching are greatly improved.

[0043] 2. When the measured object has a complex surface shape and a region with depth mutation, due to the modulation of the object surface, the projection of the light strip on the image may appear deformation, breakage or even disappearance, which brings difficulties to finding the light strip center point matching point. Due to the nature of the structured light image, the stereo matching algorithm based on gray scale is invalid. The present application solves the problem of difficult distinction of matching of multi-line structured light in binocular images through a coarse-to-fine spatial geometric constraint. Accurate and unique matching can still be obtained in the case of obvious occlusion and object height fluctuation. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a general implementation flowchart of a multi-line structured light binocular vision stereo matching method of the present application;

[0045] Figure 2 is a system structure schematic diagram of an embodiment of the present application; wherein ① is a left camera, ② is a right camera, ③ is a multi-line structured light projector, and ④ is a measured object;

[0046] Figure 3 is a light strip point pre-positioning method schematic diagram based on perspective projection relationship of the present application;

[0047] Figure 4 is a corresponding light strip feature point stereo matching algorithm schematic diagram of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application is further described in detail below in combination with specific embodiments and with reference to the drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0049] The present application is further described in detail below in combination with specific embodiments.

[0050] As shown in Figure 1 , the specific implementation steps of the method of the present application are as follows:

[0051] Step S100, left and right cameras constitute binocular stereo vision sensor, left camera and multi-line structured light projector constitute left structured light sensor;

[0052] Specifically, the binocular vision three-dimensional measurement system based on multi-line structured light is as shown in the figure Figure 2 It mainly includes left camera ①, 7-line structured light projector ③, right camera ② and measured object ④. The 7-line structured light projector ③ projects 7 light strips to the measured object ④, and the left and right cameras collect images at the same time.

[0053] Step S200, calibrate the binocular stereo vision sensor and the left structured light sensor respectively, obtain the left and right camera intrinsic parameters, extrinsic parameters and light plane parameter equation, establish the relationship between the structured light in space and the light strip in the image, and obtain the predetermined positioning line segment set;

[0054] Specifically, it includes:

[0055] Step S210, calibrate the binocular stereo vision sensor and the left structured light sensor, and obtain the left and right camera intrinsic parameters, extrinsic parameters and the light plane equation of the left structured light sensor.

[0056] The left camera intrinsic parameters The right camera intrinsic parameters are respectively:

[0057]

[0058] Wherein, and are the effective focal lengths of the corresponding camera on the u-axis and the v-axis, is the optical center of the corresponding camera, is the non-perpendicular factor of the corresponding camera u-axis and y-axis; the extrinsic parameters rotation matrix R and translation vector T of the left and right cameras are:

[0059]

[0060] Wherein, is the s-th element of the rotation matrix R, , , are the three components of the translation vector T; the left structured light sensor light plane equation is:

[0061] (i = 0,1,…,m)

[0062] Wherein, is the light plane equation coefficient, and m is the number of light strips projected by the multi-line structured light projector, in this example, m = 7;

[0063] Step S220, the relationship between the structured light in space and the light bar in the image is established, and a set of predetermined line segments PLST is obtained. Specifically, the relationship between the structured light in space and the light bar in the image is established, and within the depth of field of the left and right cameras, the nearest and farthest spatial positions are found, which make the projection points of the line ends of the multi-line structured light on the image clearly visible; the pixel coordinates of the center points of each projection light bar of the structured light on the image at the nearest and farthest positions are recorded, the corresponding center points of the corresponding light bars are connected, and a set of predetermined line segments based on the lookup table is established.

[0064] As shown in Figure 3 , within the depth of field of the left and right cameras, the nearest and farthest spatial positions are found, which make the projection points of the line ends of the multi-line structured light on the image clearly visible; a certain three-dimensional space point P within the depth of field will satisfy:

[0065]

[0066] Among them, and are a certain point on the left and right image planes, , , , are the projections of and in the left and right image planes, represents a line segment composed of two end points;

[0067] In the actual imaging process, the bending of the structured light plane caused by lens distortion is inevitable, so the projection of the structured light on the image is a curve with small curvature. In order to ensure accuracy, it cannot be simply modeled as a straight line, and considering that the curve form is not complex, a cubic polynomial model is used to fit the light bar curve, and the specific implementation process is as follows:

[0068] Take the arc length of the light bar on the nearest plane of the left camera i representing the index of the light bar) as the reference, and divide the end points by 1 pixel (Stride = 1) as the interval . Therefore, on the line segment composed of two end points , (X = L or R), the end point coordinates of the adjacent line segment should satisfy:

[0069]

[0070]

[0071] and,

[0072]

[0073] where D (D = near or far) represents or corresponding coordinates or light bars under the corresponding plane. and represent the difference between the x coordinate and the y coordinate of the adjacent line segments and under the corresponding plane, , , , are the cubic polynomial coefficients corresponding to the fitted light bar curve respectively, is the arc length of the corresponding light bar on the corresponding plane, is the number of points on the light bar on the reference plane that are evenly divided;

[0074] The corresponding points of the corresponding light bars (light bars on the same light plane) extracted on the left and right planes and are connected into line segments, thereby obtaining a set of predetermined positioning line segments PLS:

[0075]

[0076] wherein,

[0077]

[0078] wherein, n is the size of the set of predetermined positioning line segments, and the elements store the corresponding left and right image predetermined positioning line segments and .

[0079] For all pixels in the left image, record the index of all predetermined positioning line segments that satisfy the formula , and establish a lookup table-based set of predetermined positioning line segments PLS, which takes pixel coordinates as an index and internally stores the corresponding left and right predetermined positioning line segments for the pixel;

[0080]

[0081] wherein,

[0082]

[0083] i , j ​​respectively represent the number of rows and columns of the image, For pixel coordinates (i, j), the corresponding predetermined line segment set, which contains line segments to .

[0084] Step S300, based on the Hessian matrix, the light bar center extraction is carried out on the images taken by the left and right cameras, and the left and right light bar center point sets are obtained respectively;

[0085] Specifically, based on the Hessian matrix, the left and right image light bar center point coordinates are obtained, and the left and right light bar center point sets are constructed and The specific process is described in detail in the article "C. Steger, "Analytical and empirical performance evaluation of subpixel line and edge detection", Empir. Eval. Methods Comput. Vis., pp. 188-210, Apr. 1998."

[0086] Step S400, based on the predetermined line segment set PLST, the left and right image light bar corresponding relationship and the light bar belonging to the light plane are obtained.

[0087] Specifically, the left and right images are subjected to connected component analysis to obtain the connected component corresponding to each light bar center point, and based on the predetermined line segment set, the light bar points extracted on the left image are subjected to the predetermined positioning method to find the corresponding point position on the right image, so as to roughly obtain the candidate light bar connected component on the right image corresponding to the connected component of the light bar point and the candidate light plane;

[0088] The step S400 includes:

[0089] Step S410, the left and right images are subjected to connected component analysis to obtain the connected component to which each light bar center point belongs;

[0090] The left and right light bar images are subjected to connected component analysis, and the pixels having 8-neighborhood adjacent relationship are regarded as the same connected component. The connected component to which each light bar center point belongs in and is marked, and the light bar center point set under the belonging different connected component and , wherein , , n and m represent the connected component index;

[0091] Step S420: In a connected region of a light stripe in the left image, uniformly select the center points of the light stripes. For each selected center point, calculate and store the conditions (formula) based on the pre-positioned line segment set. The left image prepositioning line segment is obtained; correspondingly, the right image prepositioning line segment and the light plane to which the prepositioning line segment belongs are also obtained; the epipolar line corresponding to the center point of the light stripe is calculated; the intersection point of the epipolar line and the prepositioning line segment is calculated; the connected domain of the light stripe in the right image to which the intersection point belongs is analyzed; if the intersection point is located in a certain connected domain of the right image, then record: the center point of the light stripe in the left image, the intersection point in the right image, the connected domain of the light stripe to which the intersection point belongs, and the light plane to which it belongs; traverse all connected domains of the light stripe in the left image, thereby obtaining all possible right image light stripes corresponding to the light stripe in the left image and their corresponding light planes.

[0092] Specifically, traverse the connected regions of the light stripes in the left image. Select 20 points evenly (if there are fewer than 20 points, select the original number) and iterate through them. Center point of each light stripe Based on PLST, find those that meet the conditions. Simultaneously calculate its corresponding epipolar line on the right image. ; Calculate polar lines and Pre-positioning line segment intersection Analyze the connected region of the light stripe in the right image to which the intersection point belongs. If the intersection point is located in a certain connected region of the right image... , light plane In the middle, then record , It stores all possible right image light bars corresponding to the left image light bars and their corresponding light planes, where the superscript ij represents the left image. The j-th point in the connected component has an index m that represents the corresponding right connected component of the image. .

[0093] Step S500: Based on the results obtained in step S400, and based on the minimum distance constraint from the spatial point to the light plane, further refine the light stripe matching results, and finally obtain the light stripe pairs with unique correspondence and their respective light planes.

[0094] Specifically, the light bar dot precise positioning module is based on The matching results are further refined to finally obtain light stripes with unique correspondences and their respective light planes.

[0095] Specifically, it includes:

[0096] Step S510, based on the results obtained in step S420; in a connected region of a light stripe in the left image, based on the principle of triangulation, calculate the three-dimensional spatial coordinates of the intersection points of different right images corresponding to it; the three-dimensional points obtained by the intersection points under the same connected region of the right image belong to the same set, and the three-dimensional points obtained by the intersection points under different connected regions of the right image belong to different sets; traverse all connected regions of the left image light stripe, thereby obtaining the set of all three-dimensional spatial points of the left image light stripe and its possible corresponding right image light stripe;

[0097] Specifically, traversal Based on the principle of triangulation, candidate matching point pairs are... and By incorporating the data into a binocular stereo vision measurement model, 3D reconstruction is achieved, and the corresponding 3D coordinates are obtained. The three-dimensional points obtained by finding the intersection points of the same light stripe connected region in the right image belong to the same set. Thus, the set of all possible three-dimensional points corresponding to the light stripe connected region in the left image is obtained. Where the superscript i represents the left image Connected components, the subscript mk represents the connected components of the right image. and light plane ;

[0098] Step S520: Based on the set of three-dimensional spatial points corresponding to all light stripes in the left image obtained in step S510, calculate the average distance between all points in the set and the corresponding light plane; if multiple connected regions of light stripes in the right image corresponding to the light stripes in the left image belong to the same light plane, then merge them into the same set; take the set with the smallest average distance as the candidate set; if the average distance at this time is less than a certain threshold, then the connected regions of light stripes corresponding to the left and right images at this time are considered as the correct corresponding light stripe pairs, and the corresponding light plane can also be obtained.

[0099] Specifically, traversal ,calculate All points to the corresponding light plane average distance ,in Representing three-dimensional points to the light plane The distance, where n is the size of the set. In the left connected region of the image. Among them, choose the one with the smallest average distance. As a candidate set, if at this time If the value is less than the set threshold of 1.5, then the connected region of the light stripe in the left image is considered to be... The corresponding connected region of the light stripe in the right image is And the light plane to which the light stripe belongs .

[0100] Step S600, based on the results obtained in step S500, on the light strip with a unique corresponding relationship, based on the epipolar constraint calculation matching points, according to the principle of triangulation, the three-dimensional coordinates of the object surface is calculated.

[0101] Specifically, the corresponding light strip feature point matching algorithm module, as shown in Figure 4 , based on the epipolar constraint calculation matching points, according to the principle of triangulation, the three-dimensional coordinates of the object surface is calculated.

[0102] Further, the step S600 includes:

[0103] Step S610, based on the results obtained in step S520, the corresponding light strip results of left and right image pairs; traverse all the light strip center points of the left image, calculate its corresponding epipolar line in the right image, search for a group of light strip center points within a certain threshold distance from the epipolar line in the corresponding right image light strip connected domain; linear fitting is performed on the searched light strip center points, and the intersection of the fitting straight line and the epipolar line is calculated, which is the matching point;

[0104] Specifically, traverse the left image light strip connected domain , for each light strip center point in it , calculate its corresponding epipolar line ; based on the results obtained in step S520, traverse all the light strip center points in the right image light strip connected domain corresponding to the left image light strip connected domain , search for all light strip center points within a threshold distance of 1.5 from the epipolar line and store them in set . Linear fitting is performed based on all points in to obtain the fitting straight line . The intersection of and is recorded, and the matching point pair and is recorded in set Result ;

[0105] Step S620, based on the results obtained in S610; traverse all the matching point pairs, based on the principle of triangulation, bring the matching point pairs into the binocular stereo vision measurement model, calculate the corresponding three-dimensional points, and realize three-dimensional reconstruction.

[0106] Specifically, traverse all the matching point pairs in Result , based on the principle of triangulation, bring the matching point pairs into the binocular stereo vision measurement model, calculate the corresponding three-dimensional points, and realize three-dimensional reconstruction.

[0107] The application further provides a multi-line structured light binocular vision stereo matching system, comprising:

[0108] A construction module, left and right cameras form a binocular stereo vision sensor, and the left camera and a multi-line structured light projector form a left structured light sensor;

[0109] A calibration module, the binocular stereo vision sensor and the left structured light sensor are calibrated respectively, left and right camera intrinsic parameters, extrinsic parameters and light plane parameter equations are obtained, a relationship between structured light in space and light bars in images is established, and a predetermined line segment set is obtained;

[0110] A light bar center point extraction module, based on a Hessian matrix, light bar center extraction is performed on images captured by the left and right cameras, and left and right light bar center point sets are obtained respectively;

[0111] A light bar point pre-positioning module, connected domain analysis is performed on the left and right images, connected domains corresponding to each light bar center point are obtained, based on the predetermined line segment set, a corresponding point position of the light bar center point extracted on the left image on the right image is found, a candidate light bar connected domain and a candidate light plane corresponding to the connected domain where the light bar point is located on the right image are obtained;

[0112] A light bar point accurate positioning module, based on the result obtained in step S400, based on the minimum distance constraint from a space point to a light plane, the light bar matching result is further refined, and a light bar pair with a unique corresponding relationship and a light plane to which the light bar pair belongs are finally obtained;

[0113] A light bar feature point stereo matching algorithm module, based on the result obtained in step S500, based on the epipolar constraint, a matching point is calculated on the light bar pair with a unique corresponding relationship, and a three-dimensional coordinate of an object surface is calculated.

[0114] The application further provides an electronic device, the electronic device comprising:

[0115] At least one processor; and,

[0116] A memory in communication connection with the at least one processor; wherein,

[0117] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method.

[0118] The application further provides a non-transitory computer readable storage medium, which stores computer instructions for causing the computer to execute the method.

[0119] It should be understood that the foregoing detailed description of the application, rather than limiting the application, is intended to explain and describe the current implementation of the application. Therefore, any modification, equivalent replacement or improvement made without departing from the spirit and scope of the application should be included in the protection scope of the application. In addition, the appended claims of the application are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent form of such scope and boundary.

Claims

1. A multi-line structured light binocular vision stereo matching method, characterized in that, The method includes the following steps: Step S100: The left and right cameras form a binocular stereo vision sensor, and the left camera and the multi-line structured light projector form a left structured light sensor. Step S200: Calibrate the binocular stereo vision sensor and the left structured light sensor respectively, obtain the intrinsic and extrinsic parameters of the left and right cameras and the light plane parameter equations, establish the relationship between the structured light in space and the light stripes in the image, and obtain the prepositioning line segment set; Step S300: Based on the Hessian matrix, extract the light bar center from the images captured by the left and right cameras to obtain the left and right light bar center point sets respectively; Step S400: Perform connected component analysis on the left and right images to obtain the connected component corresponding to the center point of each light stripe. Based on the prepositioned line segment set, find the corresponding point position of the center point of the light stripe extracted on the left image on the right image, and obtain the candidate light stripe connected component and candidate light plane on the right image corresponding to the connected component where the light stripe point is located. Step S500: Based on the results obtained in step S400, and based on the minimum distance constraint from the spatial point to the light plane, further refine the light stripe matching results, and finally obtain the light stripe pairs with unique correspondence and their respective light planes. In step S600, based on the results obtained in step S500, the matching point is calculated on the light stripe pairs with unique correspondences based on epipolar constraints, and the three-dimensional coordinates of the object surface are calculated.

2. The multi-line structured light binocular vision stereo matching method according to claim 1, characterized in that, The calibration step S200 for the preprocessing module includes: Step S210: Calibrate the binocular stereo vision sensor and the left structured light sensor respectively to obtain the intrinsic parameters, extrinsic parameters, and light plane parameter equations of the left and right cameras. Step S220: Establish the relationship between the structured light in space and the light stripes in the image. Within the depth of field of the left and right cameras, find the nearest and farthest spatial positions where the endpoints of the projected light stripes of the multi-line structured light on the image are clearly visible. Record the pixel coordinates of the center points of each projected light stripe on the image at the nearest and farthest positions. Connect the corresponding center points of the corresponding light stripes to establish a set of pre-positioned line segments based on a lookup table.

3. The multi-line structured light binocular vision stereo matching method according to claim 1, characterized in that, Step S400 includes: Step S410: Perform connected component analysis on the left and right images to obtain the connected component to which the center point of each light stripe belongs; Step S420: In a connected region of a light stripe in the left image, uniformly select the center points of the light stripes. For each selected center point, calculate and store the left image pre-positioning line segments that meet the conditions based on the pre-positioning line segment set. Correspondingly, the right image pre-positioning line segments and the light planes to which the pre-positioning line segments belong are also obtained. Calculate the epipolar line corresponding to the center point of the light stripe, calculate the intersection point of the epipolar line and the pre-positioning line segment, and analyze the connected region of the right image light stripe to which the intersection point belongs. If the intersection point is located in a connected region of the right image, record: the corresponding center point of the left image light stripe, the right image intersection point, the connected region of the light stripe to which the intersection point belongs, and the light plane to which it belongs. Traverse all connected regions of the light stripes in the left image, thereby obtaining all possible right image light stripes corresponding to the left image light stripes and their corresponding light planes.

4. The multi-line structured light binocular vision stereo matching method according to claim 3, characterized in that, Step S500 includes: Step S510: Based on the results obtained in step S420, in a connected region of a light stripe in the left image, the three-dimensional spatial coordinates of the corresponding intersection points of different right images are calculated according to the principle of triangulation. The three-dimensional points obtained by the intersection points under the same connected region of the right image belong to the same set, and the three-dimensional points obtained by the intersection points under different connected regions of the right image belong to different sets. By traversing all connected regions of the left image light stripe, all three-dimensional spatial point sets of the left image light stripe and its possible corresponding right image light stripe are obtained. Step S520: Based on the set of three-dimensional spatial points corresponding to all light stripes in the left image obtained in step S510, calculate the average distance between all points in the set and the corresponding light plane. If multiple connected regions of light stripes in the right image corresponding to the light stripes in the left image belong to the same light plane, then merge them into the same set and take the set with the smallest average distance as the candidate set. If the average distance at this time is less than a certain threshold, then the connected regions of light stripes corresponding to the left and right images at this time are considered as the correct corresponding light stripe pairs, and the corresponding light plane can also be obtained.

5. The multi-line structured light binocular vision stereo matching method according to claim 4, characterized in that, Step S600 includes: Step S610: Based on the corresponding light stripe results of the left and right images obtained in step S520, traverse all the light stripe center points of the left image, calculate the corresponding epipolar line on the right image, search for a group of light stripe center points within a certain threshold distance from the epipolar line in the corresponding light stripe connected domain of the right image, perform straight line fitting on the searched light stripe center points, calculate the intersection point of the fitted straight line and the epipolar line, and the intersection point is the matching point. Step S620: Based on the results obtained in S610, traverse all matching point pairs, and based on the principle of triangulation, input the matching point pairs into the binocular stereo vision measurement model to calculate the corresponding three-dimensional points and realize three-dimensional reconstruction.

6. A multi-line structured light binocular vision stereo matching system, characterized in that, include: The module consists of left and right cameras forming a binocular stereo vision sensor, and the left camera and multi-line structured light projector forming a left structured light sensor. The calibration module calibrates the binocular stereo vision sensor and the left structured light sensor respectively, obtains the intrinsic and extrinsic parameters and light plane parameter equations of the left and right cameras, establishes the relationship between structured light in space and light stripes in the image, and obtains a set of prepositioned line segments. The light bar center point extraction module, based on the Hessian matrix, extracts the light bar center from the images captured by the left and right cameras, and obtains the left and right light bar center point sets respectively; The light stripe point pre-positioning module performs connected component analysis on the left and right images to obtain the connected component corresponding to the center point of each light stripe. Based on the pre-positioning line segment set, it finds the corresponding point position of the light stripe center point extracted on the left image on the right image and obtains the candidate light stripe connected component and candidate light plane on the right image corresponding to the connected component where the light stripe point is located. The light stripe point precise positioning module, based on the results obtained in step S400 and the minimum distance constraint from the spatial point to the light plane, further refines the light stripe matching results and finally obtains light stripe pairs with unique correspondence and their respective light planes; The stereo matching algorithm module for light stripe feature points, based on the results obtained in step S500, calculates matching points on light stripe pairs with unique correspondences based on epipolar constraints, and calculates the three-dimensional coordinates of the object surface.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5.

Images