A vision-based method for reconstructing linear structures in traffic scenes

Abstract

The present invention discloses a method for reconstructing a linear structure of a traffic scene based on vision, and the method includes the following steps: S1: acquiring an image, S2: acquiring a point cloud and depth distribution, S3: acquiring two-dimensional line segments, S4: data matching, S5: three-dimensional line segment fitting, and S6: three-dimensional scene reconstruction. Compared with the prior art, when the method for reconstructing a linear structure of a traffic scene based on vision provided in this application is applied in practice, its calculation process is more lightweight, can better restore the linear features in the traffic scene, and has better scene expression ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and particularly relates to a method for structural reconstruction of traffic scenes. Background Art

[0002] Three-dimensional reconstruction has always been an important research direction in the field of computer vision, and it has a wide range of applications in virtual reality, object recognition, visualization, etc. How to quickly and effectively reconstruct a scene in a computer has always been a hot topic and a difficult point in the research field of computer vision.

[0003] Specifically in the field of intelligent transportation, three-dimensional reconstruction of traffic scenes is beneficial for better analyzing traffic conditions on the one hand, and on the other hand, the reconstructed scene model can be used as a reference to provide an analysis basis for subsequent urban traffic construction.

[0004] To achieve three-dimensional reconstruction of traffic scenes, the following methods are often adopted in the prior art: First, a front-end image sensor is used to obtain multi-view images of objects in the scene, detect feature information including point features and line features from the images, and after matching the detected feature information, perform pose estimation and depth estimation based on the matched feature information, etc., and finally process to obtain its three-dimensional scene model.

[0005] In recent years, to better achieve three-dimensional scene reconstruction in the field of intelligent transportation, the SFM and MVS methods are often adopted in the prior art. SFM (Structure from Motion) is essentially to calculate information such as the pose, internal parameters, co-visibility relationship, and sparse point cloud of the camera input view based on feature information. And MVS (Multi View Stereo) performs depth estimation and filtering and optimization of point clouds through this information, etc., and outputs a point cloud to finally obtain a three-dimensional scene model. Of course, line features in the scene can also be used instead of point features to output three-dimensional line segments to represent the scene with line segments.

[0006] It should be pointed out that there will be certain defects when the above methods are applied to the actual three-dimensional reconstruction work of traffic scenes: For the method of using the output point cloud to represent the three-dimensional scene, on the one hand, the points in the point cloud are usually stored independently and there is no structural relationship, so it cannot effectively represent the scene geometric structure information; on the other hand, the point cloud often requires a large storage space, which will lead to low reconstruction efficiency and seriously limit the efficiency of subsequent processing and analysis. And for the method of using the output 3D line segments to represent the three-dimensional scene, due to the complexity of traffic scenes, the line features are not sufficient to characterize the entire scene, and the method of outputting 3D line segments as the final reconstruction result has an incomplete representation of the scene, especially unable to restore tiny details, and for the curved surface part in the scene, the line features may also not be able to represent or have poor expressiveness. Summary of the Invention

[0007] To solve the above problems, the object of the present invention is to provide a method for linear reconstruction of a scene, which is applied to a traffic scene, combines two-dimensional line segments obtained from an image with point cloud data, facilitates obtaining three-dimensional line segments in space, and thus realizes scene reconstruction. The reconstruction process is more efficient and convenient, the expression of scene information is more complete, and better reduction of details and curved surfaces can be achieved.

[0008] Another object of the present invention is to provide a method for linear structure reconstruction of a traffic scene based on vision. The calculation process of this method is more lightweight, and after adopting this method, three-dimensional line segments are directly output, reducing the dependence of the algorithm on the computing power and storage capacity of the device.

[0009] To achieve the above object, the technical solution of the present invention is as follows:

[0010] A method for linear structure reconstruction of a traffic scene based on vision, the method comprising:

[0011] S1: Obtain an image: Based on a specified traffic scene, obtain the image information of the traffic scene;

[0012] S2: Obtain point cloud and depth distribution: Based on the image information of a specified traffic scene, obtain point cloud data and depth distribution data;

[0013] S3: Obtain two-dimensional line segments: Based on the image information of a specified traffic scene, obtain the two-dimensional line segments in the image;

[0014] S4: Data matching: For each two-dimensional line segment, find the corresponding partial point cloud data and partial depth data;

[0015] S5: Three-dimensional line segment fitting: For each two-dimensional line segment, combine the corresponding partial depth data, and fit a three-dimensional line segment that matches its corresponding partial point cloud data;

[0016] S6: Three-dimensional scene reconstruction: After all three-dimensional line segment fittings are completed, according to the structural relationship between different three-dimensional line segments, process to obtain the linear structure model of the traffic scene.

[0017] The traffic scene has obvious linear features: roads are linear, building structures are linear, trees and road signs are also linear, etc. This means that when using vision analysis technology to perform scene reconstruction on a traffic scene, attention should be paid to the linear features in the scene, and a method with good expression of the linear structure should be applied to the actual situation.

[0018] The method provided in the present application obtains the two-dimensional line segments and point cloud data of the image information recorded in the same traffic scene, respectively, and after connecting the two-dimensional line segments with the point clouds, uses the correspondence between the two-dimensional line segments and the point clouds to reversely infer the projection of the two-dimensional line segments in space, fit the three-dimensional line segments, and finally complete the reconstruction of the linear structure in the traffic scene.

[0019] In this way, the acquisition process of three-dimensional line segments is linked to the point cloud. The system front end does not rely on the extraction of point features or line features, but instead performs in-depth processing on the point cloud. By analyzing the external manifestations of common point cloud models, the linear structure appearing in the line segment expression model is expressed using line segments, and finally the three-dimensional line segments are output to achieve structural reconstruction. This process greatly reduces the amount of calculation and reduces the dependence of the entire reconstruction process on the computing power of the device. The three-dimensional line segment generation process combined with the point cloud also makes full use of the information of each point in the point cloud. The scene information is more complete and the details in the scene are better restored.

[0020] Furthermore, S1 is specifically:

[0021] S11: Record continuous images of a specified traffic scene, and collect all images of the traffic scene to obtain a complete image set I: I = {I1, ...I n}, let I c represents any image in the image set I, then

[0022] S12: Filter out a key frame subset I′ from the entire image set I: I′={I r-a ,…,I r+b}, let I r represents any image in the key frame subset I′, then

[0023] Since the key frame subset I′ is selected from the image set I, it can be seen that the key frame subset I′ is a subset of the image set I, and

[0024] Furthermore, S2 is specifically:

[0025] S21: Process all images in the key frame subset I′ to obtain the point cloud P corresponding to the key frame subset I′: P = {P1,…,P i};

[0026] S22: Process all images in the key frame subset I′ to obtain the depth distribution h corresponding to the point cloud P: h = {h1,…,h i}.

[0027] When applied to specific practice, a visual SLAM system can be used to complete the above steps. After obtaining the complete image set I by shooting a specified traffic scene, the complete image set I is input into the visual SLAM system. The visual SLAM system performs semi-dense point cloud mapping. During the semi-dense point cloud mapping process, the visual SLAM system will screen out a subset of key frames I' from the numerous images in the complete image set I, and further establish a semi-dense point cloud model in the form of local mapping based on the subset of key frames I', obtaining the point cloud P corresponding to the subset of key frames I' and the depth distribution h corresponding to the point cloud P.

[0028] It should be emphasized that there are also various methods in the prior art that can process images to obtain point cloud data and depth data. Regarding how to process the complete image set I obtained by shooting a specified traffic scene to obtain the subset of key frames I', the point cloud P, and the specific manner of the depth distribution h, those skilled in the art can use specific algorithms to complete according to the specific situation. The technical solutions provided in this application do not claim protection for this.

[0029] Further, S3 is specifically as follows:

[0030] S31: For I c , use an edge detector to perform edge detection on it, detect the two-dimensional line segments in I c , and collect all the two-dimensional line segments in the set I c to obtain the two-dimensional line segment group in I c Let represent the two-dimensional line segment group in I , then there is c in it Any line segment, then there is

[0031] S32: Repeat S31 until all images in the complete image set I have completed edge detection, and collect the two-dimensional line segment groups in all images to obtain the total two-dimensional line segment set L corresponding to the complete image set I 2D : Then there is

[0032] Further, S31 is specifically as follows: Use the EDlines algorithm to detect line segments in the image:

[0033] S311: Obtain the grayscale image: Perform grayscale processing on I c to obtain a grayscale image;

[0034] S312: Filtering: Use a Gaussian filter to suppress noise in the grayscale image. After the Gaussian filter filters the grayscale image, a smoothed grayscale image will be obtained;

[0035] S313: Calculate the gradient map: calculate the gradient magnitude and gradient direction of adjacent pixel points in the smoothed grayscale image;

[0036] S314: Find edge pixel chain: Detect a single pixel point with a peak gradient amplitude as an anchor, connect adjacent anchors and draw the edges of the anchors to obtain an edge pixel chain e: e = {e1, ..., e N};

[0037] S315: Fitting a straight line: Fitting a straight line to each pixel point in the edge pixel chain e by the least squares method, fitting one or more straight line segments; the edge pixel chain e includes a plurality of continuous pixel points, which has a width of one pixel and can intuitively reflect the boundary properties of the object. When the least squares method is used to fit the pixel line, the pixel points in the edge pixel chain are tracked in turn. When the offset of the pixel point to be fitted in front does not exceed a certain set threshold, the current pixel line is used to fit the pixel point. When the offset of the pixel point to be fitted in front exceeds a set threshold, a new line segment is generated. Then, the remaining pixel points in the edge pixel chain are recursively processed until all pixel points are processed.

[0038] S316: Line segment verification: Verify the straight line segment obtained in S315 according to the Helmholtz principle. Set I c All the straight line segments that pass the verification are I c The two-dimensional line segment group in The method of using the EDlines algorithm to detect line segments in images is not only highly efficient, fast in detection speed, and has a low false alarm rate. Compared with other edge detectors that can only output binary edge maps, the method of using the EDlines algorithm to detect line segments in images can also directly output edge pixel chains and two-dimensional line segment groups that depict the edge pixel chains. When used in edge detection of images obtained in traffic scenes, it will show superior line segment detection performance.

[0039] Furthermore, S4 is specifically:

[0040] S41: From the key frame subset I′: I′={I r-a ,…,I r+b}, according to which each key frame image I is selected r The timestamp of the total set of two-dimensional line segments L corresponding to the total set of images I 2D , filter out r One-to-one corresponding two-dimensional line segment group, set all r The corresponding two-dimensional line segment group, the total set of key frame two-dimensional line segments L′ 2D : make Represents the total set of keyframe two-dimensional line segments L′ 2D For any two-dimensional line segment group in make A 2D line segment group For any two-dimensional line segment in

[0041] The key frame subset I′ is actually a subset of the image set I. Each element in the key frame subset I′ represents a key frame image. The timestamp of each key frame image is recorded with the subscript. For each timestamp, in the total set of two-dimensional line segments L 2D There will also be a two-dimensional line segment group corresponding to the timestamp, which represents all the two-dimensional line segments detected from the key frame image screened out at the timestamp; and all the two-dimensional line segments corresponding to I r The total set of key frame two-dimensional line segments L′ obtained by the corresponding two-dimensional line segment group 2D , we can get the two-dimensional line segments corresponding to all images in the key frame subset I′, the key frame subset I′ and the total set of key frame two-dimensional line segments L′ 2D The mutual matching and one-to-one correspondence between them will provide key support for the subsequent three-dimensional line segment fitting.

[0042] S42: According to the matching result of S41, for the two-dimensional line segment Find the 2D line segment Matching partial point cloud data and partial depth data; collection and two-dimensional line segment All the matched point cloud data are obtained Matched partial point cloud data set P r : Sets and 2D Line Segments Matched partial depth data, get All matching depth data sets h r :

[0043] Furthermore, S5 is specifically:

[0044] S51: is a two-dimensional line segment Construct a coordinate system so that the two-dimensional line segment The two-dimensional line segment Matched partial point cloud data set P r : And to the two-dimensional line segment All matching depth data sets h r : All fall into the coordinate system; for the convenience of calculation, a two-dimensional line segment should be established when establishing the coordinate system The rectangular coordinate system is located in the xoy plane, which makes calculations simpler and more convenient.

[0045] S52: Based on two-dimensional line segments And the matching depth data set hr , for the partial point cloud data set P that matches the two-dimensional line segment in space to fit a three-dimensional line segment for the points, obtaining the corresponding three-dimensional line segment of the current two-dimensional line segment r in

[0046] S53: Loop S31 - S32 until a corresponding three-dimensional line segment is fitted for each two-dimensional line segment Aggregate all three-dimensional line segments to obtain the total set L of three-dimensional line segments 3D : Among them,

[0047] After S4, for each two-dimensional line segment detected in each frame image in the key frame subset I′ a partial point cloud data set P has been matched for it r : and the depth data set h r : Establish a coordinate system for it. Then, it is obvious that the two-dimensional line segment is located in a two-dimensional plane. The depth data corresponding one-to-one with the points on the line segment will be used as intermediate known quantities to connect the two-dimensional line segment and the point cloud data. According to the direction of the two-dimensional line segment fit a line segment for the point cloud data located in three-dimensional space that matches it. The obtained three-dimensional line segment is the required three-dimensional line segment

[0048] Further, S52 is specifically as follows:

[0049] S521: Take any two points on the two-dimensional line segment as point k and point k + 1, and connect the line segment k(k + 1); find the depth data that respectively matches point k and point k + 1 and and the point cloud data that respectively matches point k and point k + 1 and Here, k is 0, 1, 2, 3 ···;

[0050] S522: Connect and to obtain the line segment as the current fitting line;

[0051] S523: Set the fitting angle threshold θ;

[0052] S524: On the two-dimensional line segment Take a point k + 2 outside k(k + 1), and find the depth data matching the point k + 2 and the point cloud data The point cloud data corresponding to the point k + 2 is the point to be fitted;

[0053] S525: Calculate the point relative to the initial fitting line offset θ k+2 : Compare θ k+2 with θ. If there is θ k+2 < θ, it is determined that the point to be fitted meets the fitting condition, and jump to S526; otherwise, jump to S528;

[0054] For the current fitting line in terms of, draw an auxiliary line ρ parallel to the line segment k(k + 1) through , and let the angle between the current fitting line and the auxiliary line ρ be θ1; then it is easy to get: Conveniently obtained according to the inverse trigonometric function:

[0055] For the point there are two cases: the point has a larger offset relative to the straight line where the current fitting line is located, or: the point has a smaller offset relative to the straight line where the current fitting line is located:

[0056] When the point has a larger offset relative to the straight line where the current fitting line is located: Connect and draw an auxiliary line ρ1 parallel to the line segment k(k + 1) through the point . At this time, the point is located below the auxiliary line ρ1. Let the angle between the straight line where the line segment k(k + 1) is located and the auxiliary line ρ1 be θ2. It is easy to see that θ1 and θ2 are corresponding angles, Let the angle between and the auxiliary line ρ1 be θ 下 , then there is Therefore, it can be obtained: The angle θ k+2 between and the straight line where the line segment k(k + 1) is located 下 can be jointly obtained by θ2 and θ

[0057] When the point has a smaller offset relative to the straight line where the current fitting line is located, connect and draw a line through the point Construct an auxiliary line ρ2 parallel to the line segment k(k + 1). At this time, the point is located above the auxiliary line ρ2. Let the angle between the line where the line segment k(k + 1) lies and the auxiliary line ρ2 be θ2'. It is easy to see that θ1 and θ2' are corresponding angles. Let the angle between 上 and the auxiliary line ρ2 be θ Then we can obtain: The angle θ k+2 between the line where the line segment k(k + 1) lies and 上 can be jointly obtained from θ2' and θ:

[0058] S526: Connect the point k + 2 with the point Construct the line segment Take the the intersection point of the line where the line segment lies and the initial fitting line where the line lies Extend the initial fitting line to the point to obtain the line segment as the current fitting line;

[0059] S527: Repeat S524 - S526 until all two - dimensional line segments in the set P of the partially - matched point cloud data r are fitted;

[0060] S528: Output the current fitting line as the three - dimensional line segment corresponding to the two - dimensional line segment

[0061] The advantage of the present invention is that: compared with the prior art, when the vision - based traffic scene linear structure reconstruction method provided in this application is applied in practice, its calculation process is more lightweight, can better restore the linear features in the traffic scene, and has better scene expression ability. Brief Description of the Drawings

[0062] Figure 1 is the flowchart of the vision - based traffic scene linear structure reconstruction method provided in the specific implementation manner.

[0063] Figure 2 is the schematic diagram of constructing a coordinate system for the two - dimensional line segment in the vision - based traffic scene linear structure reconstruction method provided in the specific implementation manner and fitting out the corresponding three - dimensional line segment

[0064] Figure 3 ​​It is a schematic diagram of fitting three-dimensional line segments to partial point cloud data distributed in space by using the vision-based linear structure reconstruction method of traffic scenes provided in the specific implementation manner.

[0065] Figure 4 It is a schematic diagram of the correspondence relationship between the subset of key frames I′, the total set of two-dimensional line segments of key frames L′ 2D , the total set of two-dimensional line segments L 2D , the point cloud data P and the depth data h.

[0066] Figure 5 It is when the point has a large offset compared to the straight line where the current fitted line is located, a specific fitting schematic diagram of the three-dimensional line segment .

[0067] Figure 6 It is when the point has a small offset compared to the straight line where the current fitted line is located, a specific fitting schematic diagram of the three-dimensional line segment . Specific implementation manner

[0068] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0069] To achieve the above objectives, the technical solution of the present invention is as follows:

[0070] Please refer to Figures 1-3 .

[0071] In this specific implementation manner, a vision-based linear structure reconstruction method of traffic scenes is provided, and the method includes:

[0072] A vision-based linear structure reconstruction method of traffic scenes, and the method includes:

[0073] S1: Obtain an image: Based on a specified traffic scene, obtain the image information of the traffic scene;

[0074] S2: Obtain point cloud and depth distribution: Based on the image information of the specified traffic scene, obtain the point cloud data and the depth distribution data;

[0075] S3: Obtain two-dimensional line segments: Based on the image information of the specified traffic scene, obtain the two-dimensional line segments in the image;

[0076] S4: Data matching: For each two-dimensional line segment, find the corresponding partial point cloud data and partial depth data;

[0077] S5: 3D line segment fitting: For each 2D line segment, combine its corresponding partial depth data, and fit a 3D line segment that matches the corresponding partial point cloud data for it.

[0078] S6: 3D scene reconstruction: After all 3D line segment fittings are completed, according to the structural relationships between different 3D line segments, process to obtain a linear structure model of the traffic scene.

[0079] Furthermore, in this specific embodiment, S1 is specifically as follows:

[0080] S11: Continuously capture images of the specified traffic scene, and collect all the images of this traffic scene to obtain the image set I: I = {I1, …, I n}, Let I c represent any image in the image set I, then there is

[0081] S12: Screen out the key frame subset I′ from the image set I: I′ = {I r-a , …, I r+b}, Let I r represent any image in the key frame subset I′, then there is

[0082] Since the key frame subset I′ is screened out from the image set I, it can be known that the key frame subset I′ is a subset of the image set I, and there is

[0083] Furthermore, S2 is specifically as follows:

[0084] S21: Process all the images in the key frame subset I′ to obtain the point cloud P corresponding to the key frame subset I′: P = {P1, …, P i};

[0085] S22: Process all the images in the key frame subset I′ to obtain the depth distribution h corresponding to the point cloud P: h = {h1, …, h i}.

[0086] Furthermore, in this specific embodiment, S3 is specifically as follows:

[0087] S31: For I c , use an edge detector to perform edge detection on it, detect the 2D line segments in I c , and collect all the 2D line segments in I c to obtain the 2D line segment group in I c Let denote the 2D line segment group of I and let c represent the 2D line segment group of I For any line segment among them, there is

[0088] S32: Repeat S31 until edge detection is completed for all images in the entire image set I, and collect the two-dimensional line segment groups in all images to obtain the total two-dimensional line segment set L corresponding to the entire image set I 2D : Then there is

[0089] Furthermore, in this specific embodiment, S31 is specifically: Use the EDlines algorithm to detect line segments in the image:

[0090] S311: Obtain the grayscale image: Perform grayscale processing on I c to obtain a grayscale image;

[0091] S312: Filtering: Use a Gaussian filter to suppress noise in the grayscale image. After filtering the grayscale image with the Gaussian filter, a smoothed grayscale image will be obtained;

[0092] S313: Obtain the gradient image: Calculate the gradient magnitude and gradient direction of adjacent pixel points in the smoothed grayscale image;

[0093] S314: Obtain the edge pixel chain: Detect a single pixel point with the peak gradient magnitude as an anchor, connect adjacent anchors and draw the edges of the anchors to obtain the edge pixel chain e: e = {e1,..., e N};

[0094] S315: Fit a straight line: Use the least squares method to perform a straight line fit on each pixel point in the edge pixel chain e to fit one or more line segments;

[0095] S316: Line segment verification: Verify the line segments obtained in S315 according to the Helmholtz principle, and collect all the line segments in I c that pass the verification, which are the two-dimensional line segment groups in I c ;

[0096] Furthermore, in this specific embodiment, S4 is specifically:

[0097] S41: From the key frame subset I': I' = {I r-a ,..., I r+b}, according to the timestamps of each key frame image I r in it, in the total two-dimensional line segment set L 2D corresponding to the entire image set I, screen out the two-dimensional line segment groups that correspond one by one to I r , and collect all the two-dimensional line segment groups corresponding to I r to obtain the total key frame two-dimensional line segment set L' 2D: Let represent the total set L′ of two-dimensional line segments in the key frame 2D For any group of two-dimensional line segments in it, there is Let be a group of two-dimensional line segments For any two-dimensional line segment in it, there is

[0098] The key frame subset I′ is essentially a subset of the entire image set I. Each element in the key frame subset I′ represents a key frame image. The time stamps of each key frame image are recorded with subscripts. Then, for each time stamp, there will also be a corresponding group of two-dimensional line segments in the total set L of two-dimensional line segments 2D ; this group of two-dimensional line segments represents all the two-dimensional line segments detected from the key frame images screened at this time stamp; and the total set L′ of key frame two-dimensional line segments obtained from all the groups of two-dimensional line segments corresponding to I r can be used to obtain the two-dimensional line segments corresponding to all the images in the key frame subset I′. The mutual matching and one-to-one correspondence relationship between the key frame subset I′ and the total set L′ of key frame two-dimensional line segments will provide key support for the subsequent three-dimensional line segment fitting 2D . 2D

[0099] S42: According to the matching result of S41, for the two-dimensional line segment find the partial point cloud data and partial depth data that match this two-dimensional line segment ; collect all the point cloud data that matches the two-dimensional line segment to obtain the set P of partial point cloud data that matches r : Collect the partial depth data that matches the two-dimensional line segment to obtain the set h of all depth data that matches r :

[0100] Furthermore, in this specific embodiment, S5 is specifically as follows

[0101] S51: Construct a coordinate system for the two-dimensional line segment such that this two-dimensional line segment this two-dimensional line segment the set P of partial point cloud data that matches r : and all the depth data that matches this two-dimensional line segment the set h r : all fall within the coordinate system; for the convenience of calculation, when establishing the coordinate system, the two-dimensional line segment ​A rectangular coordinate system in the xoy plane will be the most concise and convenient for calculation.

[0102] S52: According to the two-dimensional line segment and the corresponding depth data set h r , in space, for the part of the point cloud data set P matched with the two-dimensional line segment r , fit a three-dimensional line segment for the points to obtain the three-dimensional line segment corresponding to the current two-dimensional line segment

[0103] S53: Loop S31 - S32 until a three-dimensional line segment corresponding to each two-dimensional line segment is fitted. Aggregate all the three-dimensional line segments to obtain the total set L of three-dimensional line segments 3D : Among them,

[0104] Please refer to Figure 4 : After S4, for each two-dimensional line segment detected in each frame image in the key frame subset I′ , a partial point cloud data set P r has been matched for it: and the depth data set h r : Establish a coordinate system for it. Then it is obvious that the two-dimensional line segment is located in a two-dimensional plane. The depth data corresponding one-to-one with the points on the line segment will be used as intermediate known quantities to connect the two-dimensional line segment and the point cloud data. According to the direction of the two-dimensional line segment , fit a line segment for the point cloud data located in three-dimensional space that matches it. The obtained three-dimensional line segment is the required three-dimensional line segment

[0105] Since the key frame subset I′ is selected from the entire image set I, it can be known that the key frame subset I′ is a subset of the entire image set I, and there is And since the total set L′ of key frame two-dimensional line segments 2D is selected from the total set L of two-dimensional line segments 2D according to the timestamps of the key frame images I r , so there is Also, because the point cloud P and the depth distribution h are both obtained by processing all the images in the key frame subset I′, so there is: There is a corresponding relationship between the key frame subset I′ and the point cloud P, and there is a corresponding relationship between the point cloud P and the total set L′ of key frame two-dimensional line segments 2D and there is a corresponding relationship between the total set L′ of key frame two-dimensional line segments 2DThere is an inclusion relationship with the set L of two-dimensional line segments 2D and there is also a corresponding relationship between the set L' of two-dimensional line segments of the key frame 2D and the key frame subset I'.

[0106] Please refer to Figure 5 .

[0107] In one case, S52 is specifically as follows:

[0108] S521: Take any two points on the two-dimensional line segment as point k and point k + 1, and connect the line segment k(k + 1); find the depth data and respectively matched with point k and point k + 1, as well as the point cloud data and respectively matched with point k and point k + 1. Here, k is 0, 1, 2, 3 ···;

[0109] S522: Connect and to obtain the line segment as the current fitting line;

[0110] S523: Set the fitting angle threshold θ;

[0111] S524: Take a point k + 2 outside k(k + 1) on the two-dimensional line segment and find the depth data and the point cloud data corresponding to the point cloud data of point k + 2 as the point to be fitted;

[0112] S525: Calculate the offset θ of the point relative to the initial fitting line k+2 : Compare θ k+2 with θ. If there is θ k+2 < θ, it is determined that the point to be fitted meets the fitting condition, and jump to S526; otherwise, jump to S528;

[0113] For the current fitting line , draw an auxiliary line ρ parallel to the line segment k(k + 1) through and let the included angle between the current fitting line and the auxiliary line ρ be θ1; then it is easy to obtain: Conveniently obtained according to the inverse trigonometric function:

[0114] For the point When the point Compared with the current fitting line When the offset from the straight line where it is located is large: Connect and pass through the point to make an auxiliary line ρ1 parallel to the line segment k(k + 1). At this time, the point is located below the auxiliary line ρ1. Let the angle between the straight line where the line segment k(k + 1) is located and the auxiliary line ρ1 be θ2. It is easy to see that θ1 and θ2 are corresponding angles, Let the angle between and the auxiliary line ρ1 be θ 下 , then there is Therefore, it can be obtained that: the angle θ between and the straight line where the line segment k(k + 1) is located k+2 can be jointly obtained by θ2 and θ 下 :

[0115] Please refer to Figure 6 .

[0116] In another case, S52 is specifically:

[0117] S521: Take any two points on the two-dimensional line segment as point k and point k + 1, and connect the line segment k(k + 1); find the depth data and respectively matched with point k and point k + 1, as well as the point cloud data and respectively matched with point k and point k + 1. Here, k is 0, 1, 2, 3 ···;

[0118] S522: Connect and to obtain the line segment as the current fitting line;

[0119] S523: Set the fitting angle threshold θ;

[0120] S524: Take a point k + 2 outside k(k + 1) on the two-dimensional line segment , and find the depth data and the point cloud data corresponding to the point cloud data of point k + 2 as the point to be fitted;

[0121] S525: Calculate the offset θ of the point relative to the initial fitting line k+2 : Compare θ k+2 with θ. If there is θ k+2If < θ, it is determined that the point to be fitted meets the fitting condition, and jumps to S526; otherwise, jumps to S528;

[0122] For the current fitting line draw an auxiliary line ρ parallel to the line segment k(k + 1) through and let the angle between the current fitting line and the auxiliary line ρ be θ1; then it is easy to obtain: It can be conveniently obtained according to the inverse trigonometric function:

[0123] For the point When the offset of the point relative to the straight line where the current fitting line is located is small, connect and draw an auxiliary line ρ2 parallel to the line segment k(k + 1) through the point At this time, the point is above the auxiliary line ρ2. Let the angle between the straight line where the line segment k(k + 1) is located and the auxiliary line ρ2 be θ2′. It is easy to see that θ1 and θ2′ are corresponding angles, Let the angle between and the auxiliary line ρ2 be θ 上 , then there is Therefore, it can be obtained: The angle θ k+2 between and the straight line where the line segment k(k + 1) is located 上 can be jointly obtained by θ2′ and θ

[0124] S526: Connect the point k + 2 and the point draw the line segment take the intersection point of the straight line where the line segment is located and the straight line where the initial fitting line is located extend the initial fitting line to the point to obtain the line segment as the current fitting line;

[0125] S527: Loop S524 - S526 until all two - dimensional line segments in the matching partial point cloud data set P r are fitted for all points;

[0126] S528: Output the current fitting line as the three - dimensional line segment corresponding to the two - dimensional line segment

[0127] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for reconstructing linear structures in a traffic scene based on vision, characterized in that, The method includes: S1: Obtain an image: Based on a specified traffic scene, obtain the image information of the traffic scene; S2: Obtain point cloud and depth distribution: Based on the image information of the specified traffic scene, obtain the point cloud data and depth distribution data; S3: Obtain two-dimensional line segments: Based on the image information of the specified traffic scene, obtain the two-dimensional line segments in the image; S4: Data matching: For each two-dimensional line segment, find the corresponding partial point cloud data and partial depth data; S5: Three-dimensional line segment fitting: For each two-dimensional line segment, combined with its corresponding partial depth data, fit a matching three-dimensional line segment for its corresponding partial point cloud data; S6: Three-dimensional scene reconstruction: After all three-dimensional line segment fittings are completed, according to the structural relationship between different three-dimensional line segments, process to obtain the linear structure model of the traffic scene.

2. The vision-based linear structure reconstruction method for traffic scenes according to claim 1, wherein Specifically, S1 is as follows: S11: Record consecutive images of a specified traffic scene, and collect all images of this traffic scene to obtain the complete image set I: I = {I1, … I n}, let I c represent any image in the complete image set I, then there is S12: Screen out the key frame subset I' from the entire image set I: I' = {I r-a , …, I r+b}, let I r represent any image in the key frame subset I', then there is 3. The method for reconstructing a linear structure of a traffic scene based on vision according to claim 2, wherein, Specifically, S2 is as follows: S21: Process all the images in the key-frame subset I′ to obtain the point cloud P corresponding to the key-frame subset I′: P = {P1, …, P i}; S22: Process all the images in the key-frame subset I′ to obtain the depth distribution h corresponding to the point cloud P: h = {h1, …, h i}.

4. The method for reconstructing a linear structure of a traffic scene based on vision according to claim 3, wherein Specifically, S3 is as follows: S31: For I c , perform edge detection on it using an edge detector to detect the two-dimensional line segments in I c . The set of all two-dimensional line segments in I c is obtained to get the two-dimensional line segment group in I c . Let denote a line segment in the two-dimensional line segment group of I c . Then there is . S32: Repeat S31 until edge detection is completed for all images in the entire image set I, and collect the two-dimensional line segment groups in all images to obtain the total two-dimensional line segment set L corresponding to the entire image set I 2D : Then there is 5. The vision-based linear structure reconstruction method for traffic scenes according to claim 4, wherein Specifically, S31 is as follows: S311: Obtain a grayscale image: Perform grayscale processing on I c to obtain a grayscale image; S312: Filtering: Use a Gaussian filter to filter the grayscale image to obtain a smooth grayscale image; S313: Calculate the gradient map: Calculate the gradient amplitude and gradient direction of adjacent pixel points in the smooth grayscale image; S314: Find the edge pixel chain: Detect a single pixel point with the peak gradient magnitude as the anchor, connect adjacent anchors and draw the edges of the anchors to obtain the edge pixel chain e: e = {e1, …, e N}; S315: Fit a straight line: Use the least squares method to fit a straight line to each pixel point in the edge pixel chain e, fitting one or more straight line segments; S316: Line segment verification: Verify the straight line segments obtained in S315 according to the Helmholtz principle. All the straight line segments passing the verification in the set I c are the two-dimensional line segment group in I c ​ 6. The vision-based linear structure reconstruction method for traffic scenes according to claim 5, wherein Specifically, S4 is as follows: S41: From the key-frame subset I': I' = {I r-a , …, I r+b}, filter each key-frame image I r according to its timestamp, and in the total set L 2D of two-dimensional line segments corresponding to the entire image set I, filter out the two-dimensional line segment groups corresponding one by one to I r . Combine all the two-dimensional line segment groups corresponding to I r to obtain the total set L' 2D of two-dimensional line segments for key frames: Let represent any two-dimensional line segment group in the total set L' 2D of two-dimensional line segments for key frames, then there is Let be any two-dimensional line segment in the two-dimensional line segment group , then there is S42: Based on the matching result of S41, for the two-dimensional line segment find the partial point cloud data and partial depth data that match this two-dimensional line segment ; Aggregate all the point cloud data that matches the two-dimensional line segment to obtain the partial point cloud data set P that matches r : Aggregate the partial depth data that matches the two-dimensional line segment to obtain the set h of all the depth data that matches r :

7. The method for reconstructing a linear structure of a traffic scene based on vision according to claim 6, characterized in that, Specifically, S5 is as follows: S51: Be a two-dimensional line segment Construct a coordinate system such that the two-dimensional line segment The two-dimensional line segment The set of partial point cloud data P that matches r : And the set of all depth data h that matches the two-dimensional line segment The set of all depth data h that matches r : All fall within the coordinate system; S52: According to the two-dimensional line segment and the depth data set h that matches it r , in space, for the partial point cloud data set P that matches the two-dimensional line segment r , fit a three-dimensional line segment to the points in it, and obtain the three-dimensional line segment corresponding to the current two-dimensional line segment S53: Loop S51 - S52 until for each two - dimensional line segment fit its corresponding three - dimensional line segment Collect all three - dimensional line segments to obtain the total set L of three - dimensional line segments 3D : Among them, 8. The method for reconstructing a linear structure of a traffic scene based on vision according to claim 7, characterized in that Specifically, S52 is as follows: S521: Take a two-dimensional line segment For any two points as point k and point k + 1, connect the line segment k(k + 1); find the depth data respectively matching point k and point k + 1 and as well as the point cloud data respectively matching point k and point k + 1 and Here, k is 0, 1, 2, 3 ···; S522: Connect and to obtain line segment as the initial fitting line; S523: Set the fitting angle threshold θ; S524: Take a point k+2 outside k(k+1) on the two-dimensional line segment and find the depth data and point cloud data that match the point k+2 The point cloud data corresponding to the point k+2 is the point to be fitted;​ S525: Calculation point The offset θ relative to the initial fitting line k+2 : Compare θ k+2 with θ. If there is a θ k+2 < θ, it is determined that the point to be fitted meets the fitting condition, and jump to S526; otherwise, jump to S528; S526: Connect point k+2 with point to create line segment (k+2) Take line segment (k+2) The intersection of the line where it is located and the initial fitting line is the intersection point Extend the initial fitting line to point to obtain line segment as the current fitting line; S527: Loop S524 - S526 until all 2D line segments are fitted The set P of the partial point cloud data that are matched r All the points in; S528: Output the current fitted line as a two-dimensional line segment The corresponding three-dimensional line segment

Images