Lane line multi-sensor fusion method and system based on graph optimization, and electronic device

Abstract

The application provides a lane line multi-sensor fusion method and system based on graph optimization, and an electronic device, the method comprising: acquiring image frames and lidar frames with a time difference less than a predetermined threshold, the image frames being acquired by a camera on a vehicle, the lidar frames being acquired by a lidar on the vehicle, the image frames and the lidar frames both comprising lane line information; converting the image frames and the lidar frames to the same coordinate system based on compensation for the motion of the vehicle within the time difference; and acquiring the final lane line through a graph optimization method based on the image frames and the lidar frames in the same coordinate system. The application fuses the data frames of the synchronous camera and lidar through graph optimization, can utilize the information of multiple sensors globally, and can obtain a better lane line detection result compared with an asynchronous mode.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multi-sensor fusion method, system, and electronic device for lane lines based on graph optimization. Background Technology

[0002] As a crucial environmental element in autonomous driving technology, the accuracy of lane detection results impacts many autonomous driving functions, such as lane centering. Furthermore, matching these detection results with high-precision maps allows for lateral positioning of the vehicle. However, due to limitations in autonomous driving scenarios and sensor performance, a single sensor cannot adequately meet the performance requirements of lane detection. Therefore, seeking an accurate lane detection sensor fusion solution is essential.

[0003] Cameras and LiDAR, as the two main sensors for environmental perception in autonomous driving, use two-dimensional images and three-dimensional point clouds as raw data for lane detection, respectively. The generated images are in a two-dimensional image coordinate system. Although images can detect relatively long distances, projecting them into a three-dimensional vehicle coordinate system relies on an assumed flat ground. Therefore, road slopes and vehicle bumps will affect the final lane detection accuracy. LiDAR can directly generate detection results in the vehicle coordinate system, but due to the sparsity of the point cloud, its detection range is significantly limited.

[0004] Traditional asynchronous sensor fusion schemes continuously receive detection results from sensors and process each frame as it is received. Although they utilize the detection results from different sensors, they do not fully integrate the information from different sensors to achieve the goal of leveraging each other's strengths and compensating for each other's weaknesses.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a multi-sensor fusion method, system, and electronic device for lane lines based on graph optimization.

[0007] This invention provides a multi-sensor fusion method for lane lines based on graph optimization, the method comprising:

[0008] Image frames and lidar frames with a time difference less than a predetermined threshold are acquired. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information.

[0009] Based on compensation for the vehicle's own motion within the time difference, the image frame and the lidar frame are transformed into the same coordinate system;

[0010] Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization methods.

[0011] According to the present invention, a multi-sensor fusion method for lane lines based on graph optimization acquires image frames and lidar frames with a time difference less than a predetermined threshold, comprising:

[0012] Traverse multiple buffered image frames and multiple buffered LiDAR frames in the image buffer and LiDAR buffer to form multiple sets of paired frames;

[0013] Select the last pair of paired frames in time from the multiple pairs of paired frames, and use the image frames and lidar frames in it as the image frames and lidar frames whose time difference is less than a predetermined threshold.

[0014] According to the present invention, a multi-sensor fusion method for lane lines based on graph optimization is provided, which, based on compensation for the vehicle's own motion within the time difference, transforms the image frames and lidar frames to the same coordinate system, including:

[0015] Based on the intrinsic and extrinsic parameters of the camera, the image frame is projected from the two-dimensional coordinate system to the vehicle coordinate system;

[0016] Based on compensation for the vehicle's own motion within the time difference, the lidar frame is converted into the vehicle coordinate system of the image frame.

[0017] According to the present invention, a multi-sensor fusion method for lane lines based on graph optimization is provided. Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization, including:

[0018] Based on the image frame and the lidar frame in the same coordinate system, the lane lines of the two frames are matched.

[0019] For lane lines that do not match successfully, use them as the final first part of the lane lines;

[0020] For successfully matched lane lines, a graph optimization method is used to merge the successfully matched lane lines as the final second part of the lane lines.

[0021] According to the present invention, a multi-sensor fusion method for lane lines based on graph optimization is provided. For successfully matched lane lines, the successfully matched lane lines are fused using a graph optimization method to form the final second part of the lane lines, including:

[0022] For all points from the LiDAR frame in the successfully matched lane line, a first optimization is performed. The first optimization includes minimizing the error of a first variable and a second variable. The first variable includes the ordinate value of the point, and the second variable includes the ordinate value of the point's abscissa value on the lane line equation to be optimized.

[0023] According to the present invention, a multi-sensor fusion method for lane lines based on graph optimization is provided. For successfully matched lane lines, the successfully matched lane lines are fused using a graph optimization method to form the final second part of the lane lines, including:

[0024] For all points from the image frame in the successfully matched lane lines, a second optimization is performed. The second optimization includes minimizing the error of a third variable and a fourth variable. The third variable includes the ordinate value of the point, and the fourth variable includes the sum of a first sub-variable and a second sub-variable. The first sub-variable includes the ordinate value of the abscissa value of the point on the lane line equation to be optimized, and the second sub-variable includes the offset parameter when the image frame is projected from the two-dimensional coordinate system to the vehicle coordinate system.

[0025] According to the present invention, a multi-sensor fusion method for lane lines based on graph optimization is provided. For successfully matched lane lines, the successfully matched lane lines are fused using a graph optimization method to form the final second part of the lane lines, including:

[0026] For a successfully matched lane line, a third optimization is performed, which involves minimizing the constant term coefficient error and / or linear term coefficient error of the lane line equation to be optimized between mutually parallel lane lines.

[0027] This invention also provides a multi-sensor fusion system for lane lines based on graph optimization, the system comprising:

[0028] The acquisition module is used to acquire image frames and lidar frames with a time difference of less than a predetermined threshold. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information.

[0029] The conversion module is used to convert the image frame and the lidar frame to the same coordinate system based on compensation for the vehicle's own motion within the time difference.

[0030] The optimization module is used to obtain the final lane lines based on image frames and LiDAR frames in the same coordinate system using a graph optimization method.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the graph-optimized lane line multi-sensor fusion method as described in any of the preceding claims.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the graph-optimized lane line multi-sensor fusion method as described in any of the preceding claims.

[0033] The multi-sensor fusion method, system, and electronic device for lane lines based on graph optimization provided by this invention can globally utilize information from multiple sensors by fusing data frames from synchronous cameras and lidar sensors through graph optimization, resulting in better lane line detection results compared to asynchronous methods. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating a multi-sensor fusion method for lane lines based on graph optimization provided by this invention;

[0036] Figure 2 This is a schematic diagram of the process for synchronously fusing lane lines based on two sensor buffers provided by the present invention;

[0037] Figure 3 A schematic diagram of a multi-sensor fusion system for lane lines based on graph optimization provided by the present invention;

[0038] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] The multi-sensor fusion method for lane lines based on graph optimization provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0041] Figure 1 This invention provides a flowchart illustrating a multi-sensor fusion method for lane lines based on graph optimization, as shown below. Figure 1 As shown, the present invention provides a multi-sensor fusion method for lane lines based on graph optimization, which may include the following steps.

[0042] It should be noted that, in the vehicle coordinate system below, the longitudinal coordinate value is the coordinate value on the X-axis parallel to the ground and pointing forward of the vehicle, and the lateral coordinate value is the coordinate value on the Y-axis pointing to the driver's left.

[0043] S100: Acquire image frames and LiDAR frames with a time difference less than a predetermined threshold. The image frames are acquired through a camera on the vehicle, and the LiDAR frames are acquired through a LiDAR sensor on the vehicle. Both the image frames and LiDAR frames include lane line information. Perform time matching on the received image and LiDAR lane line detection results (image frames and LiDAR frames), and use the matching result as the input for graph optimization of the current frame.

[0044] Preferably, the lane line information in the lidar frame is obtained by processing the lidar frame using a deep learning model corresponding to the lidar.

[0045] Preferably, the lane line information in the image frame is obtained by processing the image frame using a deep learning model corresponding to the camera.

[0046] Preferably, the image frames and LiDAR frames processed by the deep learning model only include lane line information.

[0047] Preferably, the vehicle includes an autonomous vehicle.

[0048] Optionally, acquiring image frames and LiDAR frames with a time difference less than a predetermined threshold includes:

[0049] Traverse multiple buffered image frames and multiple buffered LiDAR frames in the image buffer and LiDAR buffer to form multiple sets of paired frames;

[0050] Select the last pair of paired frames from multiple pairs, and use the image frames and lidar frames in it as image frames and lidar frames with a time difference less than a predetermined threshold.

[0051] Preferably, Figure 2 This is a schematic diagram of the process for synchronously fusing lane lines based on two sensor buffers provided by the present invention, as shown below. Figure 2As shown, the image and LiDAR detection results are stored in a buffer of length 5. When the number of data frames in the buffer is 5 and new data comes in, the oldest frame is discarded and the newest frame is put in.

[0052] Because the data from the two sensors have a time delay, it is necessary to synchronize the two types of data frames in terms of timestamps, find the pair of latest paired frames as the input to the subsequent fusion system, and perform motion compensation on the detection results with large time delays. The specific steps are as follows:

[0053] The buffers of the image and the LiDAR are traversed, and data frames are paired according to the time difference being less than a certain threshold to obtain multiple sets of paired frames.

[0054] Sort these paired frames in ascending order of image frame timestamps (assuming the LiDAR has a relatively large latency);

[0055] The pair of paired frames with the largest timestamp is taken as the input to the subsequent fusion system.

[0056] S200: Based on compensation for the vehicle's own motion within the time difference, the image frame and the lidar frame are transformed into the same coordinate system.

[0057] It should be noted that motion compensation is required before inputting the data into the subsequent fusion system. Also, since the LiDAR data is usually slightly behind the image, the LiDAR data is typically converted to the coordinate system corresponding to the time of the image.

[0058] Preferably, the lane line position of the LiDAR is converted to the vehicle coordinate system at the timestamp of the matched image frame, and the image frame is converted from two-dimensional to three-dimensional vehicle coordinate system.

[0059] Preferably, the lane line position of the LiDAR is converted to the vehicle coordinate system at the timestamp of the matched image frame, and the LiDAR frame is converted from the three-dimensional vehicle coordinate system to the original two-dimensional coordinate system of the image frame.

[0060] Optionally, based on compensation for the vehicle's own motion during the time difference, the image frame and the lidar frame are transformed to the same coordinate system, including:

[0061] Based on the camera's intrinsic and extrinsic parameters, the image frames are projected from the two-dimensional coordinate system to the vehicle coordinate system.

[0062] Based on compensation for the vehicle's own motion during the time difference, the lidar frame is converted into the vehicle coordinate system of the image frame.

[0063] S300: Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization.

[0064] Optionally, based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through a graph optimization method, including:

[0065] Based on image frames and LiDAR frames in the same coordinate system, lane lines in the two frames are matched.

[0066] For lane lines that do not match successfully, use them as the final first part of the lane lines;

[0067] For successfully matched lane lines, a graph optimization method is used to merge the successfully matched lane lines as the final second part of the lane lines.

[0068] Preferably, the distance between the LiDAR lane lines and the image lane lines in the acquired paired frames is calculated, and the Hungarian algorithm is used for correlation matching to obtain a one-to-one matching relationship between the LiDAR lane lines and the image lane lines. For LiDAR lane lines and image lane lines that do not match, they are transmitted as part of the final lane line result.

[0069] It should be noted that for the matched LiDAR lane lines and image lane lines, the process proceeds to the graph optimization section for data fusion. According to graph optimization theory, it is necessary to specify the vertices and edges used to construct the graph, as well as the parameters to be optimized.

[0070] Optionally, for successfully matched lane lines, a graph optimization method is used to fuse the successfully matched lane lines as the final second part of the lane lines, including:

[0071] For all points from the LiDAR frame in the successfully matched lane line, perform the first optimization, which includes minimizing the error of the first variable and the second variable. The first variable includes the ordinate value of the point, and the second variable includes the ordinate value of the abscissa value of the point on the lane line equation to be optimized.

[0072] Preferably, the first optimization, as a constraint on an edge, is formulated for a specific 3D point i in the LiDAR lane line as follows:

[0073]

[0074] Where, x i ,y i Ω represents the longitudinal and lateral coordinates of point i in the vehicle coordinate system. i Represents the information matrix.

[0075] The specific formula for the error function is as follows:

[0076]

[0077] in, This refers to the lane line equation to be optimized, and the optimization objective is the parameters c0, c1, and c2 of this lane line equation.

[0078] Optionally, for successfully matched lane lines, a graph optimization method is used to fuse the successfully matched lane lines as the final second part of the lane lines, including:

[0079] For all points from the image frame in the successfully matched lane lines, a second optimization is performed. The second optimization includes minimizing the errors of the third and fourth variables. The third variable includes the ordinate value of the point, and the fourth variable includes the sum of the first and second sub-variables. The first sub-variable includes the ordinate value of the abscissa value of the point on the lane line equation to be optimized, and the second sub-variable includes the offset parameter when the image frame is projected from the two-dimensional coordinate system to the vehicle coordinate system.

[0080] Preferably, the second optimization, as a constraint on an edge, is formulated for a specific 3D point i in the lane lines of the image as follows:

[0081]

[0082] Where, x i ,y i The longitudinal and lateral coordinates of point i in the vehicle coordinate system represent the vehicle body coordinates.

[0083] Specifically, the difference from the aforementioned LiDAR lane lines is that the 3D points used to construct the edges are obtained by projecting 2D points from the image onto a 3D coordinate system based on the camera's intrinsic and extrinsic parameters. Therefore, an offset parameter needs to be added to its error function, and the error function formula is as follows:

[0084]

[0085] Where, x i ,y i It refers to the longitudinal and lateral coordinates of the detection point projected onto the three-dimensional vehicle coordinate system, where Δy is the offset parameter.

[0086] Furthermore, Δy includes the pitch parameter from the camera's extrinsic parameters.

[0087] Optionally, for successfully matched lane lines, a graph optimization method is used to fuse the successfully matched lane lines as the final second part of the lane lines, including:

[0088] For successfully matched lane lines, a third optimization is performed, which involves minimizing the constant term coefficient error and / or linear term coefficient error of the lane line equation to be optimized between mutually parallel lane lines.

[0089] It should be noted that, based on the parallel relationship between lane lines, the above third optimization is formed. First, all lines need to be grouped according to their positional relationship: parallel lines are grouped together, and the rest are grouped together.

[0090] Preferably, the third optimization is a constraint on an edge, wherein, for any lane line i, its corresponding lane line equation, compared with the lane line equations corresponding to all other lane lines, has the following constant term constraint formula:

[0091]

[0092] Where c0 is the constant term in the equation of lane line i, and c'0 is the constant term in the equation of the equation of the other lane line.

[0093] The error function formula is as follows:

[0094]

[0095] Similarly, the formula for the coefficient constraint of the linear term is:

[0096]

[0097] Where c1 is the coefficient of the first term of the equation corresponding to lane line i, and c'1 is the coefficient of the first term of the equation corresponding to the other lane line.

[0098] The error function formula is as follows:

[0099]

[0100] It should be noted that, since lane lines are usually approximately straight, this invention focuses on constant terms and linear constraints.

[0101] Furthermore, the constant term and the linear term constraints are assigned different weight values ​​to the information matrix based on experience.

[0102] Preferably, the first, second, and third optimizations described above can be combined to form the overall graph optimization loss function, as shown in the following formula:

[0103]

[0104] Where k represents the k-th lane line, This represents all points of all lane lines indicated by the lidar. The sum of, others similar.

[0105] Preferably, based on the optimized lane line equation, sampling is performed at fixed points to obtain the final fused lane line result.

[0106] This embodiment leverages the accuracy of LiDAR's 3D detection results, using it as the anchor point for the image optimizer. Combined with the relatively long detection range of the image, this compensates for the LiDAR's detection range, simultaneously meeting the requirements for both detection distance and accuracy. Compared to asynchronous methods, this approach enables simultaneous fusion of detection data from multiple sensors, achieving a more globally optimal fusion result.

[0107] The following describes the graph-optimized lane line multi-sensor fusion system provided by the present invention. The graph-optimized lane line multi-sensor fusion system described below can be referred to in correspondence with the graph-optimized lane line multi-sensor fusion method described above.

[0108] Figure 3 This invention provides a schematic diagram of a multi-sensor fusion system for lane lines based on graph optimization, as shown below. Figure 3 As shown, the present invention also provides a multi-sensor fusion system for lane lines based on graph optimization, the system comprising:

[0109] The acquisition module is used to acquire image frames and lidar frames with a time difference of less than a predetermined threshold. The image frames are acquired by the camera on the vehicle, and the lidar frames are acquired by the lidar on the vehicle. Both the image frames and lidar frames include lane line information.

[0110] The conversion module is used to convert image frames and lidar frames to the same coordinate system based on compensation for the vehicle's own motion within the time difference.

[0111] The optimization module is used to obtain the final lane lines based on image frames and LiDAR frames in the same coordinate system using a graph optimization method.

[0112] This embodiment optimizes and fuses data frames from both the camera and LiDAR sensors, enabling global utilization of information from multiple sensors and achieving better lane detection results compared to asynchronous methods.

[0113] Figure 4 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a graph-optimized lane line multi-sensor fusion method, the method including:

[0114] Image frames and lidar frames with a time difference less than a predetermined threshold are acquired. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information.

[0115] Based on compensation for the vehicle's own motion within the time difference, the image frame and the lidar frame are transformed into the same coordinate system;

[0116] Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization methods.

[0117] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The present invention also provides an autonomous driving vehicle, the vehicle having autonomous driving or assisted driving functions, the vehicle including the above-mentioned electronic equipment.

[0119] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the graph-optimized lane line multi-sensor fusion method provided by the above methods, the method comprising:

[0120] Image frames and lidar frames with a time difference less than a predetermined threshold are acquired. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information.

[0121] Based on compensation for the vehicle's own motion within the time difference, the image frame and the lidar frame are transformed into the same coordinate system;

[0122] Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization methods.

[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-described graph-optimized lane line multi-sensor fusion methods, the methods comprising:

[0124] Image frames and lidar frames with a time difference less than a predetermined threshold are acquired. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information.

[0125] Based on compensation for the vehicle's own motion within the time difference, the image frame and the lidar frame are transformed into the same coordinate system;

[0126] Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization methods.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion method for lane lines based on graph optimization, characterized in that, The method includes: Image frames and lidar frames with a time difference less than a predetermined threshold are acquired. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information. Based on compensation for the vehicle's own motion within the time difference, the image frame and the lidar frame are transformed into the same coordinate system; Based on image frames and LiDAR frames in the same coordinate system, the final lane lines are obtained through graph optimization methods. The final lane lines are obtained using a graph optimization method based on image frames and lidar frames in the same coordinate system, including: Based on image frames and LiDAR frames in the same coordinate system, the distance between the LiDAR lane lines and image lane lines in the obtained paired frames is calculated, and the matching relationship between the LiDAR lane lines and image lane lines is obtained by association matching through a preset algorithm. For lane lines that do not match successfully, use them as the final first part of the lane lines; For successfully matched lane lines, a graph optimization method is used to merge the successfully matched lane lines as the final second part of the lane lines.

2. The multi-sensor fusion method for lane lines based on graph optimization according to claim 1, characterized in that, Acquire image frames and LiDAR frames with a time difference less than a predetermined threshold, including: Traverse multiple buffered image frames and multiple buffered LiDAR frames in the image buffer and LiDAR buffer to form multiple sets of paired frames; Select the last pair of paired frames in time from the multiple pairs of paired frames, and use the image frames and lidar frames in it as the image frames and lidar frames whose time difference is less than a predetermined threshold.

3. The multi-sensor fusion method for lane lines based on graph optimization according to claim 1, characterized in that, Based on compensation for the vehicle's own motion during the time difference, the image frame and the lidar frame are transformed to the same coordinate system, including: Based on the intrinsic and extrinsic parameters of the camera, the image frame is projected from the two-dimensional coordinate system to the vehicle coordinate system; Based on compensation for the vehicle's own motion within the time difference, the lidar frame is converted into the vehicle coordinate system of the image frame.

4. The multi-sensor fusion method for lane lines based on graph optimization according to claim 1, characterized in that, For successfully matched lane lines, a graph optimization method is used to fuse the successfully matched lane lines into the final second part of the lane lines, including: For all points from the LiDAR frame in the successfully matched lane line, a first optimization is performed. The first optimization includes minimizing the error of a first variable and a second variable. The first variable includes the ordinate value of the point, and the second variable includes the ordinate value of the point's abscissa value on the lane line equation to be optimized.

5. The multi-sensor fusion method for lane lines based on graph optimization according to claim 4, characterized in that, For successfully matched lane lines, a graph optimization method is used to fuse the successfully matched lane lines into the final second part of the lane lines, including: For all points from the image frame in the successfully matched lane lines, a second optimization is performed. The second optimization includes minimizing the error of a third variable and a fourth variable. The third variable includes the ordinate value of the point, and the fourth variable includes the sum of a first sub-variable and a second sub-variable. The first sub-variable includes the ordinate value of the abscissa value of the point on the lane line equation to be optimized, and the second sub-variable includes the offset parameter when the image frame is projected from the two-dimensional coordinate system to the vehicle coordinate system.

6. The multi-sensor fusion method for lane lines based on graph optimization according to claim 1 or 4, characterized in that, For successfully matched lane lines, a graph optimization method is used to fuse the successfully matched lane lines into the final second part of the lane lines, including: For a successfully matched lane line, a third optimization is performed, which involves minimizing the constant term coefficient error and / or linear term coefficient error of the lane line equation to be optimized between mutually parallel lane lines.

7. A multi-sensor fusion system for lane lines based on graph optimization, characterized in that, The system for implementing the method according to any one of claims 1-6, the system comprising: The acquisition module is used to acquire image frames and lidar frames with a time difference of less than a predetermined threshold. The image frames are acquired by a camera on the vehicle, and the lidar frames are acquired by a lidar on the vehicle. Both the image frames and lidar frames include lane line information. The conversion module is used to convert the image frame and the lidar frame to the same coordinate system based on compensation for the vehicle's own motion within the time difference. The optimization module is used to obtain the final lane lines based on image frames and LiDAR frames in the same coordinate system using a graph optimization method.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the graph-optimized lane line multi-sensor fusion method as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the graph-optimized lane line multi-sensor fusion method as described in any one of claims 1-6.

Images