Lane line restoration method and system based on key point sequence

Abstract

The present invention discloses a lane line restoration method and system based on a key point sequence. By converting vector line elements into raster images, in the manner of image detection, key point sequences of lane perception markings in the images are extracted. Through the position matching and association of lane perception markings and key point sequences, all lane perception markings are divided into different lane line groups, and then the complete and comprehensive lane lines are obtained by complementing; the present invention borrows the idea of key point sequence detection to find the grouping matching relationship between perception markings, avoiding the loss of image pixel accuracy caused by directly using key point sequences, and finally accurately restoring the lane lines and avoiding the problem of lane grouping adhesion.

Description

Technical Field

[0001] The present invention relates to the technical field of high-precision map making, and particularly to a lane line restoration method and system based on a key point sequence. Background Art

[0002] When constructing lanes in a crowdsourced map, an important step is to restore linear elements such as lane markings, curbs, and guardrails extracted by vehicle-end perception to complete lane lines in the real world. In actual collection, due to reasons such as wear of the road surface lane markings themselves, occlusion by vehicles near the collection vehicle, and differences in perception algorithms, the extracted linear elements such as lane markings are usually discontinuous and fragmented. Therefore, how to match and group these fragmented linear elements and complete them into the original complete lane lines is an essential part of constructing lane-level elements in a crowdsourced map.

[0003] Regarding how to restore fragmented perceived lane markings to complete lane lines, traditional solutions are mainly implemented based on geometric calculations and statistics. Along the transverse direction perpendicular to the road direction, calculate the transverse distances of these perceived markings on the road surface, construct a similarity matrix based on the distance differences between each pair, and divide them into different lane line groups by means of DSBCAN clustering. For the perceived markings within the group, along the road direction, complete them by means of overall fitting or local interpolation to obtain multiple complete lane lines. The defect of this type of method is that it is very unsuitable for the lane change scenario, mainly manifested in that when lanes are added, reduced, or changed, it is difficult to accurately describe the transverse distances between pairwise perceived linear elements, and clustering with unreliable transverse distances will cause lane group adhesion. For example, the perceived markings corresponding to two lane lines during a lane change are divided into one group, so that complete and correct lane lines cannot be obtained. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above technical deficiencies and propose a lane line restoration method and system based on a key point sequence, which accurately groups all lane perceived markings based on the position matching and association between the lane perceived markings and the key point sequence, and solves the problem of existing lane group adhesion.

[0005] To achieve the above technical purpose, the first aspect of the technical solution of the present invention provides a lane line restoration method based on a key point sequence, which includes:

[0006] Obtain a lane vector image of a road information collection vehicle, where the lane vector image includes a vehicle driving trajectory and lane perceived markings;

[0007] Segment the lane vector image along the vehicle driving trajectory direction with a unit image size and convert the segmented image into a raster image;

[0008] Build a key point sequence detection model, and use the key point sequence detection model to predict the raster image to obtain the key point sequence of the lane perception markings in the raster image;

[0009] Use the predicted key point sequence as the matching association sequence between the lane perception markings, and match and group the lane perception markings according to different key point sequences;

[0010] Fit and complement the grouped lane perception markings to obtain a complete lane line.

[0011] The second aspect of the present invention provides a lane line restoration system based on a key point sequence, which includes the following functional modules:

[0012] An information acquisition module for acquiring the lane vector image of the road information collection vehicle, where the lane vector image contains the vehicle driving trajectory and lane perception markings;

[0013] An image segmentation module for segmenting the lane vector image in the direction of the vehicle driving trajectory at a unit image size and converting the segmented image into a raster image;

[0014] A key point prediction module for building a key point sequence detection model and using the key point sequence detection model to predict the raster image to obtain the key point sequence of the lane perception markings in the raster image;

[0015] A marking matching and grouping module for using the predicted key point sequence as the matching association sequence between the lane perception markings and matching and grouping the lane perception markings according to different key point sequences;

[0016] A marking fitting and complementing module for fitting and complementing the grouped lane perception markings to obtain a complete lane line.

[0017] The third aspect of the present invention provides a server, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned lane line restoration method based on a key point sequence is implemented.

[0018] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned lane line restoration method based on a key point sequence is implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The present invention converts vector line elements into raster images, extracts the key point sequence of lane perception markings in the image in the way of image detection, divides all lane perception markings into different lane line groups through the position matching and association between the lane perception markings and the key point sequence, and then completes the filling to obtain complete and comprehensive lane lines, so as to accurately restore the lane lines and avoid the problem of lane grouping adhesion.

[0021] The present invention replaces the DBSCAN clustering using geometric features in the traditional solution with a deep neural network model based on key point sequence detection to realize the classification and grouping of fragmented perception markings. Under the large amount of data in the crowdsourcing scenario, the mapping effect can be continuously iteratively optimized, and the scalability is stronger;

[0022] The present invention makes full use of the characteristics of rich trajectories in the crowdsourcing scenario. In the scenario where perception markings are missing, the deep neural network model is used to find the potential relationship between the trajectories and the markings, which plays a certain auxiliary role in the grouping of perception markings, and the adaptability and robustness are stronger;

[0023] The present invention borrows the idea of key point sequence detection to find the grouping matching relationship between perception markings, avoids the loss of image pixel accuracy caused by directly using the key point sequence, and finally fits and completes with the vector elements themselves, with higher accuracy. Description of the Drawings

[0024] Figure 1 is the flowchart of the lane line restoration method based on key point sequence according to the embodiment of the present invention;

[0025] Figure 2 is Figure 1 the sub-step flowchart of step S2 in

[0026] Figure 3 is the module block diagram of the lane line restoration system based on key point sequence according to the embodiment of the present invention. Detailed Embodiment

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, and are not used to limit the present invention.

[0028] The embodiment of the present invention provides a lane line restoration method based on key point sequence, as Figure 1 shown, which includes:

[0029] S1. Obtain the lane vector image of the road information collection vehicle, where the lane vector image includes vehicle driving trajectories and lane perception markings.

[0030] Among them, to facilitate the distinction between the vehicle driving trajectory and the lane perception markings, different colors are used to distinguish the vehicle driving trajectory and the lane perception markings in the lane vector image.

[0031] S2. Segment the lane vector image in the unit image size along the vehicle driving trajectory direction, and convert the segmented image into a raster image.

[0032] As Figure 2 shown, the step S2 specifically includes the following sub-steps:

[0033] S21. Set a sliding window with a unit image size, and set the single sliding distance of the sliding window.

[0034] S22. Take a point in the lane vector image as the origin of the sliding window, slide the sliding window in the lane vector image along the vehicle driving trajectory direction, and regard the image circled by each slide of the sliding window as the segmented image.

[0035] S23. Convert the segmented image into a raster image by the heat map method.

[0036] That is, set a sliding window with a single sliding distance of k, take a point along the vehicle driving trajectory direction as the origin of the current sliding window, take the length L in the vehicle driving trajectory direction from this origin as the length of the sliding window, take the total length W as the width of the sliding window with this origin as the midpoint in the horizontal direction, and convert the elements such as the perception markings and trajectories within the sliding window range into a raster image by the heat map method. The image pixel ratio is 1:10, that is, one pixel point is 10 cm, and the size of the raster image is 10L, 10W. The single sliding distance of the sliding window is less than its unit length in the vehicle driving trajectory direction, that is, L>k, ensuring that there is an overlapping area of L-k between adjacent sliding windows, which provides convenience for subsequent grouping fitting and complementing.

[0037] Considering the scenario where the lane perception markings are severely missing, the vehicle driving trajectory data within the sliding window range is used as supplementary reference, that is, the vehicle driving trajectory and the lane perception markings in the lane vector map are segmented in the unit image size along the driving trajectory direction, and are also converted into a raster image for use as the training image of the subsequent key point sequence detection model.

[0038] S3. Build a key point sequence detection model, and use the key point sequence detection model to predict the raster image to obtain the key point sequence of the lane perception markings in the raster image.

[0039] The key point sequence detection model is used to predict the key points (Kx, Ky) of the lane line on the raster image and their association relationship (dx, dy) with the starting point. The association relationship (dx, dy) between the key point (Kx, Ky) and the starting point is specifically the horizontal and vertical distance differences dx, dy between the starting point and the key point. When the key point sequence detection model is trained, there is such a distance difference relationship between the key points extracted by it and the starting point. Therefore, the key points with the above-mentioned association relationship with the starting point can be found through the key point sequence detection model.

[0040] Based on the key points predicted on the raster image, the starting point information of the lane line can be calculated inversely according to the association relationship (dx, dy) between the key point (Kx, Ky) and the starting point. Perform distance clustering on the starting point, that is, take the midpoint of the clustering of N approximately identical starting points as the starting point of the current lane line. Since different lane lines have different starting points, the corresponding key points are grouped and classified according to the starting points after clustering.

[0041] Specifically, the training set of the key point sequence detection model is made by using the existing high-precision map lane line data, that is, the existing high-precision map lane line data is segmented into images along the road trajectory direction with the unit image size, and the segmented images are converted into training set raster images. The training set raster images are used to pre-train the key point sequence detection model.

[0042] Taking the raster image converted from the lane vector image as the input image, use the trained key point sequence detection model to make predictions to obtain the key points in the raster image. Preferably, in the process of converting the lane vector image into a raster image, different colors are used to distinguish the vehicle driving trajectory and the lane perception marking line, so that in the area where the lane perception marking line is missing, the potential relationship between the vehicle driving trajectory and the lane perception marking line can be explored to improve the prediction accuracy and integrity of the key point sequence detection model.

[0043] Calculate the starting point information of the lane line inversely according to the key points, perform distance clustering on the starting point, group and classify the corresponding key points according to the starting points after clustering; and sort the key points in the direction of the vehicle driving trajectory, that is, in the direction from the bottom to the top of the image, to obtain an ordered key point sequence of multiple lane lines.

[0044] S4. Use the predicted key point sequence as the matching association sequence between the lane perception marking lines, and perform matching grouping on the lane perception marking lines according to different key point sequences.

[0045] Since converting vector data to raster images will result in certain accuracy losses, for example, the pixel point accuracy of 10 cm cannot accurately describe the specific accuracy information below 10 cm, there will be a certain deviation between the predicted key point sequence and the original lane line. Therefore, in the present invention, only the predicted key point sequence is used as the matching association sequence between the original perception markings, and based on its position matching relationship with the perception markings, the perception markings are divided into different lane line groups according to different key point sequences; that is, the lane perception markings corresponding to the lane line groups of the key points are grouped into the same lane line groups.

[0046] There is an overlapping area of (L - k) in the key point sequences within adjacent sliding windows. Through this overlapping area, it is possible to determine whether the lane perception markings in adjacent sliding windows are the same lane line. That is, if there is an overlapping relationship in the key point sequences within the overlapping area, it is determined that the lane perception markings in adjacent sliding windows are the same lane line, and the lane lines between adjacent sliding windows are grouped and fused; if there is no overlapping relationship or no key points in the key point sequences within the overlapping area, no fusion processing is performed.

[0047] S5. Fit and complete the grouped lane perception markings to obtain complete lane lines.

[0048] After grouping and fusing, there will be two forms of lane markings within a single group. One is that there may be multiple perception markings collected in multiple trips perpendicular to the road driving direction, and in this case, curve fitting can be performed by the least squares method to obtain the best lane line; the other is that there is a missing part along the road driving direction, that is, there is no perception marking. In this case, reasonable inference and filling can be performed based on the lane lines that have been fitted before and after the missing area. At this time, the trajectory form of the missing area can be comprehensively referred to, and the markings in the missing area can be filled in the most likely form and maintained in a continuous relationship with the lane lines that have been fitted before and after. In this way, all groups are traversed and processed to obtain multiple relatively complete lane lines within the road.

[0049] In the present invention, by converting vector line elements into raster images and extracting the key point sequences of lane perception markings in the images in the manner of image detection, all lane perception markings are divided into different lane line groups through the position matching and association between the lane perception markings and the key point sequences, and then the complete and comprehensive lane lines are obtained through fitting. In the present invention, a deep neural network model based on key point sequence detection is used to replace the DBSCAN clustering using geometric features in the traditional solution to realize the classification and grouping of fragmented perception markings. Under the large amount of data in the crowdsourcing scenario, the mapping effect can be continuously iteratively optimized, and the scalability is stronger. Moreover, the characteristics of rich trajectories in the crowdsourcing scenario are fully utilized. In the scenario where perception markings are missing, the deep neural network model is used to find the potential relationship between the trajectories and the markings, which plays a certain auxiliary role in the grouping of perception markings, and the adaptability and robustness are stronger. At the same time, the idea of key point sequence detection is used to find the grouping matching relationship between perception markings, avoiding the loss of image pixel accuracy caused by directly using key point sequences, and finally fitting and complementing with the vector elements themselves, with higher accuracy.

[0050] As Figure 3 shown, an embodiment of the present invention also discloses a lane line restoration system based on key point sequences, which includes the following functional modules:

[0051] An information acquisition module 10, configured to acquire a lane vector image of a road information collection vehicle, where the lane vector image includes vehicle driving trajectories and lane perception markings;

[0052] An image segmentation module 20, configured to segment the lane vector image in the vehicle driving trajectory direction at a unit image size and convert the segmented image into a raster image;

[0053] A key point prediction module 30, configured to build a key point sequence detection model and use the key point sequence detection model to predict the raster image to obtain the key point sequences of the lane perception markings in the raster image;

[0054] A marking matching and grouping module 40, configured to use the predicted key point sequences as the matching and association sequences between lane perception markings, and perform matching and grouping on lane perception markings according to different key point sequences;

[0055] A marking fitting and complementing module 50, configured to fit and complement the grouped lane perception markings to obtain complete lane lines.

[0056] The execution manner of the lane line restoration system based on key point sequences in this embodiment is basically the same as the above-mentioned lane line restoration method based on key point sequences, so no detailed description will be given here.

[0057] The server in this embodiment is a device that provides computing services, usually referring to a computer with high computing power that is provided to multiple consumers through a network. The server in this embodiment includes: a memory, a processor, and a system bus. The memory includes a program that can run thereon. Those skilled in the art can understand that the structure of the terminal device in this embodiment does not constitute a limitation on the terminal device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0058] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing of the terminal by running the software programs and modules stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the terminal (such as audio data, a phone book, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0059] A runnable program of a lane line restoration method based on a key point sequence is included in the memory. The runnable program can be divided into one or more modules / units. The one or more modules / units are stored in the memory and executed by the processor to complete the process of information acquisition and implementation. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and these instruction segments are used to describe the execution process of the computer program in the server. For example, the computer program can be divided into an information acquisition module 10, an image segmentation module 20, a key point prediction module 30, a marking matching and grouping module 40, and a marking fitting and complementing module 50.

[0060] The processor is the control center of the server, connecting various parts of the entire terminal device through various interfaces and lines. By running or executing the software programs and / or modules stored in the memory, and calling the data stored in the memory, it executes various functions of the terminal and processes data, thereby monitoring the terminal as a whole. Optionally, the processor may include one or more processing units; preferably, the processor may integrate an application processor and a modulation and demodulation processor. Among them, the application processor mainly processes the operating system, application programs, etc., and the modulation and demodulation processor mainly processes wireless communication. It can be understood that the above modulation and demodulation processor may not be integrated into the processor either.

[0061] The system bus is used to connect various functional components inside a computer and can transmit data information, address information, and control information. Its types can be, for example, PCI bus, ISA bus, VESA bus, etc. The instructions of the processor are transmitted to the memory through the bus, and the memory feeds back data to the processor. The system bus is responsible for the data and instruction interaction between the processor and the memory. Of course, the system bus can also connect to other devices, such as network interfaces, display devices, etc.

[0062] The server should at least include a CPU, chipset, memory, disk system, etc. Other components will not be elaborated here.

[0063] In the embodiment of the present invention, the executable program executed by the processor included in the terminal is specifically: a lane line restoration method based on a key point sequence, which includes the following steps:

[0064] Obtain the lane vector image of the road information collection vehicle, where the lane vector image contains the vehicle driving trajectory and lane perception markings;

[0065] Segment the lane vector image along the vehicle driving trajectory direction with a unit image size and convert the segmented image into a raster image;

[0066] Build a key point sequence detection model, and use the key point sequence detection model to predict the raster image to obtain the key point sequence of the lane perception markings in the raster image;

[0067] Take the predicted key point sequence as the matching association sequence between the lane perception markings, and group the lane perception markings according to different key point sequences;

[0068] Fit and complete the grouped lane perception markings to obtain a complete lane line.

[0069] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0070] In the above embodiments, the descriptions of each embodiment have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0071] Those of ordinary skill in the art can realize that the modules, units, and / or method steps of the various embodiments described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0072] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A lane line restoration method based on a key point sequence, characterized in that, Including: Obtain the lane vector image of the road information collection vehicle, where the lane vector image contains the vehicle driving trajectory and lane perception markings; Segment the lane vector image in the direction of the vehicle driving trajectory with a unit image size, and convert the segmented image into a raster image; Build a key point sequence detection model, and use the key point sequence detection model to predict the raster image to obtain the key point sequence of the lane perception markings in the raster image; Take the predicted key point sequence as the matching association sequence between the lane perception markings, and group the lane perception markings according to different key point sequences; Fit and complete the grouped lane perception markings to obtain a complete lane line; The step of segmenting the lane vector image in the direction of the vehicle driving trajectory with a unit image size and converting the segmented image into a raster image specifically includes: Set a sliding window with a unit image size, and set the single sliding distance of the sliding window; Take a point in the lane vector image as the origin of the sliding window, slide the sliding window in the lane vector image in the direction of the vehicle driving trajectory, and take the image enclosed by the sliding window each time it slides as the segmented image; Convert the segmented image into a raster image by the heat map method.

2. The lane line restoration method based on a key point sequence according to claim 1, wherein Distinguish the vehicle driving trajectory and lane perception markings in the lane vector image with different colors.

3. The lane line restoration method based on the key point sequence according to claim 1, characterized in that, The step of segmenting the lane vector image in the direction of the vehicle driving trajectory with a unit image size specifically includes: segmenting the vehicle driving trajectory and lane perception markings in the lane vector image in the direction of the driving trajectory with a unit image size.

4. The lane line restoration method based on the key point sequence according to claim 1, wherein There is a partial overlapping area between adjacent sliding windows.

5. The lane line restoration method based on a key point sequence according to claim 4, characterized in that Group and fuse the lane lines between adjacent sliding windows according to the key point sequence overlapping relationship in the overlapping area.

6. The lane line restoration method based on the key point sequence according to claim 1, characterized in that The step of fitting and completing the grouped lane perception markings to obtain a complete lane line specifically includes: For multiple lane perception markings existing in the direction perpendicular to the road driving direction, perform curve fitting by the least squares method to obtain the best lane line; For the missing lane perception markings existing in the direction along the road driving direction, supplement the missing lane perception markings with reference to the trajectory form of the missing area.

7. A lane line restoration system based on a key point sequence, characterized in that, Including the following functional modules: An information acquisition module, which is used to obtain the lane vector image of the road information collection vehicle, where the lane vector image contains the vehicle driving trajectory and lane perception markings; An image segmentation module, which is used to segment the lane vector image in the direction of the vehicle driving trajectory with a unit image size and convert the segmented image into a raster image; A key point prediction module, which is used to build a key point sequence detection model and use the key point sequence detection model to predict the raster image to obtain the key point sequence of the lane perception markings in the raster image; A marking matching and grouping module, which is used to take the predicted key point sequence as the matching association sequence between the lane perception markings and group the lane perception markings according to different key point sequences; A marking fitting and completing module, which is used to fit and complete the grouped lane perception markings to obtain a complete lane line; Segment the lane vector image in the direction of the vehicle driving trajectory at the unit image size, and convert the segmented image into a raster image; specifically including: Set a sliding window with a unit image size, and set the single sliding distance of the sliding window; Take a point in the lane vector image as the origin of the sliding window, slide the sliding window in the lane vector image along the vehicle driving trajectory direction, and take the image enclosed by each sliding of the sliding window as the segmented image; Convert the segmented image into a raster image by the heat map method.

8. A server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the lane line restoration method based on the key point sequence according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the lane line restoration method based on the key point sequence according to any one of claims 1 to 6.

Images