A lane line tracking method and a lane line tracking system

Abstract

The application provides a lane line tracking method and a lane line tracking system, and relates to the technical field of automatic driving. The application predicts the actual lane line detection result of the last frame by using a CNN-LSTM model to obtain the first prediction result of the lane line of the current frame, uses the advantage of feature extraction of the CNN network to extract the feature of the lane line detection result, then inputs the extracted feature vector into the LSTM network to predict the lane line, and combines the CNN network with the LSTM network to newly add a convolution layer and a pooling layer in the original LSTM network, so that the prediction accuracy of the lane line is improved. Then, the actual lane line detection result of the current frame and the first prediction result of the lane line of the current frame are associated by using the Hungarian algorithm to obtain a first set, and fitting is performed on the first set, so that the lane line tracking result is obtained, the predicted lane line and the actually detected lane line are fused into a more accurate lane line, and the lane line tracking effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a lane line tracking method and a lane line tracking system. BACKGROUND

[0002] Lane line tracking technology is one of the important functions of the perception module in automatic driving technology, and plays an important role in the process of automatic driving. Due to the complexity of the road environment, higher requirements are put forward for lane line tracking technology. In addition, in order to protect the safety of the driver, the lane line tracking technology needs to achieve a certain real-time and accuracy before it can be used. Kalman filtering is a traditional lane line tracking technology. The method of Kalman filtering is to predict the position of the next frame of lane line according to the short-term historical motion state of the lane line. This method has good tracking effect for relatively ideal lane lines. For lane lines that are interrupted, blocked or blurred, or in the case that the motion state of the vehicle changes greatly, the accuracy of the lane line prediction result obtained by the existing method is poor, resulting in poor lane line tracking effect. SUMMARY

[0003] One object of the first aspect of the present application is to provide a lane line tracking method to solve the technical problem of poor lane line prediction accuracy in the prior art.

[0004] Another object of the first aspect of the present application is to improve the stability of lane line tracking.

[0005] An object of the second aspect of the present application is to provide a lane line tracking system.

[0006] According to the object of the first aspect of the present application, the present application provides a lane line tracking method, comprising the following steps:

[0007] Obtaining the actual lane line detection result of the previous frame and the actual lane line detection result of the current frame;

[0008] Using a CNN-LSTM model to predict the actual lane line detection result of the previous frame to obtain a first prediction result of the lane line of the current frame;

[0009] Using the Hungarian algorithm to associate the actual lane line detection result of the current frame and the first prediction result of the lane line of the current frame to obtain a first association result, the first association result comprising a first set of matched actual lane line detection results of the current frame and first prediction results of the lane line of the current frame;

[0010] Fitting the first set to obtain a lane line tracking result.

[0011] Optionally, the first association result further comprises a second set of the actual lane line detection results of the current frame which are not matched and a third set of the lane line first prediction results of the current frame which are not matched.

[0012] Optionally, the step of associating the actual lane line detection results of the current frame and the lane line first prediction results of the current frame by using the Hungarian algorithm to obtain a first association result comprises the following steps:

[0013] Calculating an intersection-over-union value according to the actual lane line detection results of the current frame and the lane line first prediction results of the current frame.

[0014] Taking the actual lane line detection results of the current frame and the lane line first prediction results of the current frame corresponding to a first target intersection-over-union value as the first set, the first target intersection-over-union value being the intersection-over-union value greater than a preset value.

[0015] Optionally, the step of calculating an intersection-over-union value according to the actual lane line detection results of the current frame and the lane line first prediction results of the current frame further comprises the following steps:

[0016] Taking the actual lane line detection results of the current frame corresponding to a second target intersection-over-union value as the second set and the lane line first prediction results of the current frame corresponding to the second target intersection-over-union value as the third set, the second target intersection-over-union value being the intersection-over-union value less than or equal to the preset value.

[0017] Optionally, the step of taking the actual lane line detection results of the current frame corresponding to a second target intersection-over-union value as the second set and the lane line first prediction results of the current frame corresponding to the second target intersection-over-union value as the third set further comprises the following steps:

[0018] If the actual lane line detection result appears in the second set for a continuous preset time length, the actual lane line detection result is deleted.

[0019] Optionally, the step of taking the actual lane line detection results of the current frame corresponding to a second target intersection-over-union value as the second set and the lane line first prediction results of the current frame corresponding to the second target intersection-over-union value as the third set further comprises the following steps:

[0020] Initializing the lane line first prediction results in the third set.

[0021] Optionally, the step of calculating an intersection-over-union value according to the actual lane line detection results of the current frame and the lane line first prediction results of the current frame comprises the following steps:

[0022] extend each point in the actual lane line detection result of the current frame leftward and rightward by a preset length to obtain two first edge points respectively, and extend each point in the lane line first prediction result of the current frame leftward and rightward by a preset length to obtain two second edge points respectively;

[0023] connect all the first edge points in sequence to obtain a first region, and connect all the second edge points in sequence to obtain a second region;

[0024] calculate a ratio of an intersection region and a union region of the first region and the second region, thereby obtaining the intersection-union ratio.

[0025] Optionally, the steps of obtaining the actual lane line detection result of the previous frame and the actual lane line detection result of the current frame further include the following steps:

[0026] predict the actual lane line detection result of the previous frame by using an extended Kalman filtering algorithm to obtain a lane line second prediction result of the current frame;

[0027] associate the actual lane line detection result of the current frame and the lane line second prediction result of the current frame by using a Hungarian algorithm to obtain a second association result, the second association result including a fourth set of matched actual lane line detection result of the current frame and lane line second prediction result of the current frame;

[0028] fit the first set and the fourth set to obtain a lane line tracking result.

[0029] Optionally, the second association result further includes a fifth set of actual lane line detection result of the current frame that is not matched and a sixth set of lane line second prediction result of the current frame that is not matched.

[0030] According to the second aspect of the present application, the present application further provides a lane line tracking system, comprising:

[0031] a control module, the control module including a memory and a processor, the memory storing a calculation program, the calculation program being executed by the processor to implement the above-mentioned tracking method.

[0032] The application utilizes the CNN-LSTM model to predict the actual lane line detection result of the previous frame to obtain the lane line first prediction result of the current frame, utilizes the advantage of feature extraction of the CNN network, uses the CNN network for feature extraction of the lane line detection result, then inputs the extracted feature vector into the LSTM network to predict the lane line, the above technical solution combines the CNN network with the LSTM network, newly adds the convolution layer and the pooling layer in the original LSTM network, thereby improving the prediction accuracy of the lane line. Then the Hungarian algorithm is used to associate the actual lane line detection result of the current frame and the lane line first prediction result of the current frame, a first set of the matched actual lane line detection result of the current frame and the lane line first prediction result of the current frame is obtained, and the first set is fitted, thereby obtaining the lane line tracking result, the predicted lane line and the actually detected lane line are fused into a more accurate lane line, thereby improving the lane line tracking effect.

[0033] Further, the application not only utilizes the CNN-LSTM model to predict the lane line first prediction result of the current frame from the actual lane line detection result of the previous frame, but also utilizes the extended Kalman filtering algorithm to predict the lane line second prediction result of the current frame from the actual lane line detection result of the previous frame, then utilizes the Hungarian algorithm to associate the two respectively, thereby obtaining the first set and the fourth set, and fitting the first set and the fourth set to obtain the lane line tracking result, thereby improving the stability of the lane line tracking.

[0034] The above and other objects, advantages and features of the application will become more apparent from the following detailed description of some embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] Some specific embodiments of the application will be described in detail below with reference to the attached drawings. The same reference numerals in the drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that the drawings are not necessarily drawn to scale. In the drawings:

[0036] Figure 1 is a schematic flow chart of a lane line tracking method according to one embodiment of the application;

[0037] Figure 2 is a schematic flow chart of a lane line tracking method according to another embodiment of the application;

[0038] Figure 3 is a schematic flow chart of a lane line tracking method according to still another embodiment of the application;

[0039] Figure 4is a schematic diagram of an intersection region of a first region and a second region according to one embodiment of the present application;

[0040] Figure 5 is a schematic diagram of a union region of a first region and a second region according to one embodiment of the present application;

[0041] Figure 6 is a schematic flow chart of a lane line tracking method according to another embodiment of the present application;

[0042] Figure 7 is a schematic connection block diagram of a lane line tracking system according to one embodiment of the present application.

[0043] Reference Signs:

[0044] 100 - lane line tracking system, 10 - control module, 11 - memory, 12 - processor, 30 - actual lane line, 40 - predicted lane line. DETAILED DESCRIPTION

[0045] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which like or similar elements or elements having the same or similar functions are denoted by the same or similar reference signs throughout the drawings. The embodiments described below by way of example are intended to explain the present application, and are not to be understood as limiting the present application.

[0046] Figure 1 is a schematic flow chart of a lane line tracking method according to one embodiment of the present application. As shown in Figure 1 in one specific embodiment, the lane line tracking method comprises the following steps:

[0047] Step S100, obtaining actual lane line detection results of a previous frame and actual lane line detection results of a current frame;

[0048] Step S200, predicting the actual lane line detection results of the previous frame using a CNN-LSTM model to obtain a first predicted result of the lane line of the current frame;

[0049] Step S300, correlating the actual lane line detection results of the current frame and the first predicted result of the lane line of the current frame using a Hungarian algorithm to obtain a first correlation result, the first correlation result including a first set of matched actual lane line detection results of the current frame and the first predicted result of the lane line of the current frame;

[0050] Step S400, fitting the first set to obtain a lane line tracking result.

[0051] In step S200, the CNN network in the CNN-LSTM model represents a convolutional neural network (CNN), and the LSTM network represents a long-short-term memory (LSTM).

[0052] The embodiment aims to solve the problem that the LSTM network is weak in feature extraction of lane line position change rules, and uses the advantage of the CNN network in feature extraction, uses the CNN network to extract features from the lane line detection result, and then inputs the extracted feature vector into the LSTM network to predict the lane line. The embodiment combines the CNN network with the LSTM network, adds a convolution layer and a pooling layer to the original LSTM network, thereby improving the prediction accuracy of the lane line.

[0053] The embodiment fuses the predicted lane line and the actually detected lane line into a more accurate lane line, thereby improving the lane line tracking effect.

[0054] In the embodiment, the number of convolution kernels in the convolution layer of the CNN network is n = 64, the convolution kernel width is FH = 2, the padding boundary is P = 0, the convolution kernel step is S = 1, the pooling window is w = 1, the activation function uses a ReLU function, the number of layer neurons in the LSTM network is n = 32, the loss function of the model is a mean square error loss function, the optimization function is an Adam function, the batch normalization quantity is Batch size = 10, and the iteration number epoch is set to 200.

[0055] In the embodiment, the first association result further includes a second set of actual lane line detection results of the current frame that are not matched and a third set of lane line first prediction results of the current frame that are not matched. It can be understood that the association result obtained through the Hungarian algorithm includes three sets, and the embodiment selects the set of actual lane line detection results of the current frame and the lane line first prediction results of the current frame in the three sets to fit, thereby improving the lane line tracking effect.

[0056] Figure 2 is a schematic flowchart of a lane line tracking method according to another embodiment of the application. As shown in Figure 2 In the embodiment, step S300 specifically includes the following steps:

[0057] Step S310, calculating an intersection-over-union value according to the actual lane line detection result of the current frame and the lane line first prediction result of the current frame;

[0058] Step S320, taking the actual lane line detection result of the current frame and the lane line first prediction result of the current frame corresponding to the first target intersection-over-union value as a first set, and the first target intersection-over-union value is an intersection-over-union value greater than a preset value.

[0059] Step S330: The actual lane detection results of the current frame corresponding to the second target intersection-union ratio are used as the second set, and the first lane prediction results of the current frame corresponding to the second target intersection-union ratio are used as the third set. The second target intersection-union ratio is the intersection-union ratio that is less than or equal to a preset value. Steps S320 and S330 are not sequential.

[0060] Here, using The detection result of the i-th lane line in frame t is represented by... This represents the first prediction result for the i-th lane in frame t, if the intersection-merge ratio is... When this happens, the lane line matching result will be... Where k represents the tracking result of the k-th lane line, and the set of lane line matching results for the t-th frame is denoted as P. t , which is the first set, where A represents the preset value.

[0061] In this embodiment, after step S330, the following step is also included:

[0062] If an actual lane line detection result appears in the second set within a consecutive preset time period, then that actual lane line detection result is deleted. This can be understood as follows: if the current actual lane line detection result matches the first lane line prediction result, then if, within the subsequent consecutive preset time period, the actual lane line detection result does not match the first lane line prediction result at any given moment, then that actual lane line detection result is deleted. The preset time period can be set to any value between 3 and 6 frames, for example, 3 frames, 5 frames, or 6 frames. In a preferred embodiment, the preset time period is 5 frames.

[0063] In this embodiment, after step S330, the following step is also included:

[0064] The first lane prediction results in the third set are initialized. Specifically, if the first lane prediction results appear consecutively within a preset time period, then the first lane prediction results are used as a new lane line. The preset time period can be set to any value between 2 and 4 frames, for example, 2, 3, or 4 frames. In a preferred embodiment, the preset time period is 3 frames.

[0065] In this embodiment, before step S100, the following steps are also included:

[0066] If the actual lane line detection result appears for the first time, the actual lane line detection result is initialized, and only when the actual lane line detection result appears continuously for a preset time length, the next frame of the actual lane line detection result is predicted. The preset time length can be set to any value between 2 frames and 4 frames, for example, 2 frames, 3 frames or 4 frames, etc. In a preferred embodiment, the preset time length is 3 frames.

[0067] Figure 3 is a schematic flow chart of a lane line tracking method according to yet another embodiment of the present application, Figure 4 is a schematic diagram of an intersection region of a first region and a second region according to an embodiment of the present application, Figure 5 is a schematic diagram of a union region of a first region and a second region according to an embodiment of the present application. As Figures 3 to 5 shown, in this embodiment, step S310 further comprises the following steps:

[0068] Step S311, respectively extending the points in the actual lane line detection result of the current frame by a preset length to the left and to the right to obtain two first edge points, and respectively extending each point in the first predicted lane line result of the current frame by a preset length to the left and to the right to obtain two second edge points;

[0069] Step S312, sequentially connecting all the first edge points to obtain a first region, and sequentially connecting all the second edge points to obtain a second region;

[0070] Step S313, calculating the intersection region and the union region of the first region and the second region, and obtaining an intersection-union ratio. Here, the intersection region and the union region of the first region and the second region are respectively shown in Figure 4 and Figure 5 , the actual lane line 30 has a first region, the predicted lane line 40 has a second region, and both have an intersection region and a union region.

[0071] Figure 6 is a schematic flow chart of a lane line tracking method according to yet another embodiment of the present application. As Figure 6 shown, in this embodiment, after step S100, the following steps are further included:

[0072] Step S210, predicting the actual lane line detection result of the previous frame by using an extended Kalman filtering algorithm to obtain a second predicted lane line result of the current frame;

[0073] In step S220, the actual lane line detection result of the current frame and the second lane line prediction result of the current frame are associated by using the Hungarian algorithm to obtain a second association result, and the second association result includes a fourth set of matched actual lane line detection result of the current frame and second lane line prediction result of the current frame.

[0074] In step S400', the first set and the fourth set are fitted to obtain the lane line tracking result. Here, there is no sequence relationship between step S210 and step S200.

[0075] In step S210, the lane line is predicted by using the extended Kalman filter algorithm, which has a nonlinear fitting capability, so that the lane line tracking result is more accurate and stable.

[0076] The embodiment not only uses the CNN-LSTM model to predict the first lane line prediction result of the current frame from the actual lane line detection result of the previous frame, but also uses the extended Kalman filter algorithm to predict the second lane line prediction result of the current frame from the actual lane line detection result of the previous frame, and then uses the Hungarian algorithm to associate the two results respectively, so as to obtain the first set and the fourth set, and fit the first set and the fourth set to obtain the lane line tracking result, thereby improving the stability of the lane line tracking. Here, the formation condition of the fourth set is the same as that of the first set, that is, the intersection-over-union value is calculated according to the actual lane line detection result of the current frame and the second lane line prediction result of the current frame, and then the actual lane line detection result of the current frame corresponding to the third target intersection-over-union value and the second lane line prediction result of the current frame are taken as the fourth set.

[0077] In this embodiment, the second association result further includes a fifth set of actual lane line detection results of the current frame that are not matched and a sixth set of second lane line prediction results of the current frame that are not matched. Here, the formation condition of the fifth set is the same as that of the second set, and the processing manner of the fifth set is the same as that of the second set, the formation condition of the sixth set is the same as that of the third set, and the processing manner of the sixth set is the same as that of the third set. That is, the actual lane line detection result of the current frame corresponding to the fourth target intersection-over-union value is taken as the fifth set, and the second lane line prediction result of the current frame corresponding to the fourth target intersection-over-union value is taken as the sixth set, and the fourth target intersection-over-union value is an intersection-over-union value less than or equal to a preset value.

[0078] The embodiment directly predicts the lane line detection result by using the CNN-LSTM model, which greatly improves the lane line tracking speed. Due to the powerful nonlinear fitting capability of the LSTM network, the lane line tracking result is more accurate and stable.

[0079] Figure 7is a schematic connection block diagram of a lane line tracking system according to an embodiment of the present application. As Figure 7 shown, in this embodiment, the lane line tracking system 100 comprises a control module 10, which comprises a memory 11 and a processor 12, the memory 11 storing therein a calculation program for implementing the above-mentioned tracking method when executed by the processor 12. The processor 12 can be a central processing unit (CPU) or a digital processing unit, etc. The processor 12 transmits and receives data through a communication interface. The memory 11 is used to store the program executed by the processor 12. The memory 11 is any medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, and can also be a combination of a plurality of memories 11. The above-mentioned calculation program can be downloaded from a computer readable storage medium to a corresponding calculation / processing device or to a computer or an external storage device via a network (such as the Internet, a local area network, a wide area network, and / or a wireless network). Here, the control module 10 can be a body controller.

[0080] For the description of this embodiment, the "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device. More specific examples (non-exhaustive list) of computer readable medium include the following: electrical connections having one or more wires (electronic devices), portable computer diskettes (magnetic devices), random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memories), fiber optic devices, and portable compact disk read-only memories (CDROMs). In addition, the computer readable medium can even be paper or other suitable medium on which the program is printed, as the program can be electronically obtained, for example by optical scanning of the paper or other medium, followed by electronic conversion of the obtained program into a computer readable medium, and then storing the program in the computer memory 11.

[0081] So far, those skilled in the art should recognize that although the present application has been shown and described in detail in several exemplary embodiments, many other variations or modifications can be directly determined or deduced according to the disclosure of the present application without departing from the spirit and scope of the present application, which conform to the principles of the present application. Therefore, the scope of the present application should be understood and recognized as covering all these other variations or modifications.

Claims

1. A lane line tracking method, characterized in that, Includes the following steps: Obtain the actual lane line detection results of the previous frame and the actual lane line detection results of the current frame; The CNN-LSTM model is used to predict the actual lane line detection results of the previous frame to obtain the first predicted lane line result of the current frame. The Hungarian algorithm is used to associate the actual lane detection result and the first lane prediction result of the current frame to obtain a first association result. The first association result includes a first set of matched actual lane detection results and first lane prediction results of the current frame. The first set is fitted to obtain the lane line tracking results; The steps of obtaining the actual lane line detection results of the previous frame and the actual lane line detection results of the current frame are followed by the following steps: The extended Kalman filter algorithm is used to predict the actual lane line detection results of the previous frame to obtain the second lane line prediction result of the current frame. The Hungarian algorithm is used to associate the actual lane detection result and the second lane prediction result of the current frame to obtain a second association result. The second association result includes a fourth set of matched actual lane detection results and second lane prediction results of the current frame. By fitting the first set and the fourth set, the lane line tracking results are obtained.

2. The lane line tracking method according to claim 1, characterized in that, The first association result also includes a second set of actual lane line detection results for unmatched current frames and a third set of lane line first prediction results for unmatched current frames.

3. The lane line tracking method according to claim 2, characterized in that, The step of associating the actual lane detection result of the current frame with the first lane prediction result of the current frame using the Hungarian algorithm to obtain the first association result specifically includes the following steps: The intersection-union ratio is calculated based on the actual lane detection result of the current frame and the first lane prediction result of the current frame; The first set is the actual lane detection result of the current frame and the first lane prediction result of the current frame corresponding to the first target intersection-merge ratio, wherein the first target intersection-merge ratio is the intersection-merge ratio that is greater than a preset value.

4. The lane line tracking method according to claim 3, characterized in that, The step of calculating the intersection-merge ratio based on the actual lane detection result and the first lane prediction result of the current frame further includes the following steps: The actual lane detection result of the current frame corresponding to the second target intersection-merge ratio is taken as the second set, and the first lane prediction result of the current frame corresponding to the second target intersection-merge ratio is taken as the third set. The second target intersection-merge ratio is the intersection-merge ratio that is less than or equal to the preset value.

5. The lane line tracking method according to claim 4, characterized in that, The step of taking the actual lane detection result of the current frame corresponding to the second target intersection-union ratio as the second set and the first lane prediction result of the current frame corresponding to the third set, is followed by the following steps: If the actual lane line detection result appears in the second set for a consecutive preset time period, then the actual lane line detection result is deleted.

6. The lane line tracking method according to claim 5, characterized in that, The step of taking the actual lane detection result of the current frame corresponding to the second target intersection-union ratio as the second set and the first lane prediction result of the current frame corresponding to the third set, is followed by the following steps: The first prediction result of the lane line in the third set is initialized.

7. The lane tracking method according to any one of claims 3-6, characterized in that, The step of calculating the intersection-merge ratio based on the actual lane detection result of the current frame and the first lane prediction result of the current frame specifically includes the following steps: The points in the actual lane detection results of the current frame are extended to the left and right by a preset length to obtain two first edge points respectively, and each point in the first lane prediction results of the current frame is extended to the left and right by a preset length to obtain two second edge points respectively. Connect all the first edge points sequentially to obtain the first region, and connect all the second edge points sequentially to obtain the second region; The intersection and union regions of the first region and the second region are calculated to obtain the intersection-union ratio.

8. The lane line tracking method according to claim 1, characterized in that, The second association result also includes a fifth set of actual lane line detection results for unmatched current frames and a sixth set of second lane line prediction results for unmatched current frames.

9. A lane line tracking system, characterized in that, include: A control module, comprising a memory and a processor, wherein the memory stores a calculation program, which, when executed by the processor, is used to implement the lane tracking method according to any one of claims 1-8.

Images