Method and device for realizing lane line tracing based on image recognition

Abstract

The invention discloses a method and a device for realizing lane line tracing based on image recognition. The method comprises the following steps: acquiring a lane image training set; building a deep learning model for detecting a lane line; training the deep learning model through the lane image training set to obtain a lane line recognition model; acquiring a real-time image, identifying a lane line in the real-time image through a lane line identification model, and acquiring a depth coordinate of the lane line; obtaining current operation data of the vehicle, and obtaining target operation data and target coordinate points of the vehicle according to the depth coordinates and the current operation data; and controlling the vehicle to run to the target coordinate point according to the vehicle target operation data. After the depth coordinates of the lane line are obtained through an image processing method, the operation data of the vehicle at the next moment are adjusted in combination with the current operation data of the vehicle, that is, the detected lane line information can be calculated and converted into the instruction needing to be executed by vehicle operation in real time, and therefore the lane line tracing function of the vehicle is rapidly and effectively achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of lane line recognition, and in particular to a method and device for lane line tracking based on image recognition. Background Art

[0002] Lane detection technology is a key technology in the field of autonomous driving. It uses image recognition to obtain lane information and plan driving paths. Lane detection technology can guide vehicles to the correct driving area, providing a basis for autonomous driving behaviors such as automatic cruise control, lane keeping, and lane overtaking. It also provides drivers with warnings when the vehicle deviates from its lane, promoting safe driving.

[0003] However, most existing lane line detection technologies extract lane lines based on machine learning and plan the drivable area, and are unable to achieve tracking based on lane lines. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for lane line tracking based on image recognition, so as to control the vehicle to track along the lane.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for lane line tracking based on image recognition, comprising: Get the lane image training set; Build a deep learning model for lane detection; Training the deep learning model using the lane image training set to obtain a lane line recognition model; Acquire a real-time image, identify lane lines in the real-time image using the lane line recognition model, and obtain depth coordinates of the lane lines; Acquire the current running data of the vehicle, and obtain the target running data and target coordinate point of the vehicle according to the depth coordinate and the current running data; The vehicle is controlled to travel to the target coordinate point according to the vehicle target operation data.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A device for lane line tracking based on image recognition includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned method for lane line tracking based on image recognition is implemented.

[0007] The beneficial effects of the present invention are: a lane line recognition model is obtained by training a deep learning model based on a lane image training set, so that the lane line recognition model has strong generalization and stability. After obtaining the depth coordinates of the lane line through an image processing method, the vehicle's operating data at the next moment is adjusted in combination with the vehicle's current operating data. That is, the detected lane line information can be calculated and converted into instructions required for vehicle operation in real time, thereby quickly and effectively realizing the vehicle's lane line tracking function. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a flowchart of a method for lane line tracking based on image recognition in an embodiment of the present invention; Figure 2 This is a flowchart of another step of a method for lane line tracking based on image recognition in an embodiment of the present invention; Figure 3 The figure is a schematic structural diagram of a device for lane line tracking based on image recognition in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0010] A method for lane line tracking based on image recognition, comprising: Get the lane image training set; Build a deep learning model for lane detection; Training the deep learning model using the lane image training set to obtain a lane line recognition model; Acquire a real-time image, identify lane lines in the real-time image using the lane line recognition model, and obtain depth coordinates of the lane lines; Acquire the current running data of the vehicle, and obtain the target running data and target coordinate point of the vehicle according to the depth coordinate and the current running data; The vehicle is controlled to travel to the target coordinate point according to the vehicle target operation data.

[0011] From the above description, it can be seen that the beneficial effect of the present invention is that: the lane line recognition model is obtained by training the deep learning model based on the lane image training set, so that the lane line recognition model has strong generalization and stability. After obtaining the depth coordinates of the lane line through the image processing method, the vehicle's operating data at the next moment is adjusted in combination with the vehicle's current operating data, that is, the detected lane line information can be calculated and converted into instructions required to be executed for vehicle operation in real time, thereby quickly and effectively realizing the vehicle's lane line tracking function.

[0012] Furthermore, the deep learning model includes an image processing module, a feature extraction module and a post-processing module; The training of the deep learning model using the lane image training set includes: Adjusting the lane image training set to a preset size through the normalization layer and the convolution layer of the image processing module in sequence; Extracting image features from the image output by the image processing module by the feature extraction module; The post-processing module extracts lane line prediction key points based on the output results of the feature extraction module, and obtains the lane line based on the key points.

[0013] From the above description, it can be seen that by setting up an image processing module, a feature extraction module and a post-processing module, and adjusting the lane image training set to a preset size through the image processing module, then extracting image features through the feature extraction module, and finally extracting lane line prediction key points through the post-processing module and obtaining the lane lines, accurate recognition of the lane lines can be achieved.

[0014] Furthermore, the feature extraction module includes an hourglass network consisting of an attention unit, an encoder, a feature enhancement unit and a decoder; Extracting image features from the image output by the image processing module by the feature extraction module includes: The lane image training set is sequentially passed through the attention unit to extract the region of interest of the lane line; Extracting lane line features within the region of interest by the encoder; enhancing the extracted lane line features by the feature enhancement unit; The lane line features are output to the post-processing module through the decoder.

[0015] As can be seen from the above description, setting an attention unit at the front end of the encoder can extract the lane line ROI (region of interest) to reduce the amount of computation, and inserting a feature enhancement module between the encoder and decoder can enhance the extracted lane line features, thereby improving the lane line recognition accuracy.

[0016] Furthermore, the deep learning model also includes a feature distillation module; The training of the deep learning model using the lane image training set further includes: The image features output by the feature extraction module are enhanced by the feature distillation module, and the enhanced image features are output to the post-processing module.

[0017] From the above description, we can see that setting a feature distillation module in the deep learning model, that is, adding a feature distillation module during the training process, can optimize the expression of features, make them more informative, and improve the training effect.

[0018] Furthermore, each of the key points includes an offset and a feature similarity, and obtaining a lane line based on the key points includes: Select the bottom key point among all the key points in the image as the lane line starting point; Selecting the key point closest to the starting point as the target point, and if the target point and the starting point have the same offset and feature similarity less than a preset threshold, determining that the target point and the starting point are located on the same target lane line; Taking the target point as a new starting point, and selecting the key point closest to the new starting point as the next target point, thereby traversing all the key points; If the number of the key points on the target lane line reaches a preset number, the target lane line is used as a valid lane line.

[0019] From the above description, it can be seen that the target lane line is identified by traversing all key points in sequence, and only when the number of key points on the target lane line reaches the preset number, the corresponding target lane line is used as the valid lane line to ensure the accuracy of lane line identification.

[0020] Furthermore, after selecting the bottommost key point among all the key points in the image as the lane line starting point, the method further includes: Obtaining an offset direction according to an offset relationship between the key point and the image center; The key point located at the bottom of the offset direction among all the key points in the image is selected as the lane line starting point.

[0021] From the above description, it can be seen that since the lane line extends from one side of the image to the other side when the vehicle turns, the starting position of the lane line when turning can be effectively obtained by obtaining the offset direction based on the offset relationship between the key point and the image center.

[0022] Furthermore, taking the target lane line as a valid lane line includes: Perform third-order Bezier curve fitting on all the key points on the target lane line to obtain the effective lane line.

[0023] From the above description, it can be seen that by fitting a third-order Bezier curve to all key points on the target lane line, each key point can be accurately connected to form a complete lane line image.

[0024] Furthermore, the acquiring of the real-time image includes: Acquire the real-time image through a binocular camera; The identifying the lane lines in the real-time image by using the lane line recognition model and obtaining the depth coordinates of the lane lines includes: Performing left and right eye image distortion correction and stereo correction on the real-time image to obtain a corrected image; Performing stereo matching on the corrected image to obtain a visual difference of the corrected image; Converting the visual difference into a depth map, and performing hole filling and smoothing on the depth map; Obtain the depth coordinates corresponding to the lane line coordinates from the processed depth map.

[0025] From the above description, it can be seen that by sequentially performing distortion correction, stereo correction, and stereo matching on the real-time image to obtain the visual difference of the image, and then converting the visual difference into a depth map and performing hole filling and smoothing processing, accurate depth coordinates can be obtained.

[0026] Furthermore, obtaining the vehicle target operation data and the target coordinate point according to the depth coordinate and the current operation data includes: Get the preset distance threshold between the vehicle and the lane line; Calculating the vehicle's travel coordinates based on the distance threshold and the depth coordinates; Fitting the moving coordinates by a polynomial to obtain a vehicle moving coordinate trajectory line; and taking the end point of the vehicle moving coordinate trajectory line as the target coordinate point; Calculating the curvature of the vehicle's moving coordinate trajectory and obtaining the vehicle's running speed based on the curvature; The steering wheel adjustment angle is obtained according to the angle between the current vehicle direction and the tangent of the vehicle travel coordinate trajectory line.

[0027] From the above description, it can be seen that after the vehicle's moving coordinates are obtained by calculating the distance threshold and depth coordinates, polynomial fitting is performed on the moving coordinates, so that each independent coordinate is fitted into the vehicle's moving coordinate trajectory line, and the vehicle's running speed is adjusted based on the curvature of the vehicle's moving coordinate trajectory line, and the steering wheel angle is adjusted according to the angle between the current vehicle direction and the tangent of the vehicle's moving coordinate trajectory line, so that the vehicle can track and move according to the vehicle's moving coordinate trajectory line.

[0028] Another embodiment of the present invention provides a device for lane line tracking based on image recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the various steps in the above-mentioned method for lane line tracking based on image recognition.

[0029] The method and device for lane tracking based on image recognition provided by the present invention can be applied to scenarios of automatic or assisted driving of vehicles, and are described below through specific implementation methods: Example 1 Please refer to Figure 1 as well as Figure 2 A method for lane line tracking based on image recognition, comprising: S1. Obtain a lane image training set; for example, by collecting lane line images in different colors, different lighting, and different weather environments, and filtering out unqualified images such as duplicates and blurry images, the lane lines on the collected images are annotated using image annotation software to obtain the lane image training set.

[0030] S2. Build a deep learning model for lane line detection. The algorithm framework of the deep learning model includes an image processing module, a feature extraction module, and a post-processing module. The image processing module is used to adjust the image to a preset format. The feature extraction module is used to extract the image features output by the image processing module. The feature extraction module is composed of several hourglass network modules, each of which consists of a transform encoder and a decoder. The encoder is front-ended by an SE (Squeeze and Excitation) attention module, and the encoder and decoder are separated by a RESA (Recurrent Feature-Shift Aggregator) feature enhancement module. After building the model, set the model's learning rate, training batch, and loss function. The annotated images are divided into a training set and a validation set in a 3:1 ratio. The training set is used for model training, and the validation set is used for model verification.

[0031] S3. Train the deep learning model using the lane image training set to obtain a lane line recognition model. Specifically: S31. Adjust the lane image training set to a preset size through the normalization layer and convolution layer of the image processing module in sequence; for example, the image processing module adjusts the image to a size of 64*32 through a sequence of normalization layers and convolution layers. The image can be adjusted to different sizes according to different image processing requirements.

[0032] S32, extracting image features from the image output by the image processing module by the feature extraction module; S321, extracting the lane line region of interest through the lane image training set through the attention unit in sequence; S322: extracting lane line features within the region of interest using the encoder; S323, enhancing the extracted lane line features by the feature enhancement unit; S324: The decoder outputs the lane features to the post-processing module. Finally, the feature extraction module outputs confidence, offset, feature similarity, and distillation loss. Binary cross-entropy loss (BCE) is used as the loss function for confidence and offset, and mean squared error (MSE) is used as the loss function for feature similarity. A feature distillation module is also incorporated into the training process to optimize feature representation.

[0033] S33: Extracting lane line prediction key points based on the output of the feature extraction module through the post-processing module, and obtaining lane lines based on the key points. Specifically: S331. Select the lowest key point among all the key points in the image as the lane line starting point. Since lane lines in the acquired lane image extend from the bottom of the image to the center of the image during vehicle movement, the three lowest key points in the image can be selected as lane line starting points. However, when the vehicle turns, the lane lines will extend from one side of the image to the other. Therefore, it is necessary to further determine the offset direction based on the offset relationship between the key point and the image center. The key point located at the bottom of the offset direction among all the key points in the image is then selected as the lane line starting point. Specifically, depending on whether the key point at the center of the image is on the right or left side of the image, the three key points located on the leftmost or rightmost side of the image are selected as lane line starting points.

[0034] S332: Select the key point closest to the starting point as the target point. If the target point and the starting point have the same offset and feature similarity less than a preset threshold, determine that the target point and the starting point are located on the same target lane line. S333, taking the target point as a new starting point, and selecting the key point closest to the new starting point as the next target point, thereby traversing all the key points; that is, repeating the above process until all key points are traversed.

[0035] S334: If the number of key points on the target lane line reaches a preset number, the target lane line is considered a valid lane line, and a third-order Bezier curve is fitted to all the key points on the target lane line to obtain the valid lane line. For example, if a target lane line has more than 20% of the total number of key points, it is considered a valid lane line. A third-order Bezier curve is then fitted to the clustered key points, and the fitted lane line is used as the final output.

[0036] During the model training process, the lane line recognition model is trained using the divided lane line dataset. The model is optimized by continuously adjusting the hyperparameters to ultimately obtain the optimal recognition model.

[0037] S4. Acquire a real-time image, identify lane lines in the real-time image using the lane line recognition model, and obtain the depth coordinates of the lane lines. The vehicle's image module includes a binocular camera and an edge computing module. The binocular camera is used to acquire a real-time image of the vehicle's front, and the trained lane line recognition model is imported into the edge computing module. The model is used to identify whether lane lines exist in the image and obtain the specific coordinates of the lane lines. In an optional embodiment, after determining that a lane line exists in the real-time image, the three-dimensional coordinates of the lane line are obtained, and further, its depth coordinates are obtained. The specific image processing process is as follows: S41. Perform left and right image distortion correction and stereo correction on the real-time image to obtain a corrected image; for example, use OpenCV to perform left and right image distortion correction, and use Bouguet algorithm to perform stereo correction.

[0038] S42 , performing stereo matching on the corrected image to obtain the visual difference of the corrected image; for example, using the SGBM algorithm to perform stereo matching to obtain the visual difference of the image.

[0039] S43, converting the visual difference into a depth map, and performing hole filling and smoothing processing on the depth map; S44: Obtain depth coordinates corresponding to lane line coordinates from the processed depth map.

[0040] S5. Obtain the current operating data of the vehicle, and obtain the target operating data and target coordinate point of the vehicle based on the depth coordinates and the current operating data. The vehicle's maneuvering control equipment includes a brake sensor controller, an accelerator sensor controller, a steering wheel sensor controller, a speed sensor, a gear sensor, an engine speed sensor, and an IMU (Inertial Measurement Unit), etc., which are respectively used to monitor and control the status of the corresponding devices. After detecting a lane line in front of the vehicle, the vehicle's maneuvering control equipment will collect information from each vehicle sensor and transmit it to the edge computing module. After receiving the depth coordinate of the current lane line in step S4, the edge computing module performs the following steps: S51. Obtain a preset distance threshold between the vehicle and the lane line. For example, if the preset distance threshold is 0.5 meters, then the distance threshold is 0.5 meters.

[0041] S52. Calculate the vehicle's travel coordinates based on the distance threshold and the depth coordinates; for example, if the lateral distance is represented by x and the longitudinal distance is represented by y, the vehicle's travel coordinates calculated based on the two are expressed as x-0.5 and y.

[0042] S53, fitting the travel coordinates by a polynomial to obtain a vehicle travel coordinate trajectory line; and taking the end point of the vehicle travel coordinate trajectory line as the target coordinate point; S54. Calculate the curvature of the vehicle's moving coordinate trajectory, and obtain the vehicle's running speed based on the curvature; wherein, the greater the curvature, the more curved the trajectory, and the smaller the vehicle's running speed should be.

[0043] S55. Obtain a steering wheel adjustment angle according to an angle between the current vehicle direction and a tangent line of the vehicle's moving coordinate trajectory.

[0044] S6. Control the vehicle to travel to the target coordinate point according to the vehicle target operation data. That is, adjust the throttle and brake according to the current vehicle speed and the predetermined vehicle speed under the set curvature. If the vehicle speed needs to be increased, increase the throttle. If the vehicle speed needs to be reduced, brake. Adjust the steering wheel according to the steering wheel adjustment angle. The specific control method is as follows: When the vehicle's maneuvering control device receives the vehicle operation parameters sent by the edge computing module, it first sends a signal to the steering wheel sensor controller and the throttle sensor controller, and then controls the steering wheel rotation angle and the size of the throttle. It compares the vehicle operation parameters and the target parameters based on the IMU and speed sensor to perform real-time feedback adjustment, so as to reach the specified coordinate point. Then repeat steps S4-S6 to obtain the next operation parameters of the vehicle, so that the vehicle can track along the lane line.

[0045] Example 2 Please refer to Figure 3 A device for lane line tracking based on image recognition includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the method for lane line tracking based on image recognition as described in Example 1 is implemented.

[0046] In summary, the method and device for lane line tracking based on image recognition provided by the present invention train a deep learning model based on a lane image training set to obtain a lane line recognition model, so that the lane line recognition model has strong generalization and stability. After obtaining the depth coordinates of the lane line through an image processing method, the vehicle's current operation data obtained by the motorized control device is combined to adjust the vehicle's operation data at the next moment, that is, the detected lane line information can be calculated and converted into instructions required for vehicle operation in real time, thereby quickly and effectively realizing the vehicle's lane line tracking function.

[0047] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for lane line tracking based on image recognition, characterized in that: include: Get the lane image training set; Build a deep learning model for lane detection; Training the deep learning model using the lane image training set to obtain a lane line recognition model; Acquire a real-time image, identify lane lines in the real-time image using the lane line recognition model, and obtain depth coordinates of the lane lines; Acquire the current running data of the vehicle, and obtain the target running data and target coordinate point of the vehicle according to the depth coordinate and the current running data; The vehicle is controlled to travel to the target coordinate point according to the vehicle target operation data.

2. The method for lane line tracking based on image recognition according to claim 1, characterized in that: The deep learning model includes an image processing module, a feature extraction module and a post-processing module; The training of the deep learning model using the lane image training set includes: Adjusting the lane image training set to a preset size through the normalization layer and the convolution layer of the image processing module in sequence; Extracting image features from the image output by the image processing module by the feature extraction module; The post-processing module extracts lane line prediction key points based on the output results of the feature extraction module, and obtains the lane line based on the key points.

3. The method for lane line tracking based on image recognition according to claim 2, characterized in that: The feature extraction module includes an hourglass network consisting of an attention unit, an encoder, a feature enhancement unit and a decoder; Extracting image features from the image output by the image processing module by the feature extraction module includes: The lane image training set is sequentially passed through the attention unit to extract the region of interest of the lane line; Extracting lane line features within the region of interest by the encoder; enhancing the extracted lane line features by the feature enhancement unit; The lane line features are output to the post-processing module through the decoder.

4. The method for lane line tracking based on image recognition according to claim 2 or 3, characterized in that: The deep learning model also includes a feature distillation module; The training of the deep learning model using the lane image training set further includes: The image features output by the feature extraction module are enhanced by the feature distillation module, and the enhanced image features are output to the post-processing module.

5. The method for lane line tracking based on image recognition according to claim 2, characterized in that: Each of the key points includes an offset and a feature similarity, and obtaining a lane line based on the key points includes: Select the bottom key point among all the key points in the image as the lane line starting point; Selecting the key point closest to the starting point as the target point, and if the target point and the starting point have the same offset and feature similarity less than a preset threshold, determining that the target point and the starting point are located on the same target lane line; Taking the target point as a new starting point, and selecting the key point closest to the new starting point as the next target point, thereby traversing all the key points; If the number of the key points on the target lane line reaches a preset number, the target lane line is used as a valid lane line.

6. The method for lane line tracking based on image recognition according to claim 5, characterized in that: After selecting the bottom key point among all the key points in the image as the lane line starting point, the method further includes: Obtaining an offset direction according to an offset relationship between the key point and the image center; The key point located at the bottom of the offset direction among all the key points in the image is selected as the lane line starting point.

7. The method for lane line tracking based on image recognition according to claim 5, characterized in that: The taking the target lane line as the valid lane line comprises: Perform third-order Bezier curve fitting on all the key points on the target lane line to obtain the effective lane line.

8. The method for lane line tracking based on image recognition according to claim 1, characterized in that: The acquiring of the real-time image comprises: Acquire the real-time image through a binocular camera; The identifying the lane lines in the real-time image by using the lane line recognition model and obtaining the depth coordinates of the lane lines includes: Performing left and right eye image distortion correction and stereo correction on the real-time image to obtain a corrected image; Performing stereo matching on the corrected image to obtain a visual difference of the corrected image; Converting the visual difference into a depth map, and performing hole filling and smoothing on the depth map; Obtain the depth coordinates corresponding to the lane line coordinates from the processed depth map.

9. The method for lane line tracking based on image recognition according to claim 1, characterized in that: The obtaining of the vehicle target operation data and the target coordinate point according to the depth coordinate and the current operation data includes: Get the preset distance threshold between the vehicle and the lane line; Calculating the vehicle's travel coordinates based on the distance threshold and the depth coordinates; Fitting the moving coordinates by a polynomial to obtain a vehicle moving coordinate trajectory line; and taking the end point of the vehicle moving coordinate trajectory line as the target coordinate point; Calculating the curvature of the vehicle's moving coordinate trajectory and obtaining the vehicle's running speed based on the curvature; The steering wheel adjustment angle is obtained according to the angle between the current vehicle direction and the tangent of the vehicle travel coordinate trajectory line.

10. A device for lane line tracking based on image recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the method for lane line tracking based on image recognition as described in any one of claims 1 to 9 is implemented.

Images