A real-time automatic extraction method of virtual lane line contours in UAV scenes

Abstract

A method for automatically extracting the outlines of virtual lane lines in real-time for drone scenarios begins by using a deep learning lane detection network to represent a number of pixels at the center of the lane line. Based on colorimetric feature differences, a multi-layer perceptron (MLP) machine learning method is then used to classify each lane line as real or virtual. Based on a single virtual lane line, an LSD line segment detection algorithm is used to extract the endpoints of the virtual lane line region within the image. Slope screening is then used to determine the endpoints of the virtual lane line segment. Finally, the relationship between the point and the line and the y-coordinate are determined to accurately locate the virtual lane line segment to which the endpoint belongs. This method enables real-time automatic extraction of virtual lane line outlines during drone cruising.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video image processing, relates to unmanned aerial vehicle (UAV) monitoring, and is a method for automatically extracting the real-time virtual lane line contour of a UAV scene. Background Art

[0002] Drones, with their high maneuverability, flexibility, and wide field of view, are becoming a new and effective method for intelligent road inspections. Unlike fixed road cameras, drone patrols are characterized by their dynamic nature, which also presents certain challenges for intelligent inspections.

[0003] Intelligent road inspection features such as intelligent speed measurement and the intelligent capture of abnormal parking behavior rely on camera calibration to determine the actual physical distance traveled by a vehicle, thereby determining its behavior. While fixed camera footage can be easily calibrated by selecting fixed reference objects or manually extracting the coordinates of virtual lane line endpoints, drone road patrol lacks fixed reference objects. Therefore, camera calibration typically relies on known lane line specifications combined with precise virtual lane line endpoint coordinates. However, drone road patrol often involves dynamic flight, with constantly changing lane lines. This requires both high accuracy and high speed in extracting virtual lane line contours. While existing line segment detection methods, such as Hough transform line detection and Canny edge detection, fail to achieve both high accuracy and high speed. Furthermore, while relatively fast lane line detection methods exist in the autonomous driving field, such as SwiftLane and LaneFormer, these methods are limited to lane line detection and cannot achieve real-time, full-process automation for lane line detection, virtual and real lane line classification, and virtual lane line contour extraction. Furthermore, the scenarios differ significantly from those used by drone road patrol, making them inapplicable. Summary of the Invention

[0004] The problem addressed by this invention is that, given the fixed virtual lane lines in a camera's image, camera calibration requires only the pre-determined coordinates of the virtual lane line segments' endpoints. However, as drones cruise along a road, the virtual lane lines constantly change. Existing technologies are unable to fully automate the entire process of virtual lane line detection, virtual-to-real line classification, and virtual lane line outline extraction while maintaining both accuracy and speed.

[0005] The technical solution of the present invention is: a method for automatically extracting the outline of virtual lane lines in real time in a drone scene, which identifies virtual lane lines in real time and extracts the outline of virtual lane lines in a drone cruising scene. The drone cruises along the road in the same direction or in the opposite direction as the vehicle and takes photos, including the following steps:

[0006] Step 1: Use the OpenCV open source tool to read the road video stream shot by the drone in real time, and perform size normalization and standardization preprocessing on the read video frame images;

[0007] Step 2: Train a deep learning network as a lane detection network, use the lane detection network to perform lane detection, and obtain the coordinate information of m pixels representing the center of each lane line;

[0008] Step 3: Based on the coordinate information and color feature differences of the lane line center pixel points, a multi-layer perceptron (MLP) network is used to classify each lane line as real or virtual. The resulting virtual lane line is fitted with the least squares method to obtain the linear equation of the virtual lane line. The resulting line is extended to the left and right by k pixels, and the image of the area where the line is located is intercepted.

[0009] Step 4: Grayscale the image captured in Step 3 and extract the endpoints of the line segments based on the LSD line segment detection algorithm. The coordinates of the endpoints of several line segments are obtained. The line segments that do not meet the condition that the absolute values ​​of the slopes of the edges on both sides of the virtual lane line are greater than 1 are filtered out.

[0010] Step 5: Based on the point-line relationship between the endpoints of the virtual lane line segment obtained in Step 4 and the equation of the center line of the virtual lane line obtained in Step 3, distinguish the endpoints on the left and right sides of the virtual lane line segment. Locate the virtual lane line segment to which the endpoints belong based on the y-coordinate information of the pixel point, and obtain the endpoints of the virtual lane line segment for marking the virtual lane line contour.

[0011] The present invention realizes the real-time automatic extraction of virtual lane line contours during drone cruising. In view of the fact that lane lines are constantly changing during drone cruising, the existing technology cannot realize the full process automation of virtual lane line detection, virtual and real line classification, and virtual lane line contour extraction while taking into account both extraction accuracy and extraction speed. The present invention designs an MLP machine learning method that uses the difference between the color intensity value of the center pixel of the lane line and the average color intensity value of all the center pixels of the lane line as the feature value, thereby realizing fast and accurate lane line virtual and real line classification. Based on the virtual and real classification results of the lane line, the present invention further designs a virtual lane line segment endpoint precise positioning algorithm that integrates LSD line segment detection and multiple decision-making judgment conditions for a single virtual lane line, thereby realizing endpoint positioning capabilities with high processing speed and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of a method for automatically extracting real-time virtual lane line contours in a drone scene according to the present invention.

[0013] Figure 2 This is a diagram showing the implementation effect of the present invention in automatically extracting virtual lane line contours in a real drone scene. DETAILED DESCRIPTION

[0014] The following is a detailed description of the present invention with reference to the accompanying drawings. The present invention requires that the drone cruise along the road, in the same direction or in the opposite direction as the vehicle, and when collecting road videos, the lane lines in the image should be as vertical as possible. The specific solution process is as follows: Figure 1 shown.

[0015] Step 1: Use the OpenCV open source tool to read the real-time road video stream shot by the drone, and perform size normalization and standardization preprocessing on the read image.

[0016] Step 2: Use a lane detection network to detect lanes, obtaining the coordinates of m pixels representing the center of each lane. This lane detection network includes, but is not limited to, deep learning methods such as UltraLane and PolyLaneNet. It must ensure ultra-fast detection speed and high accuracy. The detection speed and accuracy must be no less than 100 FPS and 90%, respectively, on a graphics card with a computing power of 6.1.

[0017] Step 2.1: Based on the lane line annotated dataset of video footage of road scenes shot by drones, the lane line detection network is trained in advance to obtain the corresponding weight model;

[0018] Step 2.2: Based on the training weight model obtained in Step 2.1, the real-time drone-captured images are input into the lane detection network to obtain the coordinate information of several pixel points representing the center of each lane line.

[0019] Step 3: Based on the coordinate information of the center pixel points of the lane lines and the difference in color characteristics, each lane line is classified as virtual or real through a multi-layer perceptron (MLP) network. The obtained virtual lane lines are then fitted with the least squares method to obtain the linear equation of the virtual lane lines. The obtained straight line is expanded by k pixels to the left and right, for example, by 10 pixels to the left and right to obtain the image area. The image of the area where the straight line is located is intercepted to obtain the image of the virtual lane line. The present invention is a real-time processing solution, and the extraction and determination of the virtual lane line contour is only required for camera calibration. Therefore, processing one virtual lane line can meet the requirements. If multiple virtual lane line contours need to be extracted, the pixel coordinates at the center of each virtual lane line are linearly fitted and the subsequent operations are repeated.

[0020] Step 3.1: To highlight the difference in color characteristics between virtual lane lines and real lane lines, for each lane line, the set of differences between the color intensity value of each center pixel of the lane line and the average color intensity of all center pixels of the lane line is used as the feature value, as shown in the following formula, where x i is the color intensity value of the pixel point, i ranges from 1 to m, representing the m pixels forming the center of the lane line. In this embodiment, m=40, y iis the difference of the center pixel of a single lane line. The difference set Y of the center pixel points of 40 lane lines constitutes the characteristic value of this lane line;

[0021]

[0022] Y=[y1,y2,……,y 40 ]

[0023] Step 3.2: Create a machine learning classification dataset based on drone-captured road videos with known lane line information. Pre-train the MLP lane line classification network to obtain the corresponding weight model.

[0024] Step 3.3: Based on the lane line center pixel coordinate information obtained in Step 2, the required feature information can be obtained according to the formula in Step 3.1. Input it into the MLP lane line virtual-real classification network to obtain the virtual and real line information of each lane line;

[0025] Step 3.4: Select a dotted line in the image, such as Figure 2 For the first dotted line on the left in the illustrated embodiment, the least squares fitting is performed on all 40 pixel points representing the center of the lane line to obtain the straight line equation of the lane line and intercept the image of the area where the line is located.

[0026] Step 4: Grayscale the image captured in Step 3 and extract line segment contours based on the LSD line segment detection algorithm. This will obtain the endpoint coordinates of several line segments. Based on the condition that the absolute values ​​of the slopes on both sides of the virtual lane line are greater than 1, line segments that do not meet the condition are filtered out.

[0027] Step 4.1: Grayscale the image of the area where the dotted line intercepted in Step 3 is located;

[0028] Step 4.2: Perform LSD line segment detection on the grayscale image obtained in Step 4.1 to obtain the endpoint coordinate information of several line segments;

[0029] Step 4.3: Using the line segment endpoint coordinates obtained in Step 4.2, calculate the absolute value of each line segment's slope. Using the absolute value of 1 as the threshold, filter out line segments with slopes less than 1. Remove possible interfering line segments, such as road shadows. The remaining segments are virtual lane line segments.

[0030] Step 5: Based on the point-line relationship between the virtual lane line endpoints obtained in Step 4 and the virtual lane line centerline equation obtained in Step 3, the left / right endpoints of the virtual lane line segment can be distinguished. Based on the y-coordinate information of the pixel point, the virtual lane line segment to which the endpoint belongs can be further located. The virtual lane line segment that meets the requirements is selected for endpoint annotation and output for display.

[0031] Step 5.1: Based on the equation of the imaginary lane line center fitting obtained in Step 3 and the coordinates of the imaginary lane line segment endpoints obtained in Step 4, determine whether the imaginary lane line segment endpoint is the endpoint of the imaginary lane line segment to the left of the center line or the endpoint of the imaginary lane line segment to the right of the center line according to the relationship between the point and the line;

[0032] Step 5.2: Based on the y-coordinate information of the endpoints of the virtual lane line segments, locate the dashed line segments to which the endpoints belong, from highest to lowest. The four endpoints of the two segments with the highest y-coordinate information belong to the first dashed line segment, the four endpoints of the two segments with the second highest y-coordinate information belong to the second dashed line segment, and so on. During the endpoint location process, the outline of the first dashed line segment may be incomplete. This can be filtered out by setting the y-coordinate information of the endpoints of the first dashed line segment to be lower than the image height;

[0033] Step 5.3: Use OpenCV tools to mark endpoints, such as Figure 2 As shown, the endpoints of the first virtual lane segment are L1.1, R1.1, R1.0, and L1.0, clockwise from the upper left to the lower right. The endpoints of the second virtual lane segment are L2.1, R2.1, R2.0, and L2.0, clockwise from the upper left to the lower right. The output image also returns the coordinates of the two virtual lane segment endpoints. The marker display is primarily for visually visualizing the results of the virtual lane line outline extraction.

[0034] In the present invention, based on the point-line relationship, Step 5.1 obtains the left and right endpoints of all line segments. However, since the UAV is flying dynamically, there may be a situation where there is only a part of the dotted segment at the bottom of the picture, that is, the outline of the dotted segment at the bottom of the picture may be incomplete, that is, the virtual lane line rectangle should theoretically have four endpoints, but in reality there are only two. Step 5.2 is mainly to eliminate this situation, and the endpoints are marked and output for display in Step 5.3. The solution of the present invention is not limited to the display of two virtual lane line segments. Step 5 can locate the endpoint information of all dotted line segments in the image, and any number of dotted line segments can be selected for output and display. Selecting multiple dotted line segments to extract the outline is conducive to calculating the true distance according to the lane line specifications and combining the precise outline coordinate information of the virtual lane line to achieve camera calibration.

[0035] Finally, based on the principles of computer imaging, the perspective transformation matrix can be solved to achieve camera calibration by combining the virtual lane line distance information in the national standard and the virtual lane line segment endpoint coordinate information obtained in Step 5.3. This can be further combined with vehicle detection to achieve functions such as vehicle speed calculation and abnormal parking monitoring.

Claims

1. A real-time automatic extraction method of virtual lane line contours in UAV scenes, characterized in that In a drone cruising scenario, the system recognizes virtual lane lines in real time and extracts their outlines. The drone cruises along the road in the same direction or in the opposite direction of the vehicle, taking photos. The system includes the following steps: Step 1: Use the OpenCV open source tool to read the road video stream shot by the drone in real time, and perform size normalization and standardization preprocessing on the read video frame images; Step 2: Train a deep learning network as a lane detection network, use the lane detection network to perform lane detection, and obtain the coordinate information of m pixels representing the center of each lane line; Step 3: Based on the coordinate information and color feature differences of the lane line center pixel points, a multi-layer perceptron (MLP) network is used to classify each lane line as real or virtual. The resulting virtual lane line is then fitted using the least squares method to obtain the line equation. The resulting line is then extended to the left and right by k pixels, and the image of the area where the line is located is intercepted. Step 4: Grayscale the image captured in Step 3 and extract the endpoints of the line segments based on the LSD line segment detection algorithm. The coordinates of the endpoints of several line segments are obtained. The line segments that do not meet the condition that the absolute values ​​of the slopes of the edges on both sides of the virtual lane line are greater than 1 are filtered out. Step 5: Based on the point-line relationship between the endpoints of the virtual lane line segment obtained in Step 4 and the equation of the center line of the virtual lane line obtained in Step 3, distinguish the endpoints on the left and right sides of the virtual lane line segment. Locate the virtual lane line segment to which the endpoint belongs based on the y-coordinate information of the endpoint pixel point, and obtain the endpoint of the virtual lane line segment for marking the virtual lane line contour.

2. The method for automatically extracting the real-time virtual lane line contour of a drone scene according to claim 1 is characterized by: Step 2 is as follows: Step 2.1: Based on a dataset of lane markings in drone-captured video footage of road scenes, a deep learning network is trained to obtain corresponding weights, i.e., the lane detection network. The deep learning network includes UltraLane and PolyLaneNet. The detection speed and accuracy are required to be no less than 100 FPS and 90%, respectively, on a graphics card with a computing power of 6.

1. Step 2.2: Based on the lane line detection network obtained in Step 2.1, input the real-time collected drone image to obtain the coordinate information of m pixel points representing the center of each lane line.

3. The method for automatically extracting real-time virtual lane line contours in a drone scene according to claim 1, wherein Step 3 specifically comprises: Step 3.1: To highlight the difference in color characteristics between virtual lane lines and real lane lines, for each lane line, the set of differences between the color intensity value of each center pixel of the lane line and the average color intensity of all center pixels of the lane line is used as the feature value, as shown in the following formula, where x i is the pixel color intensity value, i ranges from 1 to m, representing the m pixels that make up the center of the lane line, y i is the difference of the center pixel of a single lane line, and the difference set Y of the center pixel points of m lane lines constitutes the characteristic value of this lane line; Y=[y1,y2,……,y m ] Step 3.2: Create a machine learning classification dataset based on drone-generated road videos with known lane marking information. Pre-train a multi-layer perceptron (MLP) network to obtain the corresponding network weight model, i.e., the lane marking classification network. Step 3.3: Based on the lane line center pixel coordinate information obtained in Step 2, the characteristic value is obtained according to the formula in Step 3.

1. The characteristic value is input into the multi-layer perceptron (MLP) network to classify the lane line as virtual or real, and obtain the virtual lane line. Step 3.4: Select a virtual lane line, perform least squares fitting on all pixel points representing the center of the lane line, obtain the straight line equation of the center of the virtual lane line, and intercept the image of the area where the line is located.

4. The method for automatically extracting real-time virtual lane line contours in a drone scene according to claim 1, wherein Step 4 specifically comprises: Step 4.1: Convert the image of the area where the virtual lane line is located, which was intercepted in Step 3, into grayscale. Step 4.2: Perform LSD line segment detection on the grayscale image obtained in Step 4.1 to obtain the endpoint coordinate information of several line segments; Step 4.3: Using the line segment endpoint coordinate information obtained in Step 4.2, calculate the absolute value of the slope of each line segment. Using the absolute value of 1 as the threshold, filter out line segments with a slope less than 1 to remove interference. The remaining segments are the virtual lane line segments.

5. The method for automatically extracting the real-time virtual lane line contour of a drone scene according to claim 1 is characterized by: Step 5 is as follows: Step 5.1: Based on the equation of the imaginary lane line center fitting obtained in Step 3 and the coordinates of the imaginary lane line segment endpoints obtained in Step 4, determine whether the imaginary lane line segment endpoint is the endpoint of the imaginary lane line segment to the left of the center line or the endpoint of the imaginary lane line segment to the right of the center line according to the relationship between the point and the line; Step 5.2: Based on the y-coordinate information of the endpoints of the virtual lane line segments, locate the virtual line segments to which the endpoints belong from high to low. The four endpoints of the two line segments with the highest y-coordinate information belong to the first virtual line segment, the four endpoints of the two line segments with the second highest y-coordinate information belong to the second virtual line segment, and so on.

6. The method for automatically extracting the real-time virtual lane line contour of a drone scene according to claim 1 or 5, characterized in that The endpoints of the specified virtual lane line segments are marked and output. With the help of OpenCV tools, the endpoints of the selected virtual line segments are marked, the screen is output and the coordinate results of the endpoints of the virtual line segments are returned.

Images