Road structure information extraction method based on real-time video data

Abstract

The invention relates to a road structure information extraction method based on real-time video data. The method comprises the following steps: firstly, acquiring real-time video data, and extracting a road background picture; semantic segmentation is carried out on the road background picture, road elements are extracted, and a lane geometric structure is obtained by adopting a Hough straight line detection algorithm; extracting a vehicle driving track based on the lane geometric structure, calculating a vector angle based on the vehicle driving track, and performing track fitting by adopting Bezier curve fitting to obtain a lane driving function; a Kmeans clustering method is adopted to classify the terminal points of the vehicle driving track, and intersection driving area division is determined; and drawing a road structure map based on the lane geometric structure, the lane driving function and the intersection driving area division. The image and track information is effectively fused, the accuracy and frequency of road structure information extraction are greatly improved, and the ability to understand complex traffic scenes is improved.

Description

Technical Field

[0001] The present application relates to the technical field of traffic monitoring, and in particular to a method for extracting road structure information based on real-time video data. Background Art

[0002] Roads are the backbone of modern transportation infrastructure. Research on road information extraction holds significant practical value and significance in areas such as traffic navigation, autonomous driving, route planning, traffic management, and geographic information system updates. Extracting refined road structural information involves analyzing and extracting specific structural information from road networks, including road geometry, topological relationships, and attribute characteristics, to accurately describe the road environment and traffic flow. However, road construction is a frequent activity, and road updates are rapid. Therefore, quickly and efficiently extracting refined structural information from traffic roads not only aids route planning, vehicle deviation warnings, and traffic flow analysis, but also provides a reference for precise navigation.

[0003] Existing research focuses primarily on extracting road network centerlines and identifying intersection locations. The main approaches employed can be categorized into two types: one based on remote sensing imagery for road structure extraction, and the other based on vehicle GPS trajectories. Remote sensing images offer a wide range of data sources and relatively low acquisition costs. Furthermore, with improvements in image quality and resolution, they are widely used for road extraction. However, remote sensing images are often used for large-scale, global scenes, resulting in a crude road structure extraction that is limited to extracting road centerlines and lacks a detailed description of the road structure. Furthermore, these methods cannot achieve real-time updates. Furthermore, with the rapid development of smart car terminals, autonomous driving, and 5G communication technologies, vehicle GPS trajectory data has become more accessible, offering higher accuracy and richer feature information. Compared to other data types, trajectory data contains richer traffic dynamics information, is relatively inexpensive to acquire, and has become a primary data source for road structure extraction. However, trajectory data has certain limitations in processing road structure appearance and geometry.

[0004] Therefore, in the relevant technology, there is an urgent need for a method that can improve the accuracy and efficiency of refined extraction of road structure and lane functionality analysis. Summary of the Invention

[0005] Based on this, it is necessary to provide a road structure information extraction method based on real-time video data that can improve the accuracy and efficiency of refined road structure extraction and lane functional analysis to address the above technical problems.

[0006] In a first aspect, the present application provides a method for extracting road structure information based on real-time video data. The method comprises:

[0007] Acquire real-time video data and extract road background images;

[0008] Performing semantic segmentation on the road background image to extract road elements and using the Hough line detection algorithm to obtain lane geometry;

[0009] Extracting a vehicle driving trajectory based on the lane geometry, calculating a vector angle based on the vehicle driving trajectory, and performing trajectory fitting using Bezier curve fitting to obtain a lane driving function;

[0010] Using the Kmeans clustering method to classify the end points of the vehicle's driving trajectory and determine the intersection driving area division;

[0011] A road structure map is drawn based on the lane geometry, lane driving function, and intersection driving area division.

[0012] Optionally, in one embodiment of the present application, performing semantic segmentation on the road background image, extracting road elements, and using the Hough line detection algorithm to obtain lane geometry includes:

[0013] Using the BiSeNetV2 network model to perform semantic segmentation on the road background image to obtain a set of pixel points corresponding to traffic objects;

[0014] Clustering is performed based on the pixel point set of the corresponding traffic object using the Hough line detection algorithm to determine the initial lane line structure;

[0015] Based on the initial lane line structure, the intersection points of the intersecting straight lines are determined to form a closed figure and determine the lane geometry.

[0016] Optionally, in one embodiment of the present application, the performing semantic segmentation on the road background image, extracting road elements, and obtaining lane geometry using the Hough line detection algorithm further includes:

[0017] Performing least squares straight line fitting based on the vehicle's driving trajectory to determine the lane centerline;

[0018] Missing lane lines are filled in based on the lane centerline to determine a closed lane geometry.

[0019] Optionally, in one embodiment of the present application, extracting the vehicle driving trajectory based on the lane geometry structure includes:

[0020] The YOLOV7 target detection model is used to detect and track vehicles, and the initial vehicle trajectory is determined based on the vehicle target detection frame;

[0021] The DBSCAN density clustering method is used to clean the initial vehicle driving trajectory to obtain the target vehicle driving trajectory.

[0022] Optionally, in one embodiment of the present application, the calculating of the vector angle based on the vehicle driving trajectory and the use of Bezier curve fitting to perform trajectory fitting to obtain the lane driving function include:

[0023] The starting point, end point and midpoint of each vehicle's trajectory are selected to form two vectors, and the angle between the vectors is calculated to determine the vehicle's direction of travel;

[0024] The vehicle driving trajectories corresponding to the same vehicle driving direction are fitted using Bezier curve fitting to determine the lane driving function.

[0025] Optionally, in one embodiment of the present application, before calculating the vector angle based on the vehicle driving trajectory and performing trajectory fitting using Bezier curve fitting to obtain the lane driving function, the method includes:

[0026] The vehicle driving trajectory in the waiting area is filtered based on time and vehicle status, wherein the vehicle status is determined based on the ratio of the actual change distance of the center point of the vehicle detection frame to the length of the vehicle detection frame.

[0027] Optionally, in one embodiment of the present application, the calculating of the vector angle based on the vehicle driving trajectory and performing trajectory fitting using Bezier curve fitting to obtain the lane driving function further includes:

[0028] For the vehicle driving trajectory in the waiting area, the centroid is calculated based on the coordinates of the trajectory end point, and the lane function of the waiting area is determined based on the centroid.

[0029] Optionally, in one embodiment of the present application, the method further includes:

[0030] The vehicle driving direction is determined based on the road structure map.

[0031] In a second aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program and the processor executes the steps of the methods described in the above embodiments.

[0032] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in each of the above embodiments.

[0033] The above-mentioned method for extracting road structure information based on real-time video data first obtains real-time video data and extracts a road background image; performs semantic segmentation on the road background image to extract road elements, and uses the Hough line detection algorithm to obtain the lane geometry; extracts the vehicle driving trajectory based on the lane geometry, calculates the vector angle based on the vehicle driving trajectory, and uses Bezier curve fitting to fit the trajectory to obtain the lane driving function; uses the Kmeans clustering method to classify the end points of the vehicle driving trajectory and determine the intersection driving area division; and draws a road structure map based on the lane geometry, lane driving function, and intersection driving area division. In other words, the geometric structure of the lane is extracted by using image-based semantic segmentation technology, and the Hough line detection algorithm is used to achieve construction from point to line and then to surface. Secondly, the lane function attributes are extracted through vehicle trajectory analysis, and the target detection model and tracking algorithm are combined with density clustering to accurately identify the driving function of each lane and waiting area. Finally, through cluster analysis of the vehicle trajectory endpoints, the intersection driving area is divided, and the vehicle's turning direction is determined in real time, effectively integrating image and trajectory information, greatly improving the accuracy and frequency of road structure information extraction, and enhancing the ability to understand complex traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A diagram illustrating an application environment of a method for extracting road structure information based on real-time video data in one embodiment;

[0035] Figure 2 1 is a flow chart of a method for extracting road structure information based on real-time video data in one embodiment;

[0036] Figure 3 Schematic diagram of semantic segmentation effect in one embodiment;

[0037] Figure 4 Schematic diagram of an initial lane line structure in one embodiment;

[0038] Figure 5 A schematic diagram of a lane geometry structure in one embodiment;

[0039] Figure 6 is a schematic diagram of an initial vehicle driving trajectory in one embodiment;

[0040] Figure 7 A schematic diagram of the trajectory starting point selection and the target vehicle driving trajectory in one embodiment;

[0041] Figure 8 A schematic diagram of vector angles in different driving directions in one embodiment;

[0042] Figure 9A schematic diagram of control point selection in one embodiment;

[0043] Figure 10 A schematic diagram of the results of Bezier curve fitting in one embodiment;

[0044] Figure 11 This is a schematic diagram of a road structure map result in one embodiment;

[0045] Figure 12 A schematic diagram of selecting an endpoint judgment point in one embodiment;

[0046] Figure 13 A schematic diagram of determining whether a vehicle trajectory point is in different color blocks in one embodiment;

[0047] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] The road structure information extraction method based on real-time video data provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal communicates with the server through the network. The data storage system can store data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0050] In one embodiment, Figure 2 As shown in the figure, a road structure information extraction method based on real-time video data is provided. Figure 1 The following steps are used as an example to illustrate the server in the example:

[0051] S201: Acquire real-time video data and extract road background images.

[0052] In the embodiment of the present application, first, a surveillance camera installed at a fixed position on the road continuously collects real-time video data of a fixed scene. Since vehicles and pedestrians moving on the road will affect the subsequent image segmentation to obtain road structure information, the averaged background extraction method is used to extract the road background image. The formula is as follows:

[0053]

[0054] Among them, I(x,y) represents the image pixel point, background and foreground represent the extracted road background image and road foreground image respectively.

[0055] S203: Perform semantic segmentation on the road background image, extract road elements, and use the Hough line detection algorithm to obtain the lane geometry structure.

[0056] In the embodiment of the present application, the BiSeNetV2 network model is used to perform semantic segmentation on the road background image, extract road elements such as lane lines and crosswalks, and then the Hough line detection algorithm is used to obtain the lane geometry structure.

[0057] Specifically, in one embodiment of the present application, performing semantic segmentation on the road background image, extracting road elements, and using the Hough line detection algorithm to obtain lane geometry includes:

[0058] S301: Use the BiSeNetV2 network model to perform semantic segmentation on the road background image to obtain a set of pixel points corresponding to traffic objects.

[0059] S303: Clustering is performed based on the pixel point set of the corresponding traffic object using the Hough line detection algorithm to determine an initial lane line structure.

[0060] S305: Determine the intersection of the intersecting straight lines based on the initial lane line structure, form a closed figure, and determine the lane geometry structure.

[0061] In one embodiment of the present application, the BiSeNetV2 network model has always performed very well in semantic segmentation. BiSeNetV2 mainly consists of three parts: 1) a wider and shallower detail branch that can capture high-resolution features to extract low-level spatial detail information; 2) a narrower and deeper semantic branch that can capture low-resolution features to extract high-level semantic information; 3) a bilateral guided aggregation layer that can achieve the fusion of dual-branch features. Before application, the StochasticGradient Descent algorithm was used to optimize the model training, and the momentum parameter was set to 0.9 and the weight decay coefficient was set to 5e-4. The learning rate was dynamically adjusted using an exponential transformation learning strategy, where the initial learning rate was 5e-3 and the power was 0.9. The learning rate calculation formula for the current number of iterations is as follows, and the training batch size of the data set is 8.

[0062]

[0063] Among them, lt_start is the initial learning rate (5×10-3), iter is the number of iterations currently completed, and iters max is the upper limit of the total number of iterations, and power is the decay exponent (0.9).

[0064] The training dataset was preprocessed with normalization and standardization. Data augmentation included random horizontal flipping, random scaling, and random cropping during training. The scale of random scaling was [0.25, 2.0]. Random cropping was performed to a fixed resolution of 512×1024 for training.

[0065] By using the BiSeNetV2 network model to perform semantic segmentation on the road background image, the pixel point set of the corresponding traffic object is obtained, such as Figure 3 As shown in the figure, the result of image semantic segmentation distinguishes pixels of different categories and marks them with different colors. Based on this, the pixels of corresponding traffic objects, such as crosswalks and lane lines, can be extracted according to specific colors to obtain the corresponding point sets. Afterwards, the Hough line detection algorithm is used for clustering, and the lines are cumulatively represented in the parameter space to determine the initial lane line structure. Specifically, the Hough detected lines are clustered using the starting and end points obtained by the Hough line detection, thereby obtaining a cluster of straight lines corresponding to each lane line and stop line. Each cluster of straight lines is cleaned and filtered, including removing the lines in each cluster whose slope is significantly different from the average slope of the lines in the cluster. Finally, the longest line is selected from the remaining lines as the final initial lane line structure detected by the Hough line algorithm.

[0066] like Figure 4As shown in the figure, since the lane line results directly obtained through Hough line detection cannot form a good closed figure, it is necessary to crop the redundant parts where the lines intersect and fill in the incomplete parts. The result of Hough line detection can obtain the starting point and end point of the line segment, thereby determining a straight line. By determining the intersection points of each lane segment and the stop line, the geometric structure of the road can be drawn accordingly. At the same time, the coordinates of each intersection point obtained in this step are the pixel coordinates of the lane line and stop line at the road intersection. After obtaining the pixel coordinates of each lane line, the area range of the lane can be determined. Figure 5 The following is a schematic diagram of the final lane geometry.

[0067] In a specific application, if the target road includes a crosswalk, the pixel coordinates of the crosswalk are extracted according to the same process described above. First, the pixel points of the crosswalk are extracted based on the color corresponding to the crosswalk image segmentation. Then, Hough line detection is performed on the crosswalk. The Hough line detection results are the start and end points of the identified lines, which are concentrated above and below the crosswalk, respectively. The Hough line detection results are clustered to separate the upper and lower clusters of lines. Each cluster is cleaned to remove lines with slopes significantly different from the average slope of the cluster. Finally, the longest line in each cluster is selected as the final result.

[0068] In one embodiment of the present application, the semantic segmentation of the road background image, the extraction of road elements, and the acquisition of lane geometry using the Hough line detection algorithm further includes:

[0069] S401: Performing least squares straight line fitting based on the vehicle's driving trajectory to determine the lane centerline.

[0070] S403: Fill in the missing lane lines based on the lane centerline to determine a closed lane geometry structure.

[0071] In one embodiment of the present application, when image segmentation alone is insufficient to determine the complete road geometry, trajectory information is incorporated for further analysis. By obtaining all vehicle trajectories within the lane area, a least squares linear fit is performed on each lane to ultimately determine the centerline of each lane. Specifically, the parameters are estimated by minimizing the value of a loss function, thereby fitting the data. Using the least squares linear fit method is equivalent to finding a and b, ensuring that the following formula holds true as much as possible.

[0072]

[0073] The advantage of the least squares method is that it requires fewer parameters when fitting a curve and can effectively eliminate outliers. In addition, it can be used to fit multivariate functions, which means it can be applied to various types of functions, not just linear functions.

[0074] Afterwards, the missing lane lines are filled in with the help of the fitted center line of the lane. The fitted lane center line is translated by the distance between the lane center line and the complete lane on the other side. The lane is translated to both sides according to this distance, and finally a closed road geometric structure is formed.

[0075] S205: Extracting a vehicle driving trajectory based on the lane geometry, calculating a vector angle based on the vehicle driving trajectory, and performing trajectory fitting using Bezier curve fitting to obtain a lane driving function.

[0076] In an embodiment of the present application, based on the extracted lane geometry, the YOLOV7 target detection model is used to accurately capture the trajectory of the moving vehicle, and the vector angle is calculated based on the vehicle's driving trajectory, and finally the driving function of each lane can be obtained.

[0077] Specifically, in one embodiment of the present application, extracting the vehicle driving trajectory based on the lane geometry structure includes:

[0078] S501: Use the YOLOV7 target detection model to detect and track the vehicle, and determine the initial vehicle driving trajectory based on the vehicle target detection frame.

[0079] S503: Using the DBSCAN density clustering method to clean the initial vehicle driving trajectory to obtain the target vehicle driving trajectory.

[0080] In one embodiment of the present application, in order to extract the driving trajectory points of the vehicle, a multi-target detection model YOLOV7 that can simultaneously detect different types of vehicles and pedestrians from the roadside perspective is used to detect and track vehicles in real-time video data, and the center point of the vehicle target detection frame is used as the vehicle's trajectory. Specifically, in the continuous frames of the real-time video, the bounding box information of each vehicle is output with the help of the target detection algorithm, and its results include (x, y, w, h). Wherein x is the horizontal coordinate of the center point of the detection frame, y is the vertical coordinate of the center point of the detection frame, w is the width of the detection frame, and h is the height of the detection frame. Based on this, the center coordinates of the bounding box can be directly obtained and regarded as the vehicle's trajectory point, which is usually the center point of the rectangular detection frame. The vehicle target center points with the same ID extracted in each frame are connected in chronological order to finally form the vehicle's motion trajectory. The trajectory points of the same vehicle are all the trajectory points from the time the vehicle starts from a certain lane to the time when the vehicle cannot be detected. The trajectory of the same vehicle is confirmed based on the tracking ID of the vehicle, and then the trajectory points of the vehicle are formed, such as Figure 6As shown. At the same time, the range of the lane is determined based on the lane geometry, and then vehicles whose starting point of the first frame appears outside the lane are filtered out, such as vehicles coming from the opposite direction. This trajectory extraction method is easy to implement and does not require complex data processing and algorithms. At the same time, since only the center point of the target box needs to be extracted, the calculation speed is fast and it is suitable for real-time applications. By connecting the center points of the vehicle targets, the movement trajectory of the vehicle can be intuitively displayed and subsequent analysis can be performed. It should be noted that the multi-target detection model YOLOV7 is based on the constructed vehicle detection dataset, which is divided into a training set and a validation set in a ratio of 7:3 and trained on a graphics workstation. The main hyperparameter settings of the model are shown in Table 1.

[0081] Table 1 Model training hyperparameter settings

[0082]

[0083] Since the recognition of certain vehicle types may vary between frames, the jump in target detection types will cause the tracking ID to jump, thus affecting the trajectory. At the same time, when two vehicles intersect and are occluded, it is also easy for tracking IDs to be interchanged when the detection frames overlap. Therefore, in order to ensure the accuracy of subsequent trajectory analysis, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm is used to clean the trajectory. The DBSCAN clustering method can be used to divide the starting points of the trajectories on the same lane into the same cluster, and the starting points of the trajectories on different lanes into different clusters, and effectively filter out some trajectory noise points, such as outliers that are significantly deviated from the other end points. Figure 7 As shown in the figure, different colors of trajectory points represent different classification results. The red point represents the centroid of the starting point of each type of trajectory. At the same time, the number of clusters also represents the number of lanes on the road section. Therefore, the result of density clustering can also accurately obtain the number of lanes on the road. Find the vehicle ID corresponding to each cluster set to obtain all vehicle trajectories of that category. In actual road traffic scenarios, it is generally believed that the closer the vehicle is to the centerline of the road, the more complete the trajectory is and the better the effect is. That is, the trajectories on the same lane can be clustered together for analysis. The trajectories within a certain range close to the cluster centroid on each lane are selected as the final retained trajectories. According to the pixel coordinates of the red centroid point, all lane trajectory sets are sorted from left to right according to the lane order.

[0084] Furthermore, when detecting and tracking vehicles in real-time road video, the target detection model is less effective at identifying certain vehicle types, leading to variations in the identified types between frames. These changes in target detection types can lead to changes in the tracking ID, which in turn affects the trajectory, causing it to become broken or incomplete. Furthermore, when two vehicles intersect and become occluded, tracking IDs are more likely to be swapped when the detection frames overlap. Therefore, to ensure the accuracy of subsequent trajectory analysis, vehicle tracks captured in each lane must be cleaned and processed separately. For example, tracks in each lane with a length less than one-quarter of the image width are considered incomplete and removed.

[0085] In one embodiment of the present application, the calculating of the vector angle based on the vehicle driving trajectory and the use of Bezier curve fitting to perform trajectory fitting to obtain the lane driving function include:

[0086] S601: Select the starting point, end point and midpoint of each vehicle's driving trajectory to form two vectors, calculate the angle between the vectors, and determine the vehicle's driving direction.

[0087] S603: Perform trajectory fitting on vehicle driving trajectories corresponding to the same vehicle driving direction using Bezier curve fitting to determine lane driving functions.

[0088] In one embodiment of the present application, based on the starting point classification results, the vehicle ID corresponding to each starting point can be obtained to obtain the corresponding complete vehicle trajectory, and then the trajectories belonging to each lane can be separately classified, the trajectories on the same lane can be merged together for analysis, the trajectories on different lanes can be separated, each trajectory on each lane can be analyzed separately, the turning of each vehicle can be extracted, and then summarized to obtain the lane function of the lane. The direction of vehicle travel is determined by calculating the vector angle, and the starting point, end point and midpoint of each trajectory are selected to form two vectors. The starting point and the midpoint of the trajectory form a vector, and the direction is from the starting point to the midpoint of the trajectory. This can approximately represent the initial direction of the vehicle, such as Figure 8 As shown by vector a in the figure; the midpoint of the trajectory and the end point of the trajectory form another vector, the direction of which is from the midpoint of the trajectory to the end point of the trajectory, which can approximately represent the end direction of the vehicle, as shown in Figure 8 As shown by vector b in , the angle ultimately used to determine the vehicle's direction of travel is the angle between these two vectors. The calculation formula for the vector angle is as follows:

[0089]

[0090] Among them, cos<a,b> represents the angle between vector a and vector b, a·b is the dot product of vector a and vector b, |a| is the modulus of vector a, and |b| is the modulus of vector b. If the vectors are represented by coordinates, that is, a=(x1,y1), b=(x2,y2), then:

[0091] a·b=x1x2+y1y2

[0092]

[0093] from Figure 8 As can be seen from the figure, when a vehicle turns left, the starting direction vector must turn counterclockwise toward the ending direction vector. When turning right, the starting direction vector must turn clockwise toward the ending direction vector. This is not necessarily the case for vehicles traveling straight ahead. Therefore, the sign of the angle between the two vectors is used to determine whether the vehicle is turning left or right. The sign of the vector angle is calculated using the following formula.

[0094] cross(a,b)=x1×y2-y1×x2

[0095] If the calculated result is a positive number, it indicates that the rotation from vector a to vector b is counterclockwise, meaning the trajectory is a left turn; if it is negative, it is clockwise, meaning the trajectory is a right turn. All angle values calculated for each lane are similarly clustered. The number of clusters indicates the number of vehicle trajectories in that lane and, therefore, the lane's function. Clustering also helps eliminate anomalous trajectory angles and those of a small number of vehicles violating traffic regulations. A clustering result of two clusters indicates that the lane has two steering functions, while a cluster with only one cluster indicates that the lane has only one steering function.

[0096] Next, various turning trajectory curves for each lane are fitted, and the fitting results are used to represent the driving direction in that lane. Given that vehicle trajectories are typically curved, Bezier curve fitting is used for trajectory fitting. Bezier curve fitting is a mathematical technique that approximates an actual curve using given discrete points. Bezier curves are calculated based on Bezier polynomials, and their order corresponds to the number of control points. For example, a first-order Bezier curve has two control points, a second-order one has three, and so on. The order of a Bezier curve corresponds to the number of control points. The nth-order Bezier polynomial is shown below.

[0097]

[0098] Among them, point P is a known point and point B is the final point to be found.

[0099] When fitting a Bezier curve, the most important thing is to select appropriate control points on the original curve. By extracting all the tracks of the same type on each lane, starting from the bottom point of the track, every time you move 50 pixels upward in the vertical direction, calculate the average value of the horizontal coordinates of all the tracks in the horizontal direction as a control point, and so on, scan upward in the vertical direction until there are no track points in a certain horizontal direction, such as Figure 9 After selecting the control points, the trajectory of each lane in each driving direction is fitted with the Bezier curve, and the result is as follows: Figure 10 shown.

[0100] In addition, some first lanes on the left have both left-turn and U-turn functions, but it is difficult to separate the two functions through angle clustering. Therefore, the first lane on the left should also be separately judged and screened for U-turn trajectories and analyzed separately according to the above trajectory analysis method. Through observation, it can be found that the end point of the U-turn trajectory often falls in the left symmetrical area of the lane area. Therefore, it is only necessary to screen the trajectory in the left symmetrical area of the left turn lane where the trajectory ends to obtain the vehicle U-turn trajectory. However, at some road intersections, due to the coverage of the camera, the trajectory of the U-turn vehicle cannot be fully captured, so the corresponding U-turn trajectory cannot be screened and the lane U-turn function cannot be derived. For these scenes, the lane function can only be supplemented manually.

[0101] In one embodiment of the present application, the process of calculating the vector angle based on the vehicle driving trajectory and performing trajectory fitting using Bezier curve fitting to obtain the lane driving function includes:

[0102] The vehicle driving trajectory in the waiting area is filtered based on time and vehicle status, wherein the vehicle status is determined based on the ratio of the actual change distance of the center point of the vehicle detection frame to the length of the vehicle detection frame.

[0103] In one embodiment of the present application, intersections in urban traffic networks have long been traffic bottlenecks, playing a crucial role in traffic efficiency and capacity. Generally speaking, traffic engineers often use the method of setting up waiting areas to improve the capacity of intersections. Generally speaking, waiting areas are mainly set up at intersections with high traffic volume and large scale. By extending the main stop line inward from the intersection, the time vehicles wait to pass through the intersection is shortened. In addition, waiting areas are generally divided into two types: left-turn waiting areas and straight-ahead waiting areas. When a left-turn waiting area is present at a road intersection, vehicles enter the waiting area to wait, which will cause traffic congestion and the following vehicle will block the leading vehicle. In this case, the target detection model cannot recognize the obscured vehicle, causing the track point belonging to the vehicle to disappear at this point, and the track is terminated. When the vehicle can continue to pass and move again, although the vehicle will be redetected due to the disappearance of the obstruction, the vehicle's tracking ID has changed and is no longer the same as the original ID. Therefore, a new track begins, and the original track cannot be continued. This is inconsistent with the vehicle's actual operating trajectory, leading to errors in subsequent analysis.

[0104] In order to filter out vehicle trajectories that stop and terminate in the waiting area from among numerous trajectories, it is first necessary to analyze the driving characteristics of these vehicles. These vehicles often stop and wait after driving into the waiting area. During this period of time, the target detection model cannot recognize them due to occlusion by subsequent vehicles. Therefore, the trajectories of these vehicles are often in a stopped state in the last few frames of the trajectory. Using this as a filtering condition, it is only necessary to determine whether the vehicle trajectory is in a stopped state at the end. Based on this, when filtering vehicle trajectories, the timestamp of the last set of vehicle trajectory data is first read, and then the timestamps of the trajectory data are read forward in sequence. The trajectory data within 5 seconds before the trajectory stops is selected for analysis. It is determined whether the vehicle is always stopped within 5 seconds. If the vehicle is always stopped within 5 seconds before the trajectory terminates, it indicates that the vehicle entered the waiting area to wait and was blocked and could not be identified, thus terminating the trajectory. Such trajectories should be filtered out. Specifically, the ratio of the actual change distance of the center point of the vehicle detection frame to the length of the vehicle detection frame is used to determine the vehicle stop analysis. The specific calculation formula is as follows:

[0105]

[0106] l=l last ×δ

[0107] Among them, x last 、y last The x is the horizontal and vertical coordinates of the center point of the vehicle detection frame at the current timestamp, last' 、y last'The horizontal and vertical coordinates of the center point of the vehicle detection frame one second before this timestamp are calculated by the formula to calculate the moving distance of the center point of the vehicle detection frame within this second. last is the length of the vehicle detection frame at the current timestamp, and δ is the reduction factor of the vehicle detection frame, which is used to reduce the length of the vehicle detection frame. After a large number of experiments, it is more appropriate to take a value of 0.3.

[0108] Calculate the distance the center of the vehicle detection frame moves relative to the length of the reduced vehicle detection frame. If the distance the center of the vehicle detection frame moves is greater than the length of the vehicle detection frame, the vehicle is determined to have not been stationary within that second. If the distance the center of the vehicle detection frame moves is less than the length of the vehicle detection frame, the vehicle is determined to have been stationary within that second. If the vehicle has been stationary continuously for the last five seconds, the vehicle is considered stationary. Read the trajectories of all vehicles in a certain period of time. If the vehicle remains stationary for the last five seconds of the trajectory, it is determined that the vehicle has entered the waiting state in the waiting area and the target detection frame disappears. The subsequent trajectory cannot continue, causing the trajectory to terminate at this point. When the number of such trajectories reaches 10% of all vehicles, it is determined that a waiting area exists in the current area.

[0109] In one embodiment of the present application, the calculating of the vector angle based on the vehicle driving trajectory and the use of Bezier curve fitting to perform trajectory fitting to obtain the lane driving function further includes:

[0110] For the vehicle driving trajectory in the waiting area, the centroid is calculated based on the coordinates of the trajectory end point, and the lane function of the waiting area is determined based on the centroid.

[0111] In one embodiment of the present application, the approximate range of the waiting area is determined based on the end point of the trajectory that terminates in the waiting area, and then the vehicle trajectory whose starting point begins in the area is selected from the trajectory. After the corresponding trajectory is selected, the steering of these trajectories is determined to obtain the driving function of the corresponding waiting area. Considering that the waiting area is usually divided into a left-turn waiting area or a straight-ahead waiting area, it only has one steering function. Therefore, the center of mass is calculated for the coordinates of the selected trajectory end point, and the distance is compared with the last point of the Bezier fitting curve. If it is more inclined to the left-turn fitting curve, then the waiting area has the left-turn function. If it is closer to the straight-ahead Bezier curve, then the waiting area has the straight-ahead function.

[0112] S207: Using the Kmeans clustering method to classify the end points of the vehicle's driving trajectory to determine the intersection driving area division.

[0113] In the embodiment of the present application, the endpoints of vehicles in historical trajectory data are collected in order to analyze the endpoints by utilizing the vehicle's running trajectory. After a series of preprocessing and cluster analysis, the endpoints of the vehicle trajectory historical data within the period are analyzed after filtering out some noise points, and all endpoints are clustered to determine the gathering points of vehicle endpoints in different driving directions. Based on this, the convergence points of vehicle trajectories in each driving direction are determined, and the driving road surface can be divided into areas with different driving directions. Specifically, in combination with the determined lane area, all vehicle trajectory data are obtained and the endpoints of each vehicle's trajectory are filtered out. The pixel coordinates of the center point of the detection frame in the last frame of each vehicle's trajectory are identified as the vehicle's endpoint. Based on the determined lane driving function, the Kmeans clustering method is used to cluster vehicles with the same direction of travel together. The number of vehicle driving directions in the road scene should be the same as the number of categories. It should be noted that the cluster number k value of the Kmeans clustering method is determined based on the lane driving function, that is, the sum of the number of lane function types and the number of waiting area types. After clustering is complete, destinations belonging to the same category are placed into a single list based on their category labels. The coordinates of the centroids of each category are then calculated and sorted from left to right into a large array based on the magnitude of the centroid's horizontal coordinates. This forms the final destination array. After this sorting, destination clusters with the same index are assigned the same color according to the color list, resulting in the final destination clustering result. Within the intersection driving area, the area corresponding to the destination clustering result represents the functional area division of the intersection driving area. This division is achieved by diffusing the destination clusters to form blocks of different colors.

[0114] S209: Drawing a road structure map based on the lane geometry, lane driving function, and intersection driving area division.

[0115] In this embodiment, a black image is created based on the video resolution. Based on the lane geometry, pixels within the lane area in the black image are assigned a color not in the color list, such as a uniform yellow color (0, 255, 255). The lane structure is then drawn on top of the image based on the coordinates of the stop line and lane lines. Next, starting from the upper left corner of the image, each black pixel in the remaining portion of the image is traversed from left to right and from top to bottom. The Euclidean distance from each pixel to the endpoint cluster is calculated, and the color of the pixel is set to the color of the endpoint cluster closest to the pixel. The closest distance is calculated by sequentially calculating the distance between the pixel and all points in each cluster, with the minimum distance being considered the distance between the pixel and the endpoint cluster. The distances between the pixel and each endpoint cluster are compared. The pixel with the smallest distance to the endpoint cluster is closest to the endpoint cluster, and the pixel is assigned the color of the endpoint cluster.

[0116] In addition, different confidence levels are divided according to the different color block areas. The confidence level is the reliability of the accurate calculation results. The closer the vehicle's endpoint is to the center of the color block, the higher the reliability of the vehicle belonging to the corresponding turn of the color block. Conversely, the closer the vehicle's endpoint is to the center of the color block, the lower the reliability of the vehicle belonging to the corresponding turn of the color block. Specifically, according to the previous logic, the distance between each black pixel in the image and the different endpoint clusters is calculated. If the black pixel is closest to the second endpoint cluster from left to right, the distance between the black pixel and the centroid of the endpoint cluster is calculated and saved. Similarly, each endpoint cluster creates a distance list and saves the distance between the black pixel closest to itself and its own centroid in the list. After all the black pixels in the image have been traversed, the black pixels are colored. Taking the confidence level into consideration, we use a color gradient method when coloring. We filter the maximum distance in the distance list stored in each endpoint cluster and divide it by 10. That is, we select the farthest distance from the centroid of this type of endpoint on the image and divide this distance into 10 parts. Then, based on the initial color of the endpoint cluster sequence in the color list, we change the R, G, and B channel values ​​equidistantly to form 11 different gradient colors. Different gradient colors correspond to different confidence levels, and the confidence level increases from far to near. By performing the same operation on each endpoint cluster, we divide the maximum distance of the black pixel from the center cluster by 10, that is, the distance between the pixel and the centroid of the endpoint cluster is divided into 11 intervals. Different intervals are assigned different gradient colors and represent different confidence levels. The final map is as follows Figure 11 shown.

[0117] In one embodiment of the present application, the method further includes:

[0118] The vehicle driving direction is determined based on the road structure map.

[0119] In one embodiment of the present application, for a newly appeared vehicle, the vehicle's direction of travel can be determined based on the color of the area where the vehicle's destination pixel point is located. Figure 12 As shown in the figure, we use the detection frame center, the detection frame bottom edge midpoint and the center point of the line connecting the detection frame center and the bottom edge center point to judge the vehicle, in order to increase the accuracy. When one of the three judgment points of the vehicle falls into the lane area, such as Figure 13 If the white line in the middle is shown, the vehicle is considered to have broken its track before leaving the lane area and will be discarded. When the three judgment points of the vehicle all fall into the same interval of the same color block, the vehicle's turn and ID number are counted and the corresponding confidence level is recorded. Figure 13The vehicle shown by the pink-purple line will be recorded as a right-turning vehicle with a confidence level greater than 90%. When the three judgment points of a vehicle all fall into the same color block but in different intervals, the vehicle's turn and ID number are also counted, but the confidence values are different. When the three points fall into two intervals, the confidence level of the interval with more points is used as the basis. When the three points fall into three intervals, the confidence level of the middle one is used as the basis. Figure 13 The vehicle shown in the blue line trajectory is recorded as a left-turning vehicle with a confidence level greater than 90%. When the three judgment points of the vehicle fall into different color blocks, such as Figure 13 As shown in the green line trajectory, it means that the final turn of the vehicle cannot be determined, so the ID number of the vehicle is recorded and classified as an incorrectly judged vehicle.

[0120] In this embodiment, the vehicle's turning intention can be inferred in real time, which provides a new idea for intelligent traffic control and is of great significance to improving the efficiency and safety of intersection traffic.

[0121] In the above-mentioned method for extracting road structure information based on real-time video data, first, real-time video data is acquired and a road background image is extracted; semantic segmentation is performed on the road background image to extract road elements, and the lane geometry is obtained using the Hough line detection algorithm; vehicle driving trajectories are extracted based on the lane geometry, vector angles are calculated based on the vehicle driving trajectories, and Bezier curve fitting is used to fit the trajectories to obtain lane driving functions; the Kmeans clustering method is used to classify the end points of the vehicle driving trajectories to determine the intersection driving area division; and a road structure map is drawn based on the lane geometry, lane driving functions, and intersection driving area division. In other words, the lane geometry is extracted by using image-based semantic segmentation technology, and the Hough line detection algorithm is used to achieve construction from points to lines and then to surfaces. Secondly, lane function attributes are extracted through vehicle trajectory analysis, and the target detection model and tracking algorithm are combined with density clustering to accurately identify the driving functions of each lane and waiting area. Finally, through cluster analysis of the vehicle trajectory endpoints, the intersection driving area is divided, and the vehicle's turning direction is determined in real time, effectively integrating image and trajectory information, greatly improving the accuracy and frequency of road structure information extraction, and enhancing the ability to understand complex traffic scenarios.

[0122] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0123] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 14 As shown. The computer device includes a processor, memory, communication interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for extracting road structure information based on real-time video data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0124] Those skilled in the art will understand that Figure 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0125] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0129] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0130] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for extracting road structure information based on real-time video data, characterized in that: The method comprises: Acquire real-time video data and extract road background images; Performing semantic segmentation on the road background image to extract road elements and using the Hough line detection algorithm to obtain lane geometry; Extracting a vehicle driving trajectory based on the lane geometry, calculating a vector angle based on the vehicle driving trajectory, and performing trajectory fitting using Bezier curve fitting to obtain a lane driving function; Using the Kmeans clustering method to classify the end points of the vehicle's driving trajectory and determine the intersection driving area division; A road structure map is drawn based on the lane geometry, lane driving function, and intersection driving area division.

2. The method for extracting road structure information based on real-time video data according to claim 1, characterized in that: The semantic segmentation of the road background image, extraction of road elements, and acquisition of lane geometry using the Hough line detection algorithm include: Using the BiSeNetV2 network model to perform semantic segmentation on the road background image to obtain a set of pixel points corresponding to traffic objects; Clustering is performed based on the pixel point set of the corresponding traffic object using the Hough line detection algorithm to determine the initial lane line structure; Based on the initial lane line structure, the intersection points of the intersecting straight lines are determined to form a closed figure and determine the lane geometry.

3. The method for extracting road structure information based on real-time video data according to claim 2, characterized in that: The semantic segmentation of the road background image, extraction of road elements, and acquisition of lane geometry using the Hough line detection algorithm further includes: Performing least squares straight line fitting based on the vehicle's driving trajectory to determine the lane centerline; Missing lane lines are filled in based on the lane centerline to determine a closed lane geometry.

4. The method for extracting road structure information based on real-time video data according to claim 1, characterized in that: Extracting the vehicle driving trajectory based on the lane geometry structure includes: The YOLOV7 target detection model is used to detect and track vehicles, and the initial vehicle trajectory is determined based on the vehicle target detection frame; The DBSCAN density clustering method is used to clean the initial vehicle driving trajectory to obtain the target vehicle driving trajectory.

5. The method for extracting road structure information based on real-time video data according to claim 1, characterized in that: The vector angle is calculated based on the vehicle driving trajectory, and the trajectory is fitted using Bezier curve fitting to obtain the lane driving function, which includes: The starting point, end point and midpoint of each vehicle's trajectory are selected to form two vectors, and the angle between the vectors is calculated to determine the vehicle's direction of travel; The vehicle driving trajectories corresponding to the same vehicle driving direction are fitted using Bezier curve fitting to determine the lane driving function.

6. The method for extracting road structure information based on real-time video data according to claim 1, characterized in that: The method of calculating the vector angle based on the vehicle driving trajectory and performing trajectory fitting using Bezier curve fitting to obtain the lane driving function includes: The vehicle driving trajectory in the waiting area is filtered based on time and vehicle status, wherein the vehicle status is determined based on the ratio of the actual change distance of the center point of the vehicle detection frame to the length of the vehicle detection frame.

7. The method for extracting road structure information based on real-time video data according to claim 6, characterized in that: The calculating of the vector angle based on the vehicle driving trajectory and performing trajectory fitting using Bezier curve fitting to obtain the lane driving function further includes: For the vehicle driving trajectory in the waiting area, the centroid is calculated based on the coordinates of the trajectory end point, and the lane function of the waiting area is determined based on the centroid.

8. The method for extracting road structure information based on real-time video data according to claim 1, characterized in that: The method further comprises: The vehicle's driving direction is determined based on the road structure map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images