Method for generating road topology based on pictures from road end cameras

Abstract

The application relates to a road topology generation method based on pictures of road end cameras and relates to the field of automatic driving. The method comprises the following steps: acquiring image data and vehicle data of a target intersection in a target time period; inputting the image data into a semantic segmentation model to output object preliminary identification results corresponding to the image data; inputting the vehicle data and the image data into a multi-target tracking network to output target tracking result information corresponding to the target intersection; and combining the object preliminary identification results and the target tracking result information to generate road topology optimization data. The road topology structure at a crossroad is constructed by using the computer vision method and through data acquisition by the road end cameras, which can help the vehicle to more accurately understand the traffic information of the road end, realizes higher labeling accuracy, and saves human resources required by manual labeling.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a method for generating road topology based on roadside camera images. Background Technology

[0002] In autonomous driving scenarios, the road topology can provide vehicles with rich road information, including the positions of lane lines, stop lines, pedestrian crossings, and the topological relationships between lanes. This helps the autonomous driving system infer the driving trajectories of other vehicles and output more reasonable path decisions for the vehicle itself.

[0003] In existing technologies, most road topology generation methods rely on high-precision maps as prior art, especially in complex scenarios such as large intersections. However, the construction of high-precision maps requires extensive manual annotation, resulting in poor timeliness, high labor costs, and a high susceptibility to errors. Furthermore, the road topology generated from high-precision maps often differs from actual human driving trajectories. Currently, some methods attempt to generate local high-precision maps using vehicle-mounted sensors to reduce the time and labor costs associated with manual annotation. Additionally, there are methods that utilize crowdsourced data to generate road topology structures.

[0004] However, the methods in the relevant technologies can still be optimized in terms of accuracy. Summary of the Invention

[0005] This invention relates to a method for generating road topology structures based on roadside camera images, which can improve annotation accuracy while saving manpower. This method is applied to computer equipment and includes:

[0006] Acquire image data and vehicle data of the target intersection within the target time period. The image data is used to indicate the video content of the target intersection within the target time period, and the vehicle data is used to indicate the driving status of vehicles passing through the target intersection within the target time period.

[0007] The image data is input into the semantic segmentation model, and the output is the preliminary object recognition result corresponding to the image data;

[0008] The vehicle data and image data are input into the multi-target tracking network, and the target tracking result information corresponding to the target intersection is output.

[0009] By combining the preliminary object recognition results and target tracking results, road topology optimization data is generated, which is used to guide road topology updates.

[0010] In an optional embodiment, the semantic segmentation model is an artificial intelligence model based on semantic information extraction and semantic segmentation;

[0011] The image data is input into the semantic segmentation model, and the output is a preliminary object recognition result corresponding to the image data, including:

[0012] Input the image data into the semantic segmentation model;

[0013] High-dimensional image features are obtained by extracting features from image data using a semantic segmentation model.

[0014] High-dimensional image features are mapped using a semantic segmentation model to obtain roadside features;

[0015] Semantic segmentation is performed on the roadside features to output preliminary results of objects corresponding to the image data.

[0016] In one optional embodiment, the preliminary object results include at least one of lane line recognition results, stop line recognition results, pedestrian crossing recognition results, road sign recognition results, and obstacle recognition results.

[0017] In an optional embodiment, the image data is multi-view image data, which corresponds to at least two image views;

[0018] Vehicle data and image data are input into a multi-target tracking network, which outputs target tracking results corresponding to the target intersection, including:

[0019] Image data is synchronized and sampled to obtain a data stream corresponding to the sampling length during the sampling process; the data stream includes roadside images corresponding to the number of image viewpoints;

[0020] Determine the set of feature identification information corresponding to the target intersection based on the roadside image;

[0021] The feature identification information set is input into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection.

[0022] In an optional embodiment, the feature identification information set is input into a multi-target tracking network to obtain target tracking result information corresponding to the target intersection, including:

[0023] The feature identification information set and road features are input into the multi-target tracking network to obtain target tracking result information corresponding to the target intersection. The road features include road center point features, road deviation angle information and road boundary information.

[0024] In an optional embodiment, the road topology optimization data includes lane line optimization data;

[0025] Combining preliminary object recognition results and target tracking results, road topology optimization data is generated, including:

[0026] Initial lane topology data is determined based on preliminary object recognition results;

[0027] Spatial clustering is performed on the target tracking results to obtain spatial clustering data;

[0028] The number of overlaps was determined based on spatial clustering data;

[0029] The initial lane topology data is adjusted based on the overlap count data to obtain optimized lane data.

[0030] In an optional embodiment, the road topology optimization data includes stop line optimization data;

[0031] Combining preliminary object recognition results and target tracking results, road topology optimization data is generated, including:

[0032] Initial parking line topology data is determined based on preliminary object recognition results;

[0033] Spatial clustering is performed on the target tracking results to obtain spatial clustering data;

[0034] Vehicle stopping trajectories are fitted based on spatial clustering data to obtain a set of predicted vehicle stopping lines;

[0035] The initial parking line topology data is updated based on the vehicle stop line prediction set to obtain optimized stop line data.

[0036] In an optional embodiment, the road topology optimization data includes lane trajectory optimization data;

[0037] Combining preliminary object recognition results and target tracking results, road topology optimization data is generated, including:

[0038] Initial lane trajectory data is generated based on the preliminary object recognition results;

[0039] A lane connection matrix is ​​generated based on the target tracking results. The lane connection matrix is ​​used to represent the connectivity state of the lanes.

[0040] The initial lane trajectory information is updated based on the lane connection matrix to obtain optimized lane trajectory data. In an optional embodiment, the optimized road topology data includes optimized pedestrian crossing data;

[0041] Combining preliminary object recognition results and target tracking results, road topology optimization data is generated, including:

[0042] Initial pedestrian crossing data is generated based on preliminary object recognition results;

[0043] Generate pedestrian information based on target tracking results;

[0044] Pedestrian trajectory information is obtained by clustering and fitting pedestrian information.

[0045] The initial pedestrian crossing data is updated by using pedestrian trajectory information to obtain optimized pedestrian crossing data.

[0046] The beneficial effects of the technical solution provided by this invention include at least the following:

[0047] By utilizing roadside cameras at intersections, a road topology structure of the intersection is established, and the generated results are further optimized using vehicle 4D trajectory information and pedestrian trajectory information. Using data collected from roadside cameras and constructing the road topology structure at intersections through computer vision methods helps vehicles more accurately understand the traffic information at that intersection, achieving higher annotation accuracy while saving the manpower required for manual annotation. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This illustration shows a method for generating road topology based on roadside camera images, provided by an exemplary embodiment of the present invention.

[0050] Figure 2 This invention illustrates another method for generating road topology based on roadside camera images, provided by an exemplary embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0052] Figure 1 This invention provides a method for generating road topology based on roadside camera images, illustrated in an exemplary embodiment. The method is described using an application in a computer device as an example. The method includes:

[0053] Step 101: Obtain image data and vehicle data of the target intersection within the target time period.

[0054] In this embodiment of the invention, image data is used to indicate the video content of the target intersection within a target time period, and vehicle data is used to indicate the driving status of vehicles passing through the target intersection within the target time period. Optionally, the image data can be video data obtained by capturing the target intersection in a single dimension, or video data obtained by capturing the target intersection from multiple dimensions.

[0055] That is, in this embodiment of the invention, the content indicated by the image is a video image.

[0056] Step 102: Input the image data into the semantic segmentation model and output the preliminary object recognition result corresponding to the image data.

[0057] In this embodiment of the invention, the preliminary object recognition result is used to characterize the image content.

[0058] Optionally, the semantic segmentation model has both semantic segmentation and semantic recognition functions. In one example, the semantic recognition model involved in this embodiment of the invention is an artificial intelligence model.

[0059] Step 103: Input vehicle data and image data into the multi-target tracking network, and output the target tracking result information corresponding to the target intersection.

[0060] In this embodiment of the invention, the multi-target tracking network is implemented as an artificial intelligence model, which can determine the tracking content of elements appearing in an image.

[0061] It should be noted that, in the embodiments of the present invention, steps 102 to 103 can be implemented sequentially or in parallel. The present invention does not limit the order of obtaining the preliminary object recognition results and target tracking results information corresponding to the image data.

[0062] Step 104: Combine the preliminary object recognition results and target tracking results to generate road topology optimization data.

[0063] In this embodiment of the invention, the road topology optimization data is used to indicate the process of optimizing the preliminary identification result of the currently identified object in combination with the target tracking result information. Optionally, the road topology optimization data can be implemented as at least one of the following: the prediction and adjustment result of intersection markings, the prediction and adjustment result of vehicle paths, and the prediction and adjustment result of pedestrian passages.

[0064] In this embodiment of the invention, the road topology optimization data is used to guide road topology updates.

[0065] Figure 2This invention illustrates another method for generating road topology structures based on roadside camera images, using an exemplary embodiment of the invention as an example. The method includes:

[0066] Step 201: Obtain image data and vehicle data of the target intersection within the target time period.

[0067] This process corresponds to the process shown in step 101, and will not be described again here.

[0068] It should be noted that the image data involved in the embodiments of the present invention is multi-view image data, which corresponds to at least two image perspectives.

[0069] Step 202: Input the image data into the semantic segmentation model.

[0070] In this embodiment of the invention, the computer device includes semantic segmentation and feature extraction functions.

[0071] Step 203: Extract features from the image data using a semantic segmentation model to obtain high-dimensional image features.

[0072] Step 204: Map the high-dimensional image features using a semantic segmentation model to obtain roadside features.

[0073] Step 205: Perform semantic segmentation on the road end features and output preliminary results of the objects corresponding to the image data.

[0074] In this embodiment of the invention, the semantic segmentation model includes a semantic information extraction module and a semantic information segmentation module. The semantic information extraction module is used to extract features from real-time images captured by roadside cameras, mapping the roadside information in the images to a high-dimensional feature space to obtain a roadside feature representation; the semantic information segmentation module then uses this feature representation as input to perform semantic segmentation, obtaining an initial semantic segmentation image of the intersection scene, and further obtaining initial predicted values ​​for road markings such as lane lines, stop lines, and pedestrian crossings.

[0075] It should be noted that, in one specific implementation, the computer device first acquires real-time images P from one or a group of roadside cameras. t ={p t1 ,p t2 ,…,p tn These images contain environmental and map information for the current road segment. The real-time images P obtained from the roadside cameras... t The information is input into the semantic information extraction module to obtain the roadside feature representation information F. t The road end features are represented as F. t The image is input into the semantic information segmentation module to obtain the roadside semantic segmentation image. It can predict the semantic category of each pixel in the grid corresponding to the intersection. Where N is the total number of semantic categories, including lane lines, stop lines, pedestrian crossings, other road markings and signs, obstacles, etc. Finally, straight line segments are used to... The pixels with semantic categories of lane lines and stop lines are fitted, and the pixels with semantic category of pedestrian crossings are fitted using polygons to obtain the initial predicted values ​​of these road markings.

[0076] Step 206: Perform data stream synchronization and sampling processing on the image data to obtain a data stream corresponding to the sampling length during the sampling process.

[0077] In this embodiment of the invention, the data stream includes roadside images corresponding to the number of image viewpoints.

[0078] For N c Data streams collected from roadside cameras at different angles are synchronized and sampled to obtain a sampling interval of length N. t synchronized data stream Each data stream contains N c Images of the roadside from different angles.

[0079] Step 207: Determine the set of feature identification information corresponding to the target intersection based on the roadside image.

[0080] Step 208: Input the feature identification information set and road features into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection.

[0081] Optionally, in an embodiment of the present invention, the data stream S sync The input is fed into a multi-view image feature extraction network to obtain a set of feature representation information of the intersection. Set F r Along with the learnable detection query vector and tracking query vector, they are fed frame-by-frame into the 3D multi-object detection network and the 3D multi-object tracking network to obtain a set of vehicle target tracking results.

[0082] In this embodiment of the invention, road features include road center point features, road slant angle information, and road boundary information. In one example, the set F... r The input to the intersection location prediction network predicts the road center point O, the road deviation angle ω, and the road boundary. The prediction results are used to select a set. The final set of vehicle four-dimensional trajectories is obtained. That is, the target tracking result information.

[0083] Step 209: Combine the preliminary object recognition results and target tracking results to generate road topology optimization data.

[0084] Next, this invention provides four examples to illustrate the optimization methods for road topology optimization data.

[0085] In one example, the road topology optimization data includes lane line optimization data. This process involves the processing of the vehicle's four-dimensional trajectory set. Clustering is performed spatially. If the number of overlaps between vehicle trajectories and lane lines exceeds a set threshold, the initial segmentation is considered to have an error, and lane lines are updated based on vehicle trajectories. If there are multiple vehicle trajectories with spacing exceeding a set threshold between adjacent lane lines on a road, the initial segmentation is considered to have omissions, and lane lines are added based on vehicle trajectories. If the number of vehicle trajectories between adjacent lane lines on a road is less than a set threshold, the initial segmentation is considered to have false positives, and lane lines are deleted based on vehicle trajectories. In other words, the computer equipment determines the initial lane line topology data based on the preliminary object recognition results; spatial clustering is performed on the target tracking results to obtain spatial clustering data; the overlap count data is determined based on the spatial clustering data; and the initial lane line topology data is adjusted based on the overlap count data to obtain optimized lane line data.

[0086] In another example, the road topology optimization data includes stop line optimization data. This process involves the collection of four-dimensional vehicle trajectories. Statistical analysis is performed over time to obtain a set of stopping nodes during vehicle movement. This set is then used to fit a predicted set of stopping lines based on the vehicle's trajectory. The initial predicted values ​​of the stopping lines are then updated based on this set. In other words, the computer equipment determines the initial parking line topology data based on preliminary object recognition results; spatial clustering is performed on the target tracking results to obtain spatial clustering data; the vehicle stopping trajectory is fitted based on the spatial clustering data to obtain a predicted set of vehicle stopping lines; and the initial parking line topology data is updated based on the predicted set of vehicle stopping lines to obtain optimized stopping line data.

[0087] In another example, the road topology optimization data includes lane trajectory optimization data. In this process, based on the semantic segmentation image of the intersection scene and the optimized road marking information, an adjacency matrix A with all initial elements being 0 is constructed. mn Based on vehicle four-dimensional trajectory set Connectable lanes p i With q j The corresponding matrix element a ij Updated to 1, the adjacency matrix describing the topological relationships between lanes on this road segment is finally obtained. That is, the computer device generates initial lane trajectory data based on the preliminary object recognition results; generates a lane connection matrix based on the target tracking results, which is used to represent the connectivity of lanes; and updates the initial lane trajectory information based on the lane connection matrix to obtain optimized lane trajectory data.

[0088] In another example, the road topology optimization data includes pedestrian crossing optimization data. During the modification process, based on the semantic segmentation image of the intersection scene and the optimized road marking information, an adjacency matrix A with all initial elements being 0 is constructed. mn Based on vehicle four-dimensional trajectory set Connectable lanes p i With q j The corresponding matrix element a ij Updated to 1, the adjacency matrix describing the topological relationships between lanes on this road segment is finally obtained. That is, the computer equipment generates initial pedestrian crossing data based on the preliminary object recognition results; generates pedestrian information based on the target tracking results; performs cluster fitting processing on the pedestrian information to obtain pedestrian trajectory information; and updates the initial pedestrian crossing data with the pedestrian trajectory information to obtain optimized pedestrian crossing data.

[0089] In summary, the methods provided in the various embodiments of this invention establish the road topology of an intersection using roadside cameras and further optimize the generated results using vehicle four-dimensional trajectory information and pedestrian trajectory information. By using roadside cameras to collect data and constructing the road topology at the intersection using computer vision methods, vehicles can more accurately understand the traffic information at that intersection, achieving higher annotation accuracy while saving the manpower required for manual annotation.

[0090] The above are merely optional embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating a road topology based on pictures from road-side cameras, characterized in that, The method is applied to a computer device, and the method comprises: acquiring image data of a target intersection in a target time period and vehicle data, the image data being used to indicate video content of the target intersection in the target time period, and the vehicle data being used to indicate vehicle driving states of vehicles passing through the target intersection in the target time period; inputting the image data into a semantic segmentation model to output object preliminary identification results corresponding to the image data; inputting the vehicle data and the image data into a multi-target tracking network to output target tracking result information corresponding to the target intersection; combining the object preliminary identification results and the target tracking result information to generate road topology structure optimization data, the road topology structure optimization data being used to guide road topology structure updating; wherein the image data is multi-view image data, and the multi-view image data corresponds to at least two image views; the inputting of the vehicle data and the image data into the multi-target tracking network to output the target tracking result information corresponding to the target intersection comprises: performing data stream synchronization and sampling processing on the image data to obtain a data stream corresponding to a sampling length in the sampling processing process, the data stream including road end images corresponding to the number of image views; determining a feature identification information set corresponding to the target intersection based on the road end images; inputting the feature identification information set into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection. 2.The road topology generation method based on pictures of road-side cameras according to claim 1, characterized in that, The semantic segmentation model is an artificial intelligence model based on semantic information extraction and semantic segmentation; the inputting of the image data into the semantic segmentation model to output the object preliminary identification results corresponding to the image data comprises: inputting the image data into the semantic segmentation model; extracting features of the image data through the semantic segmentation model to obtain high-dimensional image features; performing mapping processing on the high-dimensional image features through the semantic segmentation model to obtain road end features; performing semantic segmentation on the road end features to output the object preliminary results corresponding to the image data. 3.The road topology generation method based on pictures of road-side cameras according to claim 2, characterized in that, The object preliminary results include at least one of lane line identification results, stop line identification results, pedestrian crossing identification results, road sign identification results, and obstacle identification results. 4.The road topology generation method based on pictures of road-side cameras according to claim 1, characterized in that, the inputting of the feature identification information set into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection comprises: inputting the feature identification information set and road features into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection, the road features including road center point features, road angle information, and road boundary information. 5.The road topology generation method based on pictures of road-side cameras according to claim 1, characterized in that, The road topology structure optimization data includes lane line optimization data. the combining of the object preliminary identification results and the target tracking result information to generate the road topology structure optimization data comprises: determining initial lane line topology data based on the object preliminary identification results; performing spatial clustering on the target tracking result information to obtain spatial clustering data; determine overlap times data based on the spatial clustering data; adjust the initial lane line topology data based on the overlap times data to obtain lane line optimization data. 6.The road topology generation method based on pictures of road-side cameras according to claim 1, wherein, The road topology optimization data includes stop line optimization data; The combination of the object preliminary identification result and the target tracking result information generates road topology optimization data, including: determine initial stop line topology data based on the object preliminary identification result; spatial clustering of the target tracking result information to obtain spatial clustering data; based on the spatial clustering data, vehicle stop trajectory fitting is performed to obtain a vehicle stop line prediction set; update the initial stop line topology data based on the vehicle stop line prediction set to obtain the stop line optimization data. 7.The road topology generation method based on pictures of road-side cameras according to claim 1, wherein, The road topology optimization data includes lane trajectory optimization data; The combination of the object preliminary identification result and the target tracking result information generates road topology optimization data, including: generate initial lane trajectory data based on the object preliminary identification result; generate a lane connection matrix based on the target tracking result, the lane connection matrix being used to represent the connectivity state of the lane; update the initial lane trajectory information based on the lane connection matrix to obtain the lane trajectory optimization data. 8.The road topology generation method based on pictures of road-side cameras according to claim 1, wherein, The road topology optimization data includes pedestrian crossing optimization data; The combination of the object preliminary identification result and the target tracking result information generates road topology optimization data, including: generate initial pedestrian crossing data based on the object preliminary identification result; generate pedestrian information based on the target tracking result; cluster fitting processing is performed on the pedestrian information to obtain pedestrian trajectory information; update the initial pedestrian crossing data through the pedestrian trajectory information to obtain the pedestrian crossing optimization data.

Images