Road topological structure generation method based on road end camera picture

Abstract

The invention relates to a road topological structure generation method based on a road end camera picture, and relates to the field of automatic driving. The method comprises the following steps: acquiring image data and vehicle data of a target intersection in a target time period; inputting the image data into a semantic segmentation model, and outputting an object preliminary identification result corresponding to the image data; inputting the vehicle data and the image data into a multi-target tracking network, and outputting target tracking result information corresponding to the target intersection; and generating road topological structure optimization data in combination with the object preliminary identification result and the target tracking result information. The road end camera is used for collecting data, the road topological structure at the crossroad is constructed through a computer vision method, vehicles can be helped to more accurately understand the traffic information of the road end, higher marking precision is achieved, and meanwhile manpower resources needed by manual marking are saved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method for generating a road topology structure based on roadside camera images. Background Art

[0002] In autonomous driving scenarios, the topological structure of the road can provide vehicles with rich road information, including the location of lane lines, stop lines, crosswalks, and the topological relationship between lanes, thereby helping the autonomous driving system to infer the driving trajectories of other vehicles and output more reasonable path decisions for the vehicle itself.

[0003] In the existing technology, most methods for generating road topology structures need to rely on high-precision maps as a priori, especially in complex scenes such as large intersections. However, the construction of high-precision maps requires a lot of manual annotation, which has problems such as poor timeliness and high labor costs, and errors are prone to occur in the process. At the same time, the road topology structure generated by high-precision maps is usually different from the actual driving trajectory of humans. At present, some methods have attempted to generate local high-precision maps through vehicle-side sensors to reduce the time and labor costs required for manual annotation of high-precision maps. In addition, there are some methods that use crowdsourcing data to generate road topology structures.

[0004] However, the methods in the related art can still be optimized in terms of accuracy. Summary of the invention

[0005] The present invention relates to a method for generating a road topology structure based on roadside camera images, which can improve the annotation accuracy and save human resources. The method is applied to a computer device, and the method includes:

[0006] Acquire image data and vehicle data of a target intersection within a target time period, wherein the image data is used to indicate video content of the target intersection within the target time period, and the vehicle data is used to indicate the driving status of vehicles passing through the target intersection within the target time period;

[0007] The image data is input into the semantic segmentation model, and the preliminary recognition result of the object corresponding to the image data is output;

[0008] The vehicle data and image data are input into the multi-target tracking network, and the target tracking result information corresponding to the target intersection is output;

[0009] Combined with the preliminary object recognition results and target tracking result information, road topology optimization data is generated, and the road topology optimization data is used to guide the road topology update.

[0010] In an optional embodiment, the semantic segmentation model is an artificial intelligence model based on semantic information extraction and semantic segmentation;

[0011] The image data is input into the semantic segmentation model, and the output is the preliminary recognition result of the object corresponding to the image data, including:

[0012] Feed the image data into the semantic segmentation model;

[0013] The image data is subjected to feature extraction through the semantic segmentation model to obtain high-dimensional image features;

[0014] The high-dimensional image features are mapped through the semantic segmentation model to obtain the road end features;

[0015] The road-end features are semantically segmented and the preliminary results of the objects corresponding to the image data are output.

[0016] In an optional embodiment, the preliminary object result includes at least one of a lane line recognition result, a stop line recognition result, a crosswalk recognition result, a road sign recognition result, and an obstacle recognition result.

[0017] In an optional embodiment, the image data is multi-view image data, and the multi-view image data corresponds to at least two image viewing angles;

[0018] The vehicle data and image data are input into the multi-target tracking network, and the target tracking result information corresponding to the target intersection is output, including:

[0019] Performing data stream synchronization and sampling processing on the image data to obtain a data stream corresponding to the sampling length in the sampling processing process; the data stream includes road end images corresponding to the number of image viewing angles;

[0020] Determine a feature identification information set corresponding to a target intersection based on the roadside image;

[0021] The feature identification information set is input into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection.

[0022] In an optional embodiment, the feature identification information set is input into a multi-target tracking network to obtain target tracking result information corresponding to the target intersection, including:

[0023] The feature identification information set and the road features are input into the multi-target tracking network to obtain the target tracking result information corresponding to the target intersection. The road features include road center point features, road deflection angle information and road boundary information.

[0024] In an optional embodiment, the road topology optimization data includes lane line optimization data;

[0025] Combine the preliminary object recognition results and target tracking results to generate road topology optimization data, including:

[0026] Determine initial lane line topology data based on preliminary object recognition results;

[0027] Perform spatial clustering on the target tracking result information to obtain spatial clustering data;

[0028] Determine overlap frequency data based on spatial clustering data;

[0029] The initial lane line topology data is adjusted based on the overlap number data to obtain lane line optimization data.

[0030] In an optional embodiment, the road topology optimization data includes stop line optimization data;

[0031] Combine the preliminary object recognition results and target tracking results to generate road topology optimization data, including:

[0032] Determine initial parking line topology data based on preliminary object recognition results;

[0033] Perform spatial clustering on the target tracking result information to obtain spatial clustering data;

[0034] The vehicle stop trajectory is fitted based on the spatial clustering data to obtain the vehicle stop line prediction set;

[0035] The initial stop line topology data is updated based on the vehicle stop line prediction set to obtain the stop line optimization data.

[0036] In an optional embodiment, the road topology optimization data includes lane trajectory optimization data;

[0037] Combine the preliminary object recognition results and target tracking results to generate road topology optimization data, including:

[0038] Generate initial lane trajectory data based on preliminary object recognition results;

[0039] Generate a lane connection matrix based on the target tracking results. The lane connection matrix is ​​used to represent the connectivity status of the lanes.

[0040] The initial lane trajectory information is updated based on the lane connection matrix to obtain lane trajectory optimization data.

[0041] In an optional embodiment, the road topology optimization data includes pedestrian crossing optimization data;

[0042] Combine the preliminary object recognition results and target tracking results to generate road topology optimization data, including:

[0043] Generate initial crosswalk data based on preliminary object recognition results;

[0044] Generate pedestrian information based on target tracking results;

[0045] Perform cluster fitting processing on pedestrian information to obtain pedestrian trajectory information;

[0046] The initial crosswalk data is updated through pedestrian trajectory information to obtain optimized crosswalk data.

[0047] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0048] By using the roadside cameras at the intersection, the road topology of the intersection is established, and the generated results are further optimized through the four-dimensional trajectory information of vehicles and pedestrians. Using the roadside cameras to collect data and construct the road topology at the intersection through computer vision methods can help vehicles understand the traffic information at the roadside more accurately, achieve higher annotation accuracy, and save the human resources required for manual annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A method for generating a road topology structure based on roadside camera images provided by an exemplary embodiment of the present invention is shown.

[0051] Figure 2 Another method for generating a road topology structure based on roadside camera images provided by an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0053] Figure 1 A method for generating a road topology structure based on roadside camera images provided by an exemplary embodiment of the present invention is shown, and the method is described by taking the method applied in a computer device as an example. The method includes:

[0054] Step 101, acquiring image data and vehicle data of a target intersection within a target time period.

[0055] In the embodiment of the present invention, the image data is used to indicate the video content of the target intersection within the target time period, and the vehicle data is used to indicate the vehicle driving status of the vehicle passing through the target intersection within the target time period. Optionally, the image data can be video data obtained by shooting the target intersection in a single dimension, or video data obtained by shooting the target intersection from multiple dimensions.

[0056] That is, in the embodiment of the present invention, the content indicated by the image is a video image.

[0057] Step 102: input the image data into a semantic segmentation model, and output a preliminary object recognition result corresponding to the image data.

[0058] In the embodiment of the present invention, the preliminary object recognition result is used to characterize the image content.

[0059] Optionally, the semantic segmentation model has semantic segmentation and semantic recognition functions. In one example, the allegory recognition model involved in the embodiment of the present invention is an artificial intelligence model.

[0060] Step 103, inputting the vehicle data and the image data into a multi-target tracking network, and outputting target tracking result information corresponding to the target intersection.

[0061] In an embodiment of the present invention, the multi-target tracking network is implemented as an artificial intelligence model, and the multi-target tracking network can determine the tracking content of elements appearing in the image.

[0062] It should be noted that, in the embodiment of the present invention, step 102 to step 103 can be implemented as a sequential process or a parallel process, and the present invention does not limit the order of obtaining the preliminary object recognition results and target tracking result information corresponding to the image data.

[0063] Step 104 , combining the preliminary object recognition result and the target tracking result information to generate road topology optimization data.

[0064] In an embodiment of the present invention, the road topology optimization data is used to indicate the preliminary recognition result of the currently recognized object, and the optimization process is performed in combination with the target tracking result information. Optionally, the road topology optimization data is implemented as at least one of the predicted adjustment results for the intersection markings, the predicted adjustment results for the vehicle path, and the predicted adjustment results for the pedestrian passage.

[0065] In the embodiment of the present invention, the road topology structure optimization data is used to guide the update of the road topology structure.

[0066] Figure 2Another method for generating a road topology structure based on roadside camera images provided by an exemplary embodiment of the present invention is shown, and the method is described by taking the method applied in a computer device as an example. The method includes:

[0067] Step 201, acquiring image data and vehicle data of a target intersection within a target time period.

[0068] This process corresponds to the process shown in step 101 and will not be described in detail here.

[0069] It should be noted that the image data involved in the embodiment of the present invention is multi-view image data, and the multi-view image data corresponds to at least two image viewing angles.

[0070] Step 202: input the image data into the semantic segmentation model.

[0071] In the embodiment of the present invention, the computer device includes the functions of semantic segmentation and feature extraction.

[0072] Step 203: extract features from the image data using a semantic segmentation model to obtain high-dimensional image features.

[0073] Step 204 , mapping the high-dimensional image features through a semantic segmentation model to obtain road end features.

[0074] Step 205 , semantically segment the road end features and output a preliminary result of the object corresponding to the image data.

[0075] In the embodiment of the present invention, the semantic segmentation model includes a semantic information extraction module and a semantic information segmentation module. The semantic information extraction module is used to extract features from the real-time image collected by the roadside camera, map the roadside information in the image to a high-dimensional feature space, and obtain a roadside feature representation; the semantic information segmentation module uses this feature representation as input to perform semantic segmentation, obtain an initial semantic segmentation image of the intersection scene, and further obtain the predicted initial values ​​of road markings such as lane lines, stop lines, and crosswalks.

[0076] It should be noted that, in a specific implementation, the computer device first obtains a real-time image P of one or a group of roadside cameras. t ={p t1 , p t2 ,…,p tn}, these images contain the environment and map information of the current road section. t Input into the semantic information extraction module to obtain the road end feature representation information F t . The road end feature is represented by F t Input into the semantic information segmentation module to obtain the road end semantic segmentation image The semantic category of each pixel in the grid corresponding to the intersection will be predicted Where N is the number of all semantic categories, including lane lines, stop lines, crosswalks, other road markings and road signs, obstacles, etc. Finally, use straight line segments to The pixels with the semantic categories of lane lines and stop lines are fitted, and polygons are used to fit the pixels with the semantic category of crosswalks to obtain the predicted initial values ​​of these road markings.

[0077] Step 206 , performing data stream synchronization and sampling processing on the image data to obtain a data stream corresponding to the sampling length in the sampling processing process.

[0078] In the embodiment of the present invention, the data stream includes roadside images corresponding to the number of image viewing angles.

[0079] To N c The data streams collected by the roadside cameras at different angles are synchronized and sampled to obtain a sampling interval length of N. t The synchronized data stream Each data stream contains N c Roadside images from different angles.

[0080] Step 207: determining a feature identification information set corresponding to the target intersection based on the road end image.

[0081] Step 208: input the feature identification information set and the road features into the multi-target tracking network to obtain target tracking result information corresponding to the target intersection.

[0082] Optionally, in this embodiment of the present invention, the data stream S cync Input into the multi-view image feature information extraction network to obtain the feature representation information set of the intersection The set F r Together with the learnable detection query vector and tracking query vector, it is input into the 3D multi-target detection network and the 3D multi-target tracking network frame by frame to obtain the vehicle target tracking result set.

[0083] In the embodiment of the present invention, the road features include road center point features, road deflection information, and road boundary information. r Input to the intersection location prediction network to predict the road center point O, road deviation angle ω and road boundary, and use the prediction results as the basis for filtering the set Get the final vehicle four-dimensional trajectory set That is, target tracking result information.

[0084] Step 209 , combining the preliminary object recognition result and the target tracking result information to generate road topology optimization data.

[0085] Next, the present invention provides four examples to illustrate the optimization method for road topology optimization data.

[0086] In one example, the road topology optimization data includes lane line optimization data. In this process, the vehicle four-dimensional trajectory set Clustering is performed according to the spatial dimension. If the number of overlaps between the vehicle trajectory and the lane line exceeds the set threshold, it is considered that there is an error in the initial division, and the lane line is updated based on the vehicle trajectory; if there are multiple vehicle trajectories with a spacing exceeding the set threshold between two adjacent lane lines on a certain road, it is considered that there are omissions in the initial division, and lane lines are added based on the vehicle trajectory; if the number of vehicle trajectories between two adjacent lane lines on a certain road is lower than the set threshold, it is considered that there are false positives in the initial division, and lane lines are deleted based on the vehicle trajectory. That is, the computer device determines the initial lane line topology data based on the preliminary object recognition results; spatially clusters the target tracking result information to obtain spatial clustering data; determines the overlap number data based on the spatial clustering data; and adjusts the initial lane line topology data based on the overlap number data to obtain lane line optimization data.

[0087] In another example, the road topology optimization data includes stop line optimization data. Statistics are performed according to the time dimension to obtain a set of stop nodes during vehicle travel, and a stop line prediction set based on the vehicle trajectory is further fitted, and the initial prediction value of the stop line is updated according to the set. That is, the computer device determines the initial stop line topology data based on the preliminary object recognition result; spatially clusters the target tracking result information to obtain spatial clustering data; fits the vehicle stop trajectory based on the spatial clustering data to obtain the vehicle stop line prediction set; updates the initial stop line topology data based on the vehicle stop line prediction set to obtain the stop line optimization data.

[0088] In another example, the road topology optimization data includes lane trajectory optimization data. In this process, an adjacency matrix A with all initial elements being 0 is constructed based on the semantic segmentation image of the intersection scene and the optimized road marking information. mn , based on the vehicle's four-dimensional trajectory set The lanes that can be connected p i With q j The corresponding matrix element a ij Updated to 1, and finally obtain the adjacency matrix describing the topological relationship between lanes in this road section That is, the computer device generates initial lane trajectory data based on the preliminary object recognition results; generates a lane connection matrix based on the target tracking results, and the lane connection matrix is ​​used to characterize the connectivity status of the lane; and updates the initial lane trajectory information based on the lane connection matrix to obtain lane trajectory optimization data.

[0089] In another example, the road topology optimization data includes pedestrian crossing optimization data. In the modification process, an adjacency matrix A with all initial elements 0 is constructed based on the semantic segmentation image of the intersection scene and the optimized road marking information. mn , based on the vehicle's four-dimensional trajectory set The lanes that can be connected p i With q j The corresponding matrix element a ij Updated to 1, and finally obtain the adjacency matrix describing the topological relationship between lanes in this road section That is, the computer device generates initial crosswalk data based on the preliminary object recognition result; generates pedestrian information based on the target tracking result; performs cluster fitting processing on the pedestrian information to obtain pedestrian trajectory information; updates the initial crosswalk data through the pedestrian trajectory information to obtain optimized crosswalk data.

[0090] In summary, the methods provided by various embodiments of the present invention establish the road topology structure of the intersection by using the roadside camera at the intersection, and further optimize the generated results by using the four-dimensional trajectory information of the vehicle and the trajectory information of the pedestrian. By collecting data using the roadside camera and constructing the road topology structure at the intersection by computer vision methods, it can help vehicles understand the traffic information of the roadside more accurately, achieve higher annotation accuracy, and save the human resources required for manual annotation.

[0091] The above are only optional embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating road topology structure based on roadside camera images, characterized in that: The method is applied in a computer device, and the method comprises: Acquire image data and vehicle data of a target intersection within a target time period, wherein the image data is used to indicate video content of the target intersection within the target time period, and the vehicle data is used to indicate a driving status of vehicles passing through the target intersection within the target time period; Inputting the image data into the semantic segmentation model, and outputting a preliminary object recognition result corresponding to the image data; Inputting the vehicle data and the image data into a multi-target tracking network, and outputting target tracking result information corresponding to the target intersection; In combination with the preliminary object recognition result and the target tracking result information, road topology structure optimization data is generated, and the road topology structure optimization data is used to guide the road topology structure update.

2. The method for generating road topology structure based on roadside camera images according to claim 1, characterized in that: The semantic segmentation model is an artificial intelligence model based on semantic information extraction and semantic segmentation; The step of inputting the image data into the semantic segmentation model and outputting a preliminary object recognition result corresponding to the image data includes: Inputting the image data into the semantic segmentation model; Extracting features from the image data using the semantic segmentation model to obtain high-dimensional image features; Mapping the high-dimensional image features through the semantic segmentation model to obtain road end features; Semantic segmentation is performed on the road end features, and a preliminary result of an object corresponding to the image data is outputted.

3. The method for generating road topology structure based on roadside camera images according to claim 2, characterized in that: The preliminary object result includes at least one of a lane line recognition result, a stop line recognition result, a crosswalk recognition result, a road sign recognition result, and an obstacle recognition result.

4. The method for generating road topology structure based on roadside camera images according to claim 1, characterized in that: The image data is multi-view image data, and the multi-view image data corresponds to at least two image perspectives; The step of inputting the vehicle data and the image data into a multi-target tracking network and outputting target tracking result information corresponding to the target intersection includes: Performing data stream synchronization and sampling processing on the image data to obtain a data stream corresponding to a sampling length in the sampling processing process, wherein the data stream includes road end images corresponding to the number of image viewing angles; Determine a feature identification information set corresponding to the target intersection based on the road end image; The feature identification information set is input into the multi-target tracking network to obtain target tracking result information corresponding to the target intersection.

5. The method for generating road topology structure based on roadside camera images according to claim 4, characterized in that: The step of inputting the feature identification information set into the multi-target tracking network to obtain target tracking result information corresponding to the target intersection includes: The feature identification information set and road features are input into the multi-target tracking network to obtain target tracking result information corresponding to the target intersection, wherein the road features include road center point features, road deflection angle information and road boundary information.

6. The method for generating road topology structure based on roadside camera images according to claim 1, characterized in that: The road topology optimization data includes lane line optimization data; The generating of road topology optimization data by combining the preliminary object recognition result and the target tracking result information includes: Determining initial lane line topology data based on the preliminary recognition result of the object; Performing spatial clustering on the target tracking result information to obtain spatial clustering data; Determining overlap count data based on the spatial clustering data; The initial lane line topology data is adjusted based on the overlap number data to obtain lane line optimization data.

7. The method for generating road topology structure based on roadside camera images according to claim 1, characterized in that: The road topology optimization data includes stop line optimization data; The generating of road topology optimization data by combining the preliminary object recognition result and the target tracking result information includes: Determining initial parking line topology data based on the preliminary object recognition result; Performing spatial clustering on the target tracking result information to obtain spatial clustering data; Performing vehicle stop trajectory fitting based on the spatial clustering data to obtain a vehicle stop line prediction set; The initial stop line topology data is updated based on the vehicle stop line prediction set to obtain the stop line optimization data.

8. The method for generating road topology structure based on roadside camera images according to claim 1, characterized in that: The road topology optimization data includes lane trajectory optimization data; The generating of road topology optimization data by combining the preliminary object recognition result and the target tracking result information includes: generating initial lane trajectory data based on the preliminary object recognition result; generating a lane connection matrix based on the target tracking result, wherein the lane connection matrix is ​​used to characterize the connectivity state of the lane; The initial lane trajectory information is updated based on the lane connection matrix to obtain the lane trajectory optimization data.

9. The method for generating road topology structure based on roadside camera images according to claim 1, characterized in that: The road topology optimization data includes pedestrian crossing optimization data; The generating of road topology optimization data by combining the preliminary object recognition result and the target tracking result information includes: generating initial crosswalk data based on the preliminary recognition result of the object; Generate pedestrian information based on the target tracking result; Performing cluster fitting processing on the pedestrian information to obtain pedestrian trajectory information; The initial crosswalk data is updated using the pedestrian trajectory information to obtain the optimized crosswalk data.

Images