Intersection reference line generation method, medium, equipment and vehicle

Abstract

The invention discloses an intersection reference line generation method, a medium, equipment and a vehicle, and the method comprises the steps: mining self-vehicle passing data fragments related to an intersection scene in full-scene data to form a self-vehicle passing data fragment set, and carrying out the defect data filtering and abnormal data filtering of the self-vehicle passing data fragment set, the full-scene data is recorded real driving data; taking scene data in the self-vehicle passing data fragment set before the self-vehicle passes through the intersection as training data, taking a real track in the self-vehicle passing data fragment set when the self-vehicle passes through the intersection as a training label, and constructing a training set according to the training data and the training label; training a preset intersection reference line generation model by using the training set, wherein the intersection reference line generation model is created based on a deep learning model; and deploying the trained intersection reference line generation model in an automatic driving system to generate an intersection reference line. In this way, the intersection reference line can be accurately generated in various driving environments.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a method, medium, device and vehicle for generating intersection reference lines. Background Art

[0002] The traditional rule-based reference line generation method is completed through computer vision and image processing technology. In the process of generating reference lines, this method relies on a series of predefined geometric rules and heuristic algorithms to identify and connect road markings. Therefore, it may lack sufficient flexibility and accuracy in complex and changeable road environments. In addition, the reliance on existing data makes the traditional reference line generation method unable to be updated and improved according to new data, making it difficult to apply to various traffic environments and having poor versatility. Summary of the invention

[0003] The present application mainly provides a method, medium, device and vehicle for generating intersection reference lines to solve the problems of poor flexibility and versatility in generating intersection reference lines and low accuracy.

[0004] In order to solve the above technical problems, a technical solution adopted in the present application is: providing a method for generating intersection reference lines based on a deep learning model, including: mining the self-vehicle traffic data segments related to the intersection scene in the full-scene data to form a self-vehicle traffic data segment set, and filtering defective data and abnormal data for the self-vehicle traffic data segment set; the full-scene data is the recorded real driving data; the scene data before the self-vehicle passes through the intersection in the self-vehicle traffic data segment set is used as training data, and the real trajectory of the self-vehicle when passing the intersection in the self-vehicle traffic data segment set is used as training labels, and a training set is constructed with the training data and the training labels; a preset intersection reference line generation model is trained with the training set, and the intersection reference line generation model is created based on the deep learning model; the trained intersection reference line generation model is deployed in the autonomous driving system to generate intersection reference lines.

[0005] By mining and filtering real driving data, training data suitable for the intersection reference line generation model is formed, and the intersection reference line generation model is trained based on the actual trajectory of the vehicle when passing through the intersection, thereby improving the accuracy of the intersection reference line generation model in generating reference lines, so that the vehicle can perform automatic driving according to accurate reference lines.

[0006] In some embodiments, the mining of self-vehicle traffic data segments related to intersection scenes in the full-scene data includes: determining the self-vehicle traffic data related to intersection scenes in the full-scene data based on a semantic map; slicing the self-vehicle traffic data to form the self-vehicle traffic data segments suitable for the training requirements of the intersection reference line generation model.

[0007] By slicing the self-driving vehicle traffic data into a length suitable for the training of the intersection reference line generation model, the self-driving vehicle traffic data can be efficiently processed and stored by the intersection reference line generation model, which is conducive to improving the speed of data processing and model training.

[0008] In some embodiments, the defective data filtering and abnormal data filtering of the self-vehicle passage data segment set include: filtering out the self-vehicle passage data segments in the self-vehicle passage data segment set in which navigation data is missing, positioning data is missing, or positioning is offset; filtering out the self-vehicle passage data segments in the self-vehicle passage data segment set in which the self-vehicle is stationary or parked for a long time.

[0009] Defective data filtering is used to remove incomplete and unusable data in the self-vehicle traffic data segment set, and abnormal data filtering is used to remove data that has no reference value in the self-vehicle traffic data segment set, thereby improving the quality of the self-vehicle traffic data segment set.

[0010] In some embodiments, determining the vehicle traffic data related to the intersection scene in the full-scene data based on the semantic map includes: perceiving the area where the stop line field in the semantic map is assigned to determine the intersection scene-related area in the semantic map; cropping the vehicle traffic data corresponding to the intersection scene-related area from the full-scene data to obtain the vehicle traffic data segment related to the intersection scene in the full-scene data.

[0011] By perceiving the field information of the semantic map, the area where the intersection is located is determined, so as to accurately capture the vehicle traffic data fragments related to the intersection scene in the full-scene data, avoid the interference of redundant data on the model training process, improve the speed of model processing data, and thus improve the training efficiency of the intersection reference line generation model.

[0012] In some embodiments, after filtering defective data and abnormal data on the vehicle passage data segment set, it also includes: performing scene balancing processing on the vehicle passage data segment set to balance the data proportion of each type of intersection in the vehicle passage data segment set.

[0013] By balancing the data proportions of various types of intersections in the self-driving traffic data fragment set, the intersection reference line generation model can learn the traffic characteristics of different intersections more comprehensively. At the same time, balancing the proportions of various types of data can also help reduce the model's insufficient learning of a certain type of intersection, thereby improving the generalization ability of the intersection reference line generation model in practical applications.

[0014] In some embodiments, the intersection reference line generation model includes a convolutional network downsampling layer and a linear network layer; the training of the preset intersection reference line generation model with the training set includes: inputting the training data into the convolutional network downsampling layer for information encoding and then extracting deep features; inputting the extracted deep features into the linear network layer for linear transformation to obtain the intersection reference line output by the intersection reference line generation model; calculating the error between the intersection reference line output by the intersection reference line generation model and the real trajectory in the training set; and correcting the intersection reference line generation model based on the error until the error between the real trajectory and the intersection reference line is less than a preset error threshold.

[0015] Extracting features from input information through the convolutional network downsampling layer helps to extract deep semantic information, thereby achieving more accurate positioning and identification of information. The extracted deep features are transformed into predicted reference line outputs through the linear network layer, which reduces the computational complexity and time cost and makes the output data easier to understand. By calculating the error between the intersection reference line output by the intersection reference line generation model and the actual path position within the future preset distance of the autonomous driving vehicle in the training set, the intersection reference line generation model is corrected, allowing the model to perform well under various complex road conditions, effectively improving the generalization ability of the model and the accuracy of reference line prediction.

[0016] In some embodiments, the vehicle traffic data segment concentrates on scene data before the vehicle passes through the intersection, including: feature information of the perception map, motion status information of traffic participants, navigation map information and stationary obstacle position information.

[0017] By inputting the characteristic information of the perception map, the motion status information of traffic participants, the navigation map information and the position information of stationary obstacles into the intersection reference line generation model, the intersection reference line generation model predicts the future driving path of the vehicle based on the input information and generates a reference line.

[0018] In some embodiments, the trained intersection reference line generation model is deployed in an autonomous driving system to generate intersection reference lines, including: serializing the trained intersection reference line generation model into binary model data through an inference engine; and deploying the binary model data to the autonomous driving system through a programming language.

[0019] By serializing the intersection reference line generation model into binary model data and then deploying it, it can directly correspond to machine code. When parsing, there is no need for complex syntax and format parsing like text format, which helps to greatly improve the parsing speed and speed up the model's reasoning speed.

[0020] In some embodiments, the intersection reference line generation method further includes: in response to the distance between the ego vehicle and the intersection stop line in front being less than a preset first constraint distance, generating the intersection reference line on the lane centerline of the lane where the ego vehicle is located.

[0021] By constraining the vehicle to be centered on the reference line when entering the intersection, the vehicle can be ensured to be in the center when entering the intersection, which helps reduce the risk of lane deviation and helps the autonomous driving system to better control the vehicle to perform driving actions.

[0022] In some embodiments, the intersection reference line generation method further includes: in response to the distance between the ego vehicle and the front lane being less than a preset second constraint distance, generating the intersection reference line on the lane centerline of the ego vehicle's target lane.

[0023] By constraining the vehicle to be centered on the reference line when entering the lane, the vehicle can be ensured to be centered when re-entering the intersection, which helps reduce the risk of lane deviation and helps the autonomous driving system to better control the vehicle to perform driving actions.

[0024] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the intersection reference line generation method as described above are implemented.

[0025] The beneficial effects of the storage medium refer to the beneficial effects of the above-mentioned intersection reference line generation method, which will not be repeated here.

[0026] The present application also provides a vehicle-mounted device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the intersection reference line generation method as described above are implemented.

[0027] The vehicle-mounted device is a computer-type device. The beneficial effects of the vehicle-mounted device refer to the beneficial effects of the above-mentioned intersection reference line generation method, which will not be repeated here.

[0028] The present application also provides an autonomous driving vehicle, including the vehicle-mounted device as described above. The autonomous driving vehicle can complete the steps of the above-mentioned intersection reference line generation method through the control of the vehicle-mounted device during driving.

[0029] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a method, medium, device and vehicle for generating intersection reference lines. By mining the self-vehicle traffic data segments related to the intersection scene in the full-scene data to form a self-vehicle traffic data segment set, and filtering the defective data and abnormal data in the self-vehicle traffic data segment set, the quality of the self-vehicle traffic data is optimized, thereby improving the reference line generation accuracy of the intersection reference line generation model; and the full-scene data is the recorded real driving data. The scene data before the self-vehicle passes through the intersection in the self-vehicle traffic data segment set is used as training data, and the real trajectory of the self-vehicle when passing the intersection in the self-vehicle traffic data segment set is used as training labels, and a training set is constructed with the training data and the training labels, and the preset intersection reference line generation model is trained, which is conducive to improving the accuracy of the reference line generation model created based on the deep learning model in generating intersection reference lines; the intersection reference line generation model that has completed the training is deployed in the autonomous driving system to generate intersection reference lines, so that the autonomous driving vehicle can efficiently and accurately obtain the intersection reference line during driving to assist driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0031] Figure 1 It is a flowchart of an embodiment of a method for generating a reference line at an intersection based on a deep learning model provided by the present application;

[0032] Figure 2 Yes Figure 1 The flowchart of step 10 of the method for generating a reference line for an intersection is shown as an embodiment;

[0033] Figure 3 Yes Figure 2 The schematic diagram of the flow chart of the embodiment of the step 11 of the method for generating the intersection reference line shown;

[0034] Figure 4 Yes Figure 1 The flowchart of step 10 of another embodiment of the method for generating a reference line for an intersection is shown;

[0035] Figure 5 Yes Figure 1 The flowchart of step 30 of the method for generating a reference line for an intersection is shown as an embodiment;

[0036] Figure 6 Yes Figure 1The schematic diagram of the flow chart of the embodiment of the step 40 of the method for generating the intersection reference line shown;

[0037] Figure 7 Yes Figure 1 A schematic flow chart of another embodiment of a method for generating a reference line for an intersection is shown;

[0038] Figure 8 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0039] Fig. 9 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application;

[0040] Fig.10 It is a structural schematic diagram of an embodiment of an autonomous driving vehicle provided by the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0043] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] See also Figure 1 , Figure 1: is a flow chart of an embodiment of a method for generating a reference line at an intersection based on a deep learning model provided by the present application, and the method for generating a reference line at an intersection includes:

[0045] Step 10: Mining the self-driving vehicle traffic data segments related to the intersection scene in the full scene data to form a self-driving vehicle traffic data segment set, and filtering defective data and abnormal data on the self-driving vehicle traffic data segment set.

[0046] By recording massive amounts of real driving data of drivers on the road and integrating them into full-scene data covering various driving scenarios, the self-vehicle traffic data segments related to intersection scenarios are mined from the full-scene data, and a self-vehicle traffic data segment set is constructed using multiple self-vehicle traffic data segments mined from the full-scene data. After filtering defective data and abnormal data from the self-vehicle traffic data segment set, the self-vehicle traffic data segment set is used for subsequent model training.

[0047] Full-scene data is the recorded real driving data. Full-scene data includes lane data, intersection data, traffic participant data, and self-driving data collected by various sensors such as cameras, lidars, or locators when the vehicle is driving on the road. The intersection-related data is the part of the full-scene data collected from the time the vehicle drives to the intersection to the time it leaves the intersection, including vehicle trajectories, road signs, obstacles, and other traffic participant dynamics at the intersection.

[0048] Further, see Figure 2 , mining the traffic data segments related to the intersection scene in the full scene data, including:

[0049] Step 11: Determine the vehicle traffic data related to the intersection scene in the full scene data based on the semantic map.

[0050] The actual position of the vehicle is matched with the corresponding position in the semantic map through a map matching algorithm such as global satellite positioning or Beidou satellite positioning to determine the intersection position of the vehicle; the geometric features of the intersection such as the shape of the intersection, the lane layout and the semantic features of the intersection such as the status of the traffic light or the content of the traffic sign are extracted from the semantic map; according to the characteristics of the intersection and the traffic rules, the intersection scenes are classified into: through intersections, crossroads, roundabouts, etc.; according to the classification results, the self-vehicle traffic data fragments related to the current intersection scene are extracted from the full scene data, including various information when the vehicle approaches, passes and leaves the intersection as the self-vehicle traffic data related to the intersection scene.

[0051] Semantic Map A semantic map is a high-precision map that contains rich semantic information. It contains not only geographical features such as road networks and traffic signs, but also the semantic attributes of these features, such as road type, number of lanes, road signs, traffic light status, etc. Semantic maps provide the autonomous driving system with detailed information about the static driving environment, which helps the deep learning model understand the surrounding environment more accurately.

[0052] Based on the semantic map, the vehicle traffic data related to the intersection scene in the full scene data can be accurately determined, so that the deep learning model can be trained with higher precision, which helps to improve the safety and reliability of the autonomous driving system.

[0053] Further, see Figure 3 , based on the semantic map, determining the vehicle traffic data related to the intersection scene in the full scene data, including:

[0054] Step 111: perceive the area to which the stop line field in the semantic map is assigned, so as to determine the intersection scene-related area in the semantic map.

[0055] When the vehicle is near or in an intersection, the stop line field of the semantic map will be assigned a value because the vehicle stop line is detected. When the stop line field in the semantic map is perceived to be assigned a value, the relevant area of ​​the intersection scene can be further determined through the relative position and connection relationship between the stop line and other elements in the semantic map.

[0056] Stop lines are a type of road traffic markings that indicate that vehicles must stop and observe or wait when encountering the line to ensure traffic safety. In semantic maps, stop lines are identified and marked as important road elements. Their role is to clearly define the parking limits of vehicles while driving and accurately identify intersection scenes.

[0057] The stop line field is used to store stop line related information, such as location, direction, length, and width. When the vehicle-mounted sensor collects stop line information, it uses deep learning and other algorithms to perform image processing, clarify road features from stop line related information, perform semantic annotation, accurately mark its location, direction, length and other information on the map, and assign corresponding values ​​to the stop line field.

[0058] By perceiving the area where the stop line field in the semantic map is assigned a value and analyzing it in combination with the spatial relationship of other road elements in the semantic map, the intersection scene-related area in the semantic map can be accurately determined.

[0059] Step 112: The vehicle traffic data corresponding to the intersection scene-related area is cut out from the full scene data to obtain the vehicle traffic data segment related to the intersection scene in the full scene data.

[0060] Crop along the identified intersection area boundary to remove data in areas irrelevant to the intersection scene in the full scene data, and only retain the vehicle traffic data corresponding to the intersection scene-related area as the vehicle traffic data segment related to the intersection scene in the full scene data.

[0061] By perceiving the field information of the semantic map and determining the area where the intersection is located, the vehicle traffic data fragments related to the intersection scene in the full-scene data can be accurately captured, which helps to reduce the amount of data that the model needs to process and learn at the same time and improve the speed at which the model processes data, thereby improving the training efficiency of the intersection reference line generation model.

[0062] Step 12: Slice the self-driving vehicle traffic data to form self-driving vehicle traffic data segments suitable for the training requirements of the intersection reference line generation model.

[0063] According to the training requirements of the intersection reference line generation model, the self-driving vehicle traffic data is sliced ​​and divided into multiple segments of lengths suitable for the training of the intersection reference line generation model, thereby forming multiple segments of self-driving vehicle traffic data with lengths suitable for the training of the intersection reference line generation model.

[0064] For example, the training length suitable for the intersection reference line generation model is 10 minutes. In a 65-minute self-driving car traffic data segment, it is necessary to slice out multiple 10-minute self-driving car traffic data segments. After slicing, 7 self-driving car traffic data segments that meet the requirements can be obtained.

[0065] Optionally, the slices of the vehicle traffic data segment have time intervals.

[0066] For example, if the slicing time interval is set to 5 minutes, a 65-minute self-driving traffic data can be sliced ​​into 0-10 minute segments, 15-25 minute segments, 30-40 minute segments, 45-55 minute segments and 60-65 minute segments.

[0067] By slicing the self-driving vehicle traffic data into a length suitable for the training of the intersection reference line generation model, the self-driving vehicle traffic data can be efficiently processed and stored by the intersection reference line generation model, which is conducive to improving the speed of data processing and model training.

[0068] Optionally, see Figure 4 Step 10 performs defective data filtering and abnormal data filtering on the vehicle traffic data segment set, and further includes:

[0069] Step 13: Filter out the self-driving vehicle traffic data segments with missing navigation data, missing positioning data or positioning offset in the self-driving vehicle traffic data segment set.

[0070] Traverse the self-driving vehicle traffic data fragment set to check each self-driving vehicle traffic data fragment, check whether each data fragment contains the necessary navigation data, whether the positioning data in the data fragment is complete and continuous, and analyze whether there is an abnormal offset in the positioning data; and mark the defective data that does not meet the requirements in the self-driving vehicle traffic data fragment set according to the inspection results, remove it from the self-driving vehicle traffic data fragment set, and only retain the filtered self-driving vehicle traffic data fragment set.

[0071] By removing incomplete and unusable data in the self-driving vehicle traffic data segment set through defective data filtering, the quality of the self-driving vehicle traffic data segment set is improved, which is conducive to improving the accuracy and reliability of data analysis of the intersection reference line generation model.

[0072] Step 14: Filter out the vehicle traffic data segments in which the vehicle is stationary or parked for a long time in the vehicle traffic data segment set.

[0073] Traverse the self-vehicle travel data segment set to check each self-vehicle travel data segment, check whether each data segment contains data of the vehicle being stationary or parked for a long time, mark and filter out those data segments containing the self-vehicle being stationary or parked for a long time.

[0074] By filtering defective data to remove data that has no reference value in the self-driving vehicle traffic data segment set, the quality of the self-driving vehicle traffic data segment set is improved, which is beneficial to improving the accuracy and reliability of data analysis of the intersection reference line generation model.

[0075] Optionally, after filtering defective data and abnormal data on the vehicle traffic data segment set, the method further includes: performing scene balancing processing on the vehicle traffic data segment set to balance the data proportion of each type of intersection in the vehicle traffic data segment set.

[0076] The intersection scenes in the self-driving data segment set are classified, such as straight-ahead and left-right turns at intersections, straight-ahead at non-center intersections, or complex intersections; statistical analysis is performed on the data of each type of scene, and the proportion of each scene data in the self-driving data segment set is calculated to identify which scenes have too high or too low data proportions. For scenes with a low data proportion, such as straight-ahead at non-center intersections or complex intersections, the data proportion can be increased by increasing the sampling frequency of these scenes or synthesizing new samples. For scenes with a high data proportion, such as straight-ahead and left-right turns at intersections, the data proportion can be reduced by reducing the sampling volume of these scenes, until the data proportion of each type of intersection in the self-driving data segment set is balanced.

[0077] By balancing the data proportions of various types of intersections in the self-driving traffic data fragment set, the intersection reference line generation model can learn the traffic characteristics of different intersections more comprehensively. At the same time, balancing the proportions of various types of data can also help reduce the model's insufficient learning of a certain type of intersection, thereby improving the generalization ability of the intersection reference line generation model in practical applications.

[0078] Step 20: Use the scene data before the vehicle passes through the intersection in the vehicle traffic data segment set as training data, use the actual trajectory of the vehicle when passing through the intersection in the vehicle traffic data segment set as training labels, and construct a training set with the training data and training labels.

[0079] The scene data before the vehicle passes through the intersection extracted from the vehicle traffic data fragment set is used as training data, including road information, traffic participant motion status, driving trajectory, obstacles and signs in the scene; the actual trajectory of the vehicle when passing through the intersection in the vehicle traffic data fragment set is used as training labels, so that the intersection reference line generation model can generate a reference line close to the actual trajectory of the vehicle after training; the training set is constructed with the training data and training labels to train the intersection reference line generation model, so as to improve the generation accuracy of the intersection reference line generation model.

[0080] Constructing a training set with the training data and the training labels, and training the preset intersection reference line generation model, is conducive to improving the accuracy of the intersection reference line generation model created based on the deep learning model in generating reference lines. A large amount of training data can help the intersection reference line generation model adapt to various road conditions and driving environments, making the intersection reference line generation model more flexible and versatile.

[0081] Specifically, the vehicle traffic data segment concentrates on the scene data before the vehicle passes through the intersection, including: feature information of the perception map, motion state information of traffic participants, navigation map information and position information of stationary obstacles.

[0082] Among them, the characteristic information of the perception map includes: road structure information such as road type, number of lanes, lane width, lane line type, intersection type; traffic signs and markings such as stop lines, pedestrian crossings, speed limit signs, turn arrows, etc.; road conditions such as wet, waterlogged, potholes, construction areas, etc.

[0083] The motion status information of traffic participants includes: the motion status of other vehicles such as the vehicle's position, speed, acceleration, driving direction, turning intention, relative position to the vehicle, etc.; the position, moving speed, moving direction, whether crossing the road, etc. of pedestrians; the position and trajectory of non-motor vehicles, etc.

[0084] Navigation map information includes: route planning information such as the route the vehicle should travel, the next turning point, the estimated arrival time, etc.; road restriction information such as one-way streets, prohibited turns, height limits, weight limits and other restrictions; real-time traffic conditions such as congestion, accident reports, construction information, etc.

[0085] The location information of static obstacles includes: fixed obstacles such as trees, lamp posts, buildings, parked vehicles, etc. on both sides of the road; construction areas such as fences, construction machinery, building materials, etc. These obstacles may occupy part or all of the lanes and affect the driving of the vehicle; other static information such as road edges, sidewalk boundaries, green belts, etc. This information helps the vehicle understand the overall layout of the road and the available space, so as to reasonably design the driving route of the vehicle at the intersection.

[0086] Step 30: Train the preset intersection reference line generation model using the training set.

[0087] The training data and the training set constructed by the training labels are input into the intersection reference line generation model for training.

[0088] The architecture of the deep learning model is constructed according to the training requirements for generating intersection reference lines, the training data and the training set constructed by the training labels are input into the constructed intersection reference line generation model for training, and a suitable loss function is selected to correct the intersection reference line generation model until the intersection reference line generation model meets the training requirements.

[0089] The intersection reference line generation model is created based on a deep learning model that can process image data from car cameras or traffic monitoring cameras and identify key information such as road signs, lane lines, and traffic lights.

[0090] Using semantic segmentation technology, the deep learning model can classify each pixel in the image into different categories such as roads, sidewalks, vehicles, pedestrians, etc., thereby constructing a detailed intersection environment map. Based on the results of image recognition and semantic segmentation, the deep learning model can further combine information such as traffic rules, vehicle dynamics, and real-time traffic conditions to perform path planning.

[0091] In intersection scenarios, deep learning can be used to predict the behavior of other traffic participants and generate one or more feasible driving reference lines. Further, reinforcement learning technology can be introduced. Through actual driving data, the reinforcement learning model can continuously optimize the generation strategy of intersection reference lines to adapt to different traffic scenarios and conditions.

[0092] Specifically, the intersection reference line generation model includes a convolutional network downsampling layer and a linear network layer.

[0093] The convolutional network downsampling layer constructs a deep network by stacking multiple convolutional layers, activation layers, and pooling layers to extract rich feature representations from the training data input to the intersection reference line generation model.

[0094] The linear network layer is located after the convolutional network and is used to integrate the features extracted by the convolutional layer and output the final prediction results. The linear network layer can convert the features extracted by the convolutional network into the coordinates of the reference line or other representations.

[0095] Further, see Figure 5 , training a preset intersection reference line generation model with the training set, including:

[0096] Step 31: Input the training data into the convolutional network downsampling layer for information encoding and then extract deep features.

[0097] After the feature information of the perception map, the motion status information of traffic participants, the navigation map information and the position information of stationary obstacles are input into the convolutional network downsampling layer, the convolutional network downsampling layer reduces the dimension of the feature information through the pooling operation of the downsampling layer, thereby reducing the amount of calculation and extracting more abstract features.

[0098] The convolutional network downsampling layer is a structure used to reduce the spatial dimension in the convolutional network. By reducing the spatial dimension of the feature, the amount of calculation of the subsequent convolutional layer can be reduced, making the feature extraction of the convolutional network more efficient.

[0099] The convolutional network downsampling layer reduces the spatial size of the data by aggregating information from adjacent regions in the input feature map. This aggregation operation can be viewed as a feature encoding because it combines multiple local features into a more representative feature.

[0100] By stacking multiple convolutional network downsampling layers and convolutional layers, the convolutional network can gradually extract the deep features of the input data. These features are usually more abstract and representative than the original data, and can better reflect the inherent laws and patterns of the data.

[0101] Extracting features from input information through the convolutional network downsampling layer helps to extract deep semantic information, thereby achieving more accurate positioning and identification of information.

[0102] Step 32: Input the extracted deep features into the linear network layer for linear transformation to obtain the intersection reference line output by the intersection reference line generation model.

[0103] The deep features extracted by the convolutional network downsampling layer are input into the linear network layer for linear transformation to form the vector of the reference line as the output of the intersection reference line generation model.

[0104] The linear network layer is used to integrate and abstract the features extracted by the previous layer, or by adjusting the number of neurons, the dimension of the feature vector can be changed to adapt it to the requirements of subsequent layers or tasks.

[0105] For example, the training label of the model is defined as the real trajectory of the vehicle in the next 100 meters, that is, 100 points are defined at intervals of 1 meter. After feature extraction and linear transformation, the output vector of 100x2 represents the position prediction of the intersection reference line with 100 points at intervals of 1 meter. Each point vector includes x-coordinates and y-coordinates. Starting from (0,0), the x-coordinate increases by 1 meter, and the y-coordinate remains unchanged. It contains a matrix of 100 rows and 2 columns, so the output vector is 100x2.

[0106] The extracted deep features are transformed into predicted reference line outputs through linear network layers, which reduces the computational complexity and time cost and makes the output data easier to understand.

[0107] Step 33: Calculate the error between the intersection reference line output by the intersection reference line generation model and the actual trajectory in the training set.

[0108] The error of the distance between the intersection reference line output by the intersection reference line generation model and each corresponding point of the actual trajectory is calculated, such as the mean square error or the mean absolute error, so as to quantify the difference between the intersection reference line output by the model and the actual trajectory, and realize the evaluation of the performance of the intersection reference line generation model.

[0109] Step 34: Based on the error, the intersection reference line generation model is modified until the error between the actual trajectory and the intersection reference line is less than a preset error threshold.

[0110] According to the training requirements, an error threshold is preset as a standard for evaluating the performance of the intersection reference line generation model. The error between the intersection reference line and the true trajectory is used to guide the update direction of the intersection reference line generation model parameters by calculating the gradient. After multiple training updates, the error between the true trajectory and the intersection reference line is less than the preset error threshold. At this time, the intersection reference line generation model performance is qualified and training is stopped.

[0111] For example, the preset error threshold is 0.08, and the calculated mean square error of the distance between the intersection reference line and each corresponding point of the true trajectory is 0.26. The parameters of the intersection reference line generation model are updated by calculating the gradient until the mean square error of the distance between the intersection reference line and the true trajectory is less than or equal to 0.08. At this time, the performance of the intersection reference line generation model is qualified and training is stopped.

[0112] By calculating the error between the intersection reference line output by the intersection reference line generation model and the actual trajectory of the autonomous driving vehicle within a preset distance in the future in the training set, and correcting the intersection reference line generation model, the model can perform well under various complex road conditions, effectively improving the generalization ability of the model and the accuracy of reference line prediction.

[0113] Step 40: Deploy the trained intersection reference line generation model in the autonomous driving system to generate intersection reference lines.

[0114] The trained model is encapsulated into a deployable component or service, and a clear API interface is provided for the autonomous driving system to call. The autonomous driving system calls the trained intersection reference line generation model through the API interface to generate the intersection reference line.

[0115] Specifically, see Figure 6 , deploying the trained intersection reference line generation model in the autonomous driving system to generate intersection reference lines, including:

[0116] Step 41: Serialize the trained intersection reference line generation model into binary model data through the inference engine.

[0117] The binary model is exported from the training framework of the intersection reference line generation model, and the trained intersection reference line generation model is converted into a format more suitable for deployment in the autonomous driving system through the inference engine, forming binary model data with smaller size, faster loading speed and higher operating efficiency.

[0118] The inference engine is responsible for executing the model's inference task, that is, processing the real-time road information in the autonomous driving system through the loaded intersection reference line generation model and outputting the corresponding classification results. The inference engine is a bridge between the intersection reference line generation model and the actual application scenario, which enables the trained intersection reference line generation model to perform efficient inference on different hardware and software platforms.

[0119] Binary model data is intersection reference line generation model data saved in binary format, which is more compact and efficient than text or source code format and can be directly read by a machine. It only stores the weights, structure and other necessary information of the model without including any human-readable tags or annotations.

[0120] The intersection reference line generation model is serialized into binary model data, removing all unnecessary formatting and tags, saving resource space on the device where the autonomous driving system is located; binary model data can be directly processed quickly by the computer's memory and processor, so loading binary model data is usually much faster than parsing text or source code formats, helping to reduce delays during model inference; binary model data can maintain a consistent format on different operating systems and hardware architectures, and can be easily deployed and run in multiple environments without additional conversion or modification, making it more cross-platform compatible.

[0121] Step 42: Deploy the binary model data to the autonomous driving system via a programming language.

[0122] After using the C++ API provided by the inference engine to load the binary model file, the road information data collected by the autonomous driving system, such as camera images, is preprocessed into the format expected by the binary model. The preprocessed data is then passed to the inference engine to perform inference and obtain the output results, completing the deployment of the binary model data on the autonomous driving system.

[0123] Autonomous driving vehicles can generate intersection reference lines through binary model data in the autonomous driving system, guiding the autonomous driving vehicle to pass through the intersection along the accurate route.

[0124] Optionally, see Figure 7 , Figure 7 Yes Figure 1 The flowchart of another embodiment of the method for generating a reference line for an intersection shown in FIG. 4 may further include the following steps after step 40:

[0125] Step 50: In response to the distance between the ego vehicle and the stop line of the intersection ahead being less than a preset first constraint distance, an intersection reference line is generated on the lane centerline of the lane where the ego vehicle is located.

[0126] The vehicle uses sensors such as cameras to detect the distance between the vehicle and the stop line at the intersection ahead, and estimates the distance through algorithm processing. The detected distance is compared with the preset first constraint distance. If the detected distance is less than the first constraint distance, the centering constraint process of the intersection reference line is triggered, and the intersection reference line is generated on the lane centerline of the lane where the vehicle is located.

[0127] Specifically, the first constraint distance may be 150 m, 130 m, 125 m, 110 m, 100 m, 87 m, 66 m, 52 m or 34 m, etc.

[0128] For example, when the first constraint distance is 80m, the ego vehicle detects that the current distance to the stop line ahead is 85m, and then normally guides the vehicle to change lanes or go straight. When the ego vehicle detects that the current distance to the stop line ahead is 79m, the reference line is forced to be centered on the center line of the lane where the ego vehicle is located.

[0129] By constraining the vehicle to be centered on the reference line when entering the intersection, the vehicle can be ensured to be in the center when entering the intersection, which helps reduce the risk of lane deviation and helps the autonomous driving system to better control the vehicle to perform driving actions.

[0130] Step 60: In response to the distance between the ego vehicle and the front lane being less than a preset second constraint distance, generating an intersection reference line on the lane centerline of the ego vehicle's target lane.

[0131] The vehicle uses sensors such as cameras to detect the distance between the vehicle and the stop line at the intersection ahead, and estimates the distance through algorithm processing. The detected distance is compared with the preset second constraint distance. If the detected distance is less than the second constraint distance, the centering constraint process of the intersection reference line is triggered, and the intersection reference line is generated on the lane centerline of the target lane of the vehicle.

[0132] Specifically, the second constraint distance may be 110 m, 90 m, 87 m, 66 m, 52 m, 37 m, 25 m or 12 m, etc.

[0133] For example, when the second constraint distance is 19m, the vehicle detects that the current distance to the stop line ahead is 25m, and then normally guides the vehicle to perform driving operations such as turning or going straight. When the vehicle detects that the current distance to the stop line ahead is 15m, the reference line is forced to be centered on the center line of the target lane of the vehicle.

[0134] By constraining the vehicle to be centered on the reference line when it enters the lane, it ensures that the vehicle is centered when it re-enters the intersection, which helps reduce the risk of lane deviation and helps the autonomous driving system better control the vehicle to perform driving actions.

[0135] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0136] The storage medium 300 stores program data 301. When the program data 301 is executed by the processor, the following is achieved: Figure 1 The steps of the described method for generating intersection reference lines.

[0137] The program data 301 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0138] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0139] See also Fig. 9 , Fig. 9 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application.

[0140] The vehicle-mounted device 400 is installed in the vehicle and is a computer-type device. It works in conjunction with various sensor modules and control modules in the vehicle to achieve visual perception, sensor data processing, and image display. The vehicle-mounted device 400 includes a processor 420 and a memory 410 that are interconnected. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the above-mentioned intersection reference line generation method are implemented.

[0141] See also Fig.10 , Fig.10 An autonomous driving vehicle 500 is provided, comprising the above-mentioned vehicle-mounted device 400. During driving, the vehicle 500 can complete the process of generating a reference line for an intersection based on a deep learning model through the control of the vehicle-mounted device 400.

[0142] Different from the prior art, the present application provides a method, medium, device and vehicle for generating intersection reference lines. By screening out intersection-related data from real driving data to train an intersection reference line generation model, the intersection reference line generation model can be applied to a variety of different intersection conditions and accurately generate reference lines; the intersection reference line generation model based on deep learning can also be widely used in different driving scenarios by learning new data, and has high versatility. After training, the intersection reference line generation model can accurately generate intersection reference lines in various situations and guide autonomous driving vehicles through intersections.

[0143] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0144] The present application can be used in many general or special vehicle-mounted computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating intersection reference lines based on a deep learning model, characterized in that: include: Mining the self-vehicle traffic data segments related to the intersection scene in the full-scene data to form a self-vehicle traffic data segment set, and filtering defective data and abnormal data on the self-vehicle traffic data segment set, wherein the full-scene data is recorded real driving data; Using the scene data before the vehicle passes through the intersection in the vehicle traffic data segment set as training data, using the real trajectory of the vehicle when passing through the intersection in the vehicle traffic data segment set as training labels, and constructing a training set with the training data and the training labels; Using the training set to train a preset intersection reference line generation model, the intersection reference line generation model is created based on a deep learning model; The trained intersection reference line generation model is deployed in the autonomous driving system to generate intersection reference lines.

2. The method for generating a reference line at an intersection according to claim 1, characterized in that: The mining of the vehicle traffic data segments related to the intersection scene in the full scene data includes: Determine the vehicle traffic data related to the intersection scene in the full scene data based on the semantic map; Slices are made from the self-vehicle traffic data to form self-vehicle traffic data segments suitable for the training requirements of the intersection reference line generation model.

3. The method for generating a reference line at an intersection according to claim 2, characterized in that: The filtering of defective data and abnormal data on the vehicle traffic data segment set includes: Filter out the own-vehicle traffic data segments in which navigation data is missing, positioning data is missing or positioning is offset in the own-vehicle traffic data segment set; The vehicle traffic data segments in which the vehicle is stationary or parked for a long time are filtered out from the vehicle traffic data segment set.

4. The method for generating a junction reference line according to claim 2, characterized in that: The determining, based on the semantic map, the vehicle traffic data related to the intersection scene in the full scene data includes: Perceiving an area in a semantic map where a stop line field is assigned a value, to determine an intersection scene-related area in the semantic map; The vehicle traffic data corresponding to the intersection scene related area is cropped from the full scene data to obtain the vehicle traffic data segment related to the intersection scene in the full scene data.

5. The method for generating a reference line at an intersection according to claim 1, characterized in that: After filtering defective data and abnormal data from the vehicle traffic data segment set, the method further includes: The vehicle traffic data segment set is subjected to scene balancing processing to balance the data proportions of each type of intersection in the vehicle traffic data segment set.

6. The method for generating a junction reference line according to claim 1, characterized in that: The intersection reference line generation model includes a convolutional network downsampling layer and a linear network layer; The step of training a preset intersection reference line generation model with the training set includes: Inputting the training data into the convolutional network downsampling layer for information encoding and then extracting deep features; Inputting the extracted deep features into the linear network layer for linear transformation to obtain the intersection reference line output by the intersection reference line generation model; Calculating the error between the intersection reference line output by the intersection reference line generation model and the real trajectory in the training set; The intersection reference line generation model is modified based on the error until the error between the actual trajectory and the intersection reference line is less than a preset error threshold.

7. The method for generating a junction reference line according to claim 1, characterized in that: The vehicle traffic data fragment contains scene data before the vehicle passes through the intersection, including: Perceive map feature information, traffic participant motion status information, navigation map information, and stationary obstacle location information.

8. The method for generating a reference line at an intersection according to claim 1, characterized in that: The step of deploying the trained intersection reference line generation model in an autonomous driving system to generate an intersection reference line includes: The trained intersection reference line generation model is serialized into binary model data through an inference engine; Deploy the binary model data to the autonomous driving system via a programming language.

9. The method for generating a junction reference line according to claim 1, characterized in that: The intersection reference line generation method further includes: In response to the distance between the ego vehicle and a stop line at an intersection ahead being less than a preset first constraint distance, the intersection reference line is generated on a lane centerline of a lane where the ego vehicle is located.

10. The method for generating a reference line at an intersection according to claim 1 or 9, characterized in that: The intersection reference line generation method further includes: In response to the distance between the ego vehicle and the front lane being less than a preset second constraint distance, the intersection reference line is generated on the lane center line of the ego vehicle target lane.

11. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the abnormal stationary vehicle identification method according to any one of claims 1 to 10 are implemented.

12. A vehicle-mounted device, characterized in that: The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the abnormal stationary vehicle identification method as described in any one of claims 1 to 10 are implemented.

13. An autonomous driving vehicle, characterized in that: Comprising the vehicle-mounted device as claimed in claim 12.

Images