Methods for determining planned trajectories, model training methods, and autonomous vehicles

Abstract

This disclosure provides a method for determining a planned trajectory, relating to the field of artificial intelligence technology, particularly to the fields of autonomous driving, intelligent transportation, and deep learning technology. The specific implementation scheme is as follows: Based on the current scene data of the current vehicle and historical information associated with the current scene data, the characteristics of the current scene are determined; based on the characteristics of the current scene, multiple future trajectories of reference scenes are determined from a reference scene database as candidate trajectories for the current vehicle, the future trajectories including the trajectory information of the reference vehicle within future time periods under the reference scenes; for each candidate trajectory, the characteristics of the candidate trajectory are determined, and based on the characteristics of the current scene and the characteristics of the candidate trajectory, an evaluation value of the candidate trajectory is determined; based on the evaluation value, a target trajectory is determined from the multiple candidate trajectories as the planned trajectory of the current vehicle. This disclosure also provides a training method, apparatus, electronic device, and storage medium for a deep learning model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of autonomous driving, intelligent transportation, and deep learning technology. More specifically, this disclosure provides a method for determining a planned trajectory, a method for training a deep learning model, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] Artificial intelligence-based data processing has been widely applied in various fields. In the field of autonomous driving, artificial intelligence-based data processing can help plan reasonable driving trajectories for vehicles. Summary of the Invention

[0003] This disclosure provides a method for determining a planned trajectory, a method for training a deep learning model, an apparatus, a device, and a storage medium.

[0004] According to the first aspect, a method for determining a planned trajectory is provided, the method comprising: determining the characteristics of the current scene based on the current scene data of the current vehicle and historical information associated with the current scene data; determining the future trajectories of multiple reference scenes from a reference scene database based on the characteristics of the current scene, as candidate trajectories of the current vehicle, wherein the future trajectories include trajectory information of the reference vehicle in future time periods under the reference scenes; determining the characteristics of each candidate trajectory, and determining an evaluation value of the candidate trajectory based on the characteristics of the current scene and the characteristics of the candidate trajectory; and determining a target trajectory from the multiple candidate trajectories based on the evaluation value, as the planned trajectory of the current vehicle in the current scene.

[0005] According to the second aspect, a method for training a deep learning model is provided. The method includes: determining the future trajectories of multiple reference scenes from a reference scene database based on the features of the sample scene, as candidate trajectories for the sample scene, wherein the future trajectories include trajectory information of a reference vehicle in the reference scene within a future time period; inputting the features of the sample scene and the features of the multiple candidate trajectories into a deep learning model to obtain evaluation values ​​for the multiple candidate trajectories; determining the loss of the deep learning model based on the evaluation values; and adjusting the parameters of the deep learning model based on the loss.

[0006] According to a third aspect, an apparatus for determining a planned trajectory is provided. The apparatus includes: a current scene feature determination module, configured to determine the features of the current scene based on current scene data of the current vehicle and historical information associated with the current scene data; a first candidate trajectory determination module, configured to determine the future trajectories of multiple reference scenarios from a reference scene database based on the features of the current scene, as candidate trajectories of the current vehicle, wherein the future trajectories include trajectory information of the reference vehicle in future time periods under the reference scenarios; an evaluation value determination module, configured to determine the features of each candidate trajectory and determine the evaluation value of the candidate trajectory based on the features of the current scene and the features of the candidate trajectory; and a target trajectory determination module, configured to determine the target trajectory from multiple candidate trajectories based on the evaluation value, as the planned trajectory of the current vehicle in the current scene.

[0007] According to the fourth aspect, a training apparatus for a deep learning model is provided, the apparatus comprising: a second candidate trajectory determination module, configured to determine future trajectories of multiple reference scenes from a reference scene database based on features of the sample scene, as candidate trajectories of the sample scene, wherein the future trajectories include trajectory information of a reference vehicle in a future time period under the reference scene; a processing module, configured to input the features of the sample scene and the features of the multiple candidate trajectories into a deep learning model to obtain evaluation values ​​of the multiple candidate trajectories; a loss determination module, configured to determine the loss of the deep learning model based on the evaluation values; and an adjustment module, configured to adjust the parameters of the deep learning model based on the loss.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided in this disclosure.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method provided in this disclosure when executed by a processor.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is an exemplary system architecture diagram illustrating an embodiment of the present disclosure that can be applied to a method for determining a planned trajectory and a training method for a deep learning model.

[0014] Figure 2 This is a flowchart of a method for determining a planned trajectory according to an embodiment of the present disclosure;

[0015] Figure 3 This is a schematic diagram of a method for determining features of a current scene according to an embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram of a method for determining the evaluation value of a candidate trajectory according to an embodiment of the present disclosure;

[0017] Figure 5 This is a schematic diagram of a method for determining the evaluation value of a candidate trajectory according to an embodiment of the present disclosure;

[0018] Figure 6 This is a flowchart of a training method for a deep learning model according to an embodiment of the present disclosure;

[0019] Figure 7 This is a recall curve of a candidate trajectory according to an embodiment of the present disclosure;

[0020] Figure 8 This is a block diagram of an apparatus for determining a planned trajectory according to an embodiment of the present disclosure;

[0021] Figure 9 This is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0022] Figure 10 This is a block diagram of an electronic device for a method of determining a planned trajectory and / or a method of training a deep learning model according to an embodiment of the present disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] Trajectory planning is a crucial component of autonomous driving technology, addressing the question of how autonomous vehicles should proceed. Among related technologies, trajectory retrieval-based autonomous driving trajectory planning methods can be employed.

[0025] For example, scene information during vehicle operation (e.g., information about surrounding obstacles, lane information, etc.) is encoded into scene features. Then, based on these scene features, historical scenes with high similarity to the current scene are retrieved from the database, and the future driving trajectories corresponding to these historical scenes are used as the current planned trajectory.

[0026] This trajectory retrieval-based autonomous driving trajectory planning method can recall hundreds of candidate trajectories from tens of millions of scenarios. However, the calculation of scene similarity in this method is coarse-grained, resulting in low confidence of candidate trajectories recalled based on similarity, and a low recall rate for the optimal trajectory.

[0027] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0029] Figure 1 This is a schematic diagram of an exemplary system architecture according to an embodiment of the present disclosure, illustrating a method for determining a planned trajectory and a training method for a deep learning model. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include an autonomous vehicle 101, a network 102, a server 103, and a database 104. The network 102 serves as a medium for providing a communication link between the autonomous vehicle 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] In one example, the autonomous vehicle 101 can send the scene data collected during the autonomous driving process to the server 103. The server 103 can process the scene data and add the processed scene data as reference scene data to the database 104.

[0032] When performing trajectory planning for autonomous vehicle 101, autonomous vehicle 101 can send the collected current scene data to server 103. Server 103 can query the database for the current scene data of autonomous vehicle 101 to find the top K most similar reference scenes (K is an integer greater than 1, for example, K=100) and use the future driving trajectories of the top K most similar reference scenes as candidate trajectories of autonomous vehicle 101 in the current scene.

[0033] For the Top K candidate trajectories, the SLT (SL stands for Frenet coordinate system, i.e., SL coordinate system, T represents time) rule can be invoked for safety calculation, which calculates whether the autonomous vehicle will collide with the obstacle under the candidate trajectory. If not, the candidate trajectory passes the safety calculation. The first candidate trajectory that passes the SLT safety calculation can be determined as the planned trajectory of autonomous vehicle 101.

[0034] In the SL coordinate system, the S-axis represents the vehicle's direction of travel, and L is a direction perpendicular to the vehicle's direction of travel. Constructing this SL coordinate system facilitates the calculation of whether a collision will occur between the vehicle and an obstacle in space. Furthermore, the SLT rule can also detect, in the time dimension, whether a collision with an obstacle will occur within a preset time period (e.g., 3 seconds) while traveling along a candidate trajectory.

[0035] Since SLT rule calculation is time-consuming, it's necessary to prioritize the optimal trajectory to minimize the number of SLT rule calls. The Top K reference scenarios mentioned above are sorted by scenario similarity; however, this scenario similarity calculation is coarse-grained with low confidence, resulting in a low recall rate for the optimal trajectory (Top 1 trajectory). This leads to an increase in the number of SLT rule calls, further increasing computation time and impacting the trajectory planning efficiency of the autonomous vehicle 101.

[0036] In view of this, this disclosure provides a method for determining a planned trajectory. For a current scenario that requires trajectory planning, the method recalls future trajectories of multiple reference scenarios most similar to the current scenario from a database as candidate trajectories. A fine-grained trajectory ranking model is used to reorder the recalled candidate trajectories to improve the recall rate of the optimal trajectory, thereby improving the trajectory planning efficiency of autonomous vehicles.

[0037] The method for determining the planned trajectory provided in this disclosure can be executed by the autonomous vehicle 101 or by the server 103. Correspondingly, the device for determining the planned trajectory provided in this disclosure can be installed in the autonomous vehicle 101 or in the server 103. The method for training the deep learning model provided in this disclosure can be executed by the server 103. Correspondingly, the device for training the deep learning model provided in this disclosure can be installed in the server 103.

[0038] Figure 2 This is a flowchart of a method for determining a planned trajectory according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the method 200 for determining the planned trajectory includes operations S210 to S240.

[0040] In operation S210, the characteristics of the current scene are determined based on the current scene data of the current vehicle and the historical information associated with the current scene data.

[0041] The current vehicle can be an autonomous vehicle, and the current scene can be the current driving scene or driving environment of the autonomous vehicle. During the driving process, the autonomous vehicle can collect scene data, which can be collected frame by frame, with each frame being 0.1 seconds.

[0042] Each frame of scene data corresponds to a specific moment in time, and the current scene data corresponds to the current moment. Each frame of scene data can include the scene image at the given moment and object information within that scene image. Autonomous vehicles can be referred to as "autonomous vehicles," and objects in the scene image can include the autonomous vehicle, obstacles (pedestrians, motor vehicles, non-motorized vehicles, etc.), and road information. Road information can include lane markings such as the road centerline, stop lines, and pedestrian crossings.

[0043] Each frame of scene data can be associated with historical information from a time period preceding the time in which the scene data is located. For example, the current scene image includes the location information of the vehicle (current vehicle), the current obstacle, and the current road. The current scene data can be associated with the historical trajectory of the vehicle, the historical trajectory of the current obstacle, and the current road information from a historical time period preceding the time in which the current scene data is located (e.g., 1.6 seconds prior).

[0044] Since the historical time period includes multiple historical scenes, the vehicle's historical trajectory within that time period can be composed of the vehicle's position in each historical scene, and the current obstacle's historical trajectory within that time period can be composed of the current obstacle's position in each historical scene. The road information in each historical scene image constitutes a road segment. Multiple historical scene images within a 1.6-second time period can include multiple road segments. Each road segment includes lane line information (e.g., position, color, line type), and the lane lines contained on each road segment are a segment of lane lines, which can be called a lane line fragment. Therefore, the current lane line information within the historical time period includes lane line fragment information from multiple road segments, and these lane line fragments can be stitched together to form the complete lane line information within the historical time period. Lane lines can include road center lines, stop lines, and pedestrian crossings, and lane line fragments can include lane center line fragments, stop line fragments, and pedestrian crossing fragments, etc.

[0045] Based on the current scene data and the historical information associated with the current scene data, the characteristics of the current scene can be determined.

[0046] For example, based on the vehicle's historical trajectory within 1.6 seconds prior to the current moment, the vehicle's characteristics can be determined. Based on the historical trajectory of the current obstacle within that 1.6-second period, the current obstacle's characteristics can be determined. Based on multiple segments of information about the current lane line within that 1.6-second period, the current lane line's characteristics can be determined. Based on the vehicle's characteristics, the current obstacle's characteristics, and the current lane line's characteristics, the characteristics of the current scene can be determined.

[0047] During operation S220, based on the characteristics of the current scene, the future trajectories of multiple reference scenes are determined from the reference scene database as candidate trajectories for the current vehicle.

[0048] For example, the reference scene database may include tens of millions of reference scene data, each reference scene data corresponding to a reference scene. The reference scene data may include the features of the reference scene. The features of the reference scene may be determined in a similar way to the features of the current scene. For example, they may be determined based on multiple segments of information such as the historical trajectory of the reference vehicle, the historical trajectory of the reference obstacle, and the reference lane line within a historical time period (e.g., 1.6 seconds) before the time when the reference scene is located.

[0049] The reference scene database also stores the future trajectory of the reference scene, which can be the trajectory information within a future time period (e.g., 6 seconds) after the current time of the reference scene.

[0050] For example, the similarity between the features of the current scene and the features of each reference scene in the reference scene database can be calculated. The reference scenes in the database can then be sorted in descending order of similarity. Multiple (e.g., the Top K) reference scenes can be selected as candidate scenes according to the sorted order. The future trajectories of each of these multiple (e.g., the Top K) candidate scenes can then be determined as the candidate trajectories of the current vehicle, for example, resulting in K candidate trajectories.

[0051] In operation S230, for each candidate trajectory, the characteristics of the candidate trajectory are determined, and the evaluation value of the candidate trajectory is determined based on the characteristics of the current scene and the characteristics of the candidate trajectory.

[0052] For example, for each candidate trajectory, its features can be extracted. A deep learning model with a TransformerEncoder structure can be used to extract these features. Next, multiple (e.g., K) candidate trajectories and the features of the current scene are input into a trained trajectory ranking model. This model interacts with the features of the current scene and each candidate trajectory to obtain the interaction features between the current scene and each candidate trajectory. Based on these interaction features, an evaluation value for each candidate trajectory can be determined. The higher the evaluation value, the better the candidate trajectory matches the current scene, and the more suitable it is as a planned trajectory for the current scene.

[0053] The trained trajectory ranking model described above can be trained using reference scenes and corresponding future trajectories from the aforementioned reference scene database. For example, using the reference scene as a sample scene, multiple candidate trajectories are retrieved from the database. The trajectory ranking model interacts with the features of the sample scene and the candidate trajectories to calculate the evaluation value of the candidate trajectories. Based on the evaluation value of the candidate trajectories and the actual future trajectories of the sample scene, the loss of the trajectory ranking model is determined, and the parameters of the trajectory ranking model are adjusted according to the loss. This training process is repeated until the trained trajectory ranking model is obtained.

[0054] The trajectory ranking model described above can be any deep learning model capable of processing information interaction between features, such as an MLP (Multi-Layer Perception) model.

[0055] In operation S240, the target trajectory is determined from multiple candidate trajectories based on the evaluation value, which is then used as the planned trajectory for the current vehicle in the current scenario.

[0056] For example, since the evaluation value of a candidate trajectory is determined based on the features of the current scene and the features of the candidate trajectory, it can characterize the matching degree between the current scene and the candidate trajectory. Scene similarity, on the other hand, is a rough similarity between the current scene and millions of other scenes. Therefore, the evaluation value has higher confidence than scene similarity. Multiple candidate trajectories can be re-ranked according to their evaluation values ​​from highest to lowest to obtain a re-ranked result. This re-ranking result is more accurate than the ranking result based on scene similarity, and the recall rate of the optimal trajectory is higher.

[0057] To ensure traffic safety, SLT (Safety Level Calculation) can be performed sequentially on multiple candidate trajectories in the reordering results. If the SLT calculation passes, the current vehicle will not collide with the obstacle on that candidate trajectory; otherwise, a collision will occur. The first candidate trajectory that passes the SLT calculation can be determined as the planned trajectory for the current vehicle.

[0058] Since the reordering results based on the evaluation value are more accurate and the recall rate of the optimal trajectory is high, compared with the SLT safety calculation based on the scene similarity, the SLT safety calculation based on the reordering results in this embodiment is more likely to determine the target trajectory from the top (e.g., the top 10) candidate trajectories, thereby reducing the number of SLT safety calculation calls and improving the trajectory planning efficiency of autonomous vehicles.

[0059] The embodiments of this disclosure determine future trajectories of multiple reference scenarios as candidate trajectories from a reference scenario database based on the characteristics of the current scenario, determine the evaluation value of the candidate trajectories, and determine the target trajectory based on the evaluation value. The evaluation value can more accurately reorder the candidate trajectories, thus improving the optimal trajectory recall rate and effectively enhancing the efficiency of autonomous driving trajectory planning.

[0060] Figure 3 This is a schematic diagram of a method for determining the features of a current scene according to an embodiment of the present disclosure.

[0061] According to embodiments of this disclosure, determining the features of the current scene includes: determining current vehicle features, current obstacle features, and current lane line features based on multiple segments of information from the current vehicle's historical trajectory, the current obstacle's historical trajectory, and the current lane line; determining a first interaction feature between the current vehicle and the current lane line based on the current vehicle features and the current lane line features; determining a second interaction feature between the current vehicle and the current obstacle based on the first interaction feature and the current obstacle features; and determining the features of the current scene based on the second interaction feature.

[0062] like Figure 3 As shown, the current scene data includes current vehicle information, current obstacle information, and current lane line information. The historical information associated with the current scene data includes the current vehicle historical trajectory 320, the current obstacle historical trajectory 310, and multiple segments 330 of the current lane line within a historical time period (e.g., historical 1.6s) in the current scene.

[0063] The current obstacle's historical trajectory 310 can be extracted using a Transformer Encoder model or a convolutional model to obtain the current obstacle's features 311.

[0064] Similarly, a Transformer Encoder model or a convolutional model can be used to extract features from the current vehicle's historical trajectory 320, resulting in the current vehicle features 321.

[0065] For the current lane line, since it has been divided into multiple segments over a historical period, feature extraction can be performed on each segment to obtain the features of each segment. The features of each segment can include the position, color (e.g., white or yellow), and line type (e.g., solid or dashed line) of the lane line segment. By exchanging information between the features of each segment, the complete features of the current lane line can be obtained.

[0066] A combination of convolutional and pooling networks can be used to extract the current lane line features. The convolutional network extracts features from individual segments of the current lane line; compared to the Transformer Encoder structure, the convolutional network has lower computational complexity for feature extraction. The pooling network interacts with multiple segment features, linking them together and concatenating features with consistent characteristics (e.g., consistent color, consistent line type, continuous position) to obtain the lane line feature. This lane line feature is obtained by concatenating multiple lane line segment features from historical time periods; therefore, it represents the complete lane line information within the historical time period.

[0067] Lane markings may include a road center line, stop lines, and pedestrian crossings. Multiple segments of the current lane markings may include multiple center line segments, multiple stop line segments, and multiple pedestrian crossing segments.

[0068] Multiple centerline segments can be input into a combination of convolutional and pooling networks to obtain centerline features. Similarly, multiple stopline segments can be input into a combination of convolutional and pooling networks to obtain stopline features. Finally, multiple crosswalk segments can be input into a combination of convolutional and pooling networks to obtain crosswalk features.

[0069] Based on the current vehicle feature 321 and the current lane line feature 331, a first interaction feature 340 between the current vehicle and the current lane line can be determined. The current vehicle feature 321 represents the trajectory information of the current vehicle within a historical time period, and the current lane line feature 331 represents the complete lane line information of the road traversed by the current vehicle within a historical time period. Therefore, the first interaction feature 340 can represent the interaction information between the current vehicle and the lane line within a historical time period. For example, if the current vehicle crosses the center line of the current lane, then the current vehicle has interacted with the lane center line, and a first interaction feature 340 exists between the current vehicle feature 321 and the current lane line feature 331. This first interaction feature can be extracted using a model with an MCG (Multi-Context Gating) structure. The MCG structure is a module in Multipath++ (a trajectory prediction model) used for information interaction and association between features.

[0070] For example, the current vehicle feature 321 and the current lane line feature 331 are fused and then input into the MCG structure to obtain the first interaction feature 340 between the current vehicle and the current lane line.

[0071] A second interaction feature 350 between the current vehicle and the current obstacle can be determined based on the first interaction feature 340 and the current obstacle feature 311. The first interaction feature 340 characterizes the interaction information of the current vehicle interacting with the lane lines during a historical time period, and the current obstacle feature 311 characterizes the trajectory information of the current obstacle during a historical time period. Therefore, the second interaction feature 350 can characterize the interaction information of the current vehicle interacting with both the current lane lines and the current obstacle during a historical time period. For example, if the current vehicle crosses the center line of the current lane and goes around the current obstacle, then the current vehicle has interacted with both the current obstacle and the current obstacle, and a second interaction feature 350 exists between the first interaction feature 340 and the current obstacle feature 311. This second interaction feature can be extracted using a Transformer Encoder structure model.

[0072] For example, the first interaction feature 340 and the current obstacle feature 311 are fused together and then input into the Transformer Encoder structure to obtain the second interaction feature 350 between the current vehicle and the current obstacle.

[0073] The second interaction feature 350 can be directly used as a feature of the current scene. This second interaction feature 350 includes whether the current obstacle interacts with the current vehicle. For example, if the current obstacle information interacts with the current vehicle, the obstacle is an important obstacle, represented by "1". If the current obstacle information interacts with the current vehicle, the obstacle is an unimportant obstacle, represented by "0".

[0074] This embodiment obtains the features of the current scene by determining the interaction features between the current vehicle and the current lane line, as well as the interaction features between the current vehicle and the current obstacle. This makes the features of the current scene include the interaction information between the current vehicle and the lane line, as well as the interaction information between the current vehicle and the obstacle. This interaction information is the key information for vehicle trajectory planning. Therefore, the feature representation of the current scene is more accurate, which can improve the accuracy of scene retrieval and thus improve the accuracy of trajectory determination.

[0075] After determining the characteristics of the current scene, reference scenes similar to the current scene are identified from the reference scene database based on these characteristics. The reference scene database is described below.

[0076] The reference scenario database can store features of tens of millions of reference scenarios, as well as the future trajectory of each reference scenario.

[0077] The source of reference scenario data can be road test data obtained by using a dedicated autonomous vehicle to drive on a designated road segment for a certain period of time (e.g., 1 month or 1 year). This dedicated autonomous vehicle can serve as a reference vehicle. The road test data can be collected from the perspective of the reference vehicle using a coordinate system. This coordinate system can have the location of the main vehicle as the origin, the direction of travel as the y-axis, and a direction perpendicular to the y-axis as the x-axis.

[0078] Road running data can include tens of millions (e.g., 170,000) of autonomous driving samples (ADS). Each ADS corresponds to a continuous time period (e.g., 20 seconds) of road running data. Each ADS contains approximately 200 frames of reference scene data, with each frame corresponding to one reference scene and each frame of reference scene data lasting 0.1 seconds. Each frame of reference scene data includes information about the vehicle (i.e., the reference vehicle), reference obstacles (motorized vehicles, non-motorized vehicles, pedestrians), and reference road information. Furthermore, each frame of reference scene data is associated with historical information within a historical time period (e.g., 1.6 seconds) within that reference scene, as well as the trajectory information of the reference vehicle within a future time period (e.g., 6 seconds) within that reference scene. For example, each frame of reference scene data is associated with the historical trajectory of the reference vehicle within a historical time period, the historical trajectory of reference obstacles within a historical time period, multiple segments of reference lane line information within a historical time period, and the trajectory information of the reference vehicle within a future time period.

[0079] The characteristics of the reference scene can be determined by using multiple segments of information from the historical trajectories of the reference vehicles, the historical trajectories of the reference obstacles, and the reference lane lines during the historical period prior to the time when the reference scene is located.

[0080] The method for determining the features of the reference scene is similar to that for determining the features of the current scene. For example, reference vehicle features are determined based on the historical trajectory of the reference vehicle; reference obstacle features are determined based on the historical trajectory of the reference obstacle; multiple segment features of the reference lane line are determined based on multiple segment information; and the reference lane line features are determined based on these multiple segment features. A third interaction feature between the reference vehicle and the reference lane line is determined based on the reference vehicle features and the reference lane line features; a fourth interaction feature between the reference vehicle and the reference obstacle is determined based on the third interaction feature and the reference obstacle features, serving as the features of the reference scene; and the features of the reference scene and its future trajectory are added to the reference scene database. Next, the features of the reference scene and the future trajectory of the reference vehicle within a future time period under that reference scene can be associated and stored in the reference scene database. Multiple segment features of the reference lane line are determined based on multiple segment information; and the reference lane line features are determined based on these multiple segment features.

[0081] A third interaction feature between a reference vehicle and a reference lane line may indicate, for example, that the reference vehicle crosses the center line of the reference lane. A fourth interaction feature between the third interaction feature and a reference obstacle may indicate, for example, that the reference vehicle crosses the center line of the reference lane and goes around the reference obstacle.

[0082] The features of the reference scene include the interaction information between the reference vehicle and the lane lines, as well as the interaction information between the reference vehicle and obstacles. The feature representation is more accurate and facilitates scene retrieval.

[0083] The reference scenario data in this embodiment comes from the long-term road driving data of the reference vehicle. During the long-term road driving, the reference vehicle encounters a large number of scenarios. These scenarios are used as reference scenarios, and features and future trajectories are extracted and stored in the database, which can provide data support for trajectory retrieval.

[0084] In this embodiment, the features of the current scene and the feature representations of reference scenes in the reference scene database are more accurate, which can improve the accuracy of scene similarity calculation. However, since the reference field data in the reference database is in the tens of millions, determining multiple similar reference scenes from the database based on the features of the current scene still results in a relatively coarse scene similarity calculation. To further narrow down the candidate range and reduce the number of SLT rule calls, this embodiment provides a method for determining the evaluation value of candidate trajectories. This evaluation value can finely sort the recalled candidate trajectories, improve the optimal trajectory recall rate, and effectively improve the efficiency of autonomous driving trajectory planning.

[0085] Figure 4 This is a schematic diagram of a method for determining the evaluation value of a candidate trajectory according to an embodiment of the present disclosure.

[0086] like Figure 4 As shown, candidate trajectory 401 is a future trajectory of a reference scene similar to the current scene, selected from the reference scene data database based on features 402 of the current scene. There are, for example, K candidate trajectories 401 (e.g., K = 100). The K candidate trajectories 401 are input into the feature extraction model 410, which yields the features of each of the K candidate trajectories, such as feature 1, ..., feature K. The feature extraction model 410 can be a Transformer Encoder structure.

[0087] Feature 1, ..., Feature K, along with feature 402 of the current scene, are input into trajectory ranking model 420. Trajectory ranking model 420 interacts with feature 1, ..., Feature K to obtain evaluation values ​​1, ..., and K corresponding to each feature. Evaluation values ​​1, ..., and K are used as scores for candidate trajectories 1, ..., and candidate trajectory K, respectively. Higher scores indicate a higher degree of matching between the candidate trajectory and the current scene, making it more suitable as a planned trajectory for the current scene.

[0088] The trajectory ranking model 420 can be any deep learning model capable of processing information interaction between features, such as an MLP (Multi-Layer Perception) model.

[0089] In this embodiment, when trajectory planning is required in the current scenario, the trajectory ranking model 420 can quickly and in parallel calculate the score of each candidate trajectory when faced with multiple candidate trajectories. The final candidate trajectory is determined based on the score, which can reduce the time consumption and improve the computational efficiency.

[0090] Figure 5 This is a schematic diagram of a method for determining the evaluation value of a candidate trajectory according to an embodiment of the present disclosure.

[0091] like Figure 5 As shown, this embodiment includes two parts: determining the features of the current scene and determining the evaluation value of the candidate trajectory. The part of determining the evaluation value of the candidate trajectory is as follows: Figure 5 As shown in the dashed box in the image.

[0092] The following section explains the characteristics of determining the current scene.

[0093] Input the current obstacle historical trajectory 501 and the current vehicle historical trajectory 502 into the TransformerEncoder model 510 (the model with the Transformer Encoder structure can be called the Transformer Encoder model) to obtain the current obstacle feature 511 and the current vehicle feature 512.

[0094] The current lane line consists of multiple segments 503, including multiple centerline segments, multiple stopline segments, and multiple pedestrian crossing segments. Multiple centerline segments are input into a convolutional + pooling model 520 to obtain centerline features. Multiple stopline segments are input into the convolutional + pooling model 520 to obtain stopline features. Multiple pedestrian crossing segments are input into the convolutional + pooling model 520 to obtain pedestrian crossing features. The convolutional + pooling model may include a convolutional network, a max-pooling network, and an average pooling network. The convolutional network is used to extract lane line segment features, while the max-pooling and average pooling networks are used to interact with the multiple lane line features, ensuring that each segment feature contains information from the other segment features. This extracts the most prominent segment features and the average features of all segment features to form the lane line features.

[0095] Inputting the current vehicle feature 512 and the center line feature into the MCG model 530 yields the interaction features between the current vehicle and the center line. Inputting the current vehicle feature 512 and the stop line feature into the MCG model 530 yields the interaction features between the current vehicle and the stop line. Inputting the current vehicle feature 512 and the pedestrian crossing feature into the MCG model 530 yields the interaction features between the current vehicle and the pedestrian crossing. The interaction features between the current vehicle and the center line, the current vehicle and the stop line, and the current vehicle and the pedestrian crossing are then fused together and input into the MLP model 540. The MLP model interacts and correlates these three interaction features to obtain the first interaction feature 541 between the current vehicle and the lane lines.

[0096] The current obstacle feature 511 and the first interaction feature 541 are input into the fusion module 550 to obtain the fused feature. The fused feature is then input into the Transformer Encoder model 560 to obtain the second interaction feature between the current vehicle and the current obstacle. This second interaction feature can be used as the feature 561 of the current scene.

[0097] The evaluation value for determining candidate trajectories is explained below.

[0098] Candidate trajectories 504 are future trajectories of reference scenes similar to the current scene, selected from the reference scene data database based on features 561 of the current scene. There are, for example, K candidate trajectories 504 (e.g., K = 100). The K candidate trajectories 504 are input into the Transformer Encoder model 570 to obtain the features of each of the K candidate trajectories, such as feature 1, ..., feature K.

[0099] Feature 1, ..., Feature K, along with feature 561 of the current scene, are input into MLP model 580 to obtain evaluation values ​​1, ..., and evaluation value K corresponding to each of feature 1, ..., Feature K. Evaluation values ​​1, ..., and evaluation value K are used as scores for candidate trajectories 1, ..., and candidate trajectory K, respectively. The higher the score, the higher the matching degree between the candidate trajectory and the current scene, and the more suitable it is as the planned trajectory for the current scene.

[0100] This embodiment extracts features of the current scene, calculates the evaluation value of the candidate trajectory based on the features of the current scene and the features of the candidate trajectory, and can sort multiple candidate trajectories more finely, making the trajectory determination more accurate.

[0101] Figure 6 This is a flowchart of a training method for a deep learning model according to an embodiment of the present disclosure.

[0102] like Figure 6 As shown, the training method 600 of the deep learning model includes operations S610 to S640.

[0103] During operation S610, based on the characteristics of the sample scene, the future trajectories of multiple reference scenes are determined from the reference scene database as candidate trajectories for the sample scene.

[0104] In operation S620, the features of the sample scene and the features of each of the multiple candidate trajectories are input into the deep learning model to obtain the evaluation values ​​of each of the multiple candidate trajectories.

[0105] When operating the S630, the loss of the deep learning model is determined based on the evaluation values.

[0106] When operating the S640, adjust the parameters of the deep learning model based on the loss.

[0107] This deep learning model can be used to determine the evaluation values ​​of multiple candidate trajectories. The features of the sample scene can come from a reference scene database, meaning that tens of millions of scene data points in the reference scene database can be used as training data to train the deep learning model.

[0108] For example, N reference scenes can be used as sample scenes for each training session. For each sample scene i among the N sample scenes, based on the features of that sample scene, K (e.g., K=100) similar reference scenes can be recalled from the reference scene database as their respective future trajectories, which are then used as K candidate trajectories for sample scene i. The features of sample scene i and the features of the K candidate trajectories are then input into the deep learning model to obtain the evaluation values ​​of the K candidate trajectories.

[0109] Sample scenario i has a real label trajectory, which can be the future trajectory of sample scenario i. For each candidate trajectory, based on the distance between the candidate trajectory and the label trajectory, a first probability that the candidate trajectory is the optimal trajectory can be determined. Since this first probability is calculated based on the label trajectory, it can be considered realistic and reliable.

[0110] The first probability can be calculated according to the following formula (1).

[0111]

[0112] Where, p ij Let d represent the first probability, i represent the i-th sample scenario, and d represent the first probability. ij d represents the distance between the labeled trajectory of the i-th sample scene and the j-th candidate trajectory. it Let K represent the distance between the labeled trajectory of the i-th sample scene and the t-th candidate trajectory, and K represent the number of candidate trajectories.

[0113] For each candidate trajectory, based on the evaluation value of the candidate trajectory output by the deep learning model, the second probability that the candidate trajectory is the optimal trajectory can be calculated.

[0114] The second probability can be calculated according to the following formula (2).

[0115]

[0116] Where, p′ ij Let s represent the second probability, where i represents the i-th sample scenario; ij This represents the evaluation value of the j-th candidate trajectory output by the deep learning model for the i-th sample scenario; s it Let K represent the evaluation value of the t-th candidate trajectory output by the deep learning model for the i-th sample scenario; K represents the number of candidate trajectories.

[0117] The first probability is determined based on the label, and the second probability is determined based on the model output. Therefore, the loss of the deep learning model can be determined based on the second probability and the first probability.

[0118] For example, the loss function can be determined with the goal of minimizing the difference between the first probability and the second probability; the first probability and the second probability can be input into the loss function to determine the loss of the deep learning model.

[0119] With the goal of minimizing the difference between the first probability and the second probability, the loss function can be constructed as shown in formula (3).

[0120]

[0121] Where Loss represents the loss function, i represents the i-th sample scenario, N represents the number of sample scenarios, .j represents the .j-th candidate trajectory, K represents the number of candidate trajectories, and p ij Let p' represent the first probability. ij This represents the second probability.

[0122] During model training, the first probability p can be... ij Second probability p′ ij Input the above formula (3) and train with the goal of minimizing the loss.

[0123] This embodiment uses data from a reference scenario database to train a deep learning model, enabling the deep learning model to output evaluation values ​​for candidate trajectories. Based on these evaluation values, the optimal trajectory can be determined, thereby improving the accuracy of determining the optimal trajectory.

[0124] The embodiments of this disclosure use a deep learning model to perform fine-grained sorting of candidate trajectories, which can optimize the recall rate of candidate trajectories compared to not using a deep learning model to perform fine-grained sorting of candidate trajectories.

[0125] For example, candidate trajectories include trajectory information for the next 6 seconds. For a given sample scenario, if the error between the position of the recalled Top 1 candidate trajectory at the next 1 second and the position of the actual future trajectory at the next 1 second in the sample scenario is less than a threshold (e.g., 0.5 meters), then the Top 1 candidate trajectory is considered a successfully recalled candidate trajectory at 1 second. The proportion of successful Top 1 recalls can be calculated according to this standard to obtain the 1-second Top 1 recall rate.

[0126] Similarly, if the error between the position of the recalled Top 1 candidate trajectory at the next 1 second and the position of the true future trajectory in the sample scene at the next 1 second is less than a threshold, and the error between the position of the Top 1 candidate trajectory at the next 3 seconds and the position of the true future trajectory in the sample scene at the next 3 seconds is also less than a threshold, then the Top 1 candidate trajectory is considered a candidate trajectory successfully recalled at 3 seconds. The proportion of successful Top 1 recalls can be calculated according to this standard to obtain the 3-second Top 1 recall rate.

[0127] Figure 7 This is a recall curve of a candidate trajectory according to an embodiment of the present disclosure.

[0128] like Figure 7 As shown, the horizontal axis k of this recall curve represents the top k candidate tracks that have been recalled, k = 1, ..., K. This recall curve shows the recall rate of the top 10 candidate tracks.

[0129] The recall curve also shows a comparison between the recall rate of using a deep learning model to finely rank candidate trajectories and the recall rate of not using a deep learning model to finely rank candidate trajectories.

[0130] like Figure 7 As shown, curve 701 represents the 1-second recall rate of the top 10 candidate trajectories after fine-grained ranking, curve 702 represents the 1-second recall rate of the top 10 candidate trajectories without fine-grained ranking, curve 703 represents the 3-second recall rate of the top 10 candidate trajectories after fine-grained ranking, and curve 704 represents the 3-second recall rate of the top 10 candidate trajectories without fine-grained ranking.

[0131] Referring to curves 701 and 702, the top-1 recall rate at 1 second with fine-grained sorting of candidate trajectories was 94.41%, while the top-1 recall rate at 1 second without fine-grained sorting of candidate trajectories was 80.58%.

[0132] Referring to curves 703 and 704, the top-1 recall rate for 3-second points with fine-grained ranking of candidate trajectories was 76.20%, while the top-1 recall rate for 3-second points without fine-grained ranking of candidate trajectories was 43.37%.

[0133] By comparison, it can be seen that this embodiment performs fine-grained sorting of candidate trajectories, which significantly improves the recall rate of candidate trajectories compared to related technologies that do not perform fine-grained sorting.

[0134] Furthermore, with the improvement of top-1 recall, the efficiency of calling SLT rules for safety calculation also improves accordingly. In related technologies, schemes that do not perform fine-grained sorting of candidate trajectories require an average of 3.4 SLT rule calls per scenario. This embodiment, which performs fine-grained sorting of candidate trajectories, only requires an average of 1.8 SLT rule calls, effectively saving time.

[0135] Figure 8 This is a block diagram of an apparatus for determining a planned trajectory according to an embodiment of the present disclosure.

[0136] like Figure 8 As shown, the device 800 for determining the planned trajectory includes a current scene feature determination module 801, a first candidate trajectory determination module 802, an evaluation value determination module 803, and a target trajectory determination module 804.

[0137] The current scene feature determination module 801 is used to determine the features of the current scene based on the current scene data of the current vehicle and the historical information associated with the current scene data.

[0138] The first candidate trajectory determination module 802 is used to determine the future trajectories of multiple reference scenarios from the reference scenario database based on the characteristics of the current scenario, as candidate trajectories of the current vehicle. The future trajectory includes the trajectory information of the reference vehicle in the future time period under the reference scenario.

[0139] The evaluation value determination module 803 is used to determine the features of each candidate trajectory and determine the evaluation value of the candidate trajectory based on the features of the current scene and the features of the candidate trajectory.

[0140] The target trajectory determination module 804 is used to determine the target trajectory from multiple candidate trajectories based on the evaluation value, and to use it as the planned trajectory of the current vehicle in the current scenario.

[0141] According to embodiments of this disclosure, the current scene data includes current vehicle information, current obstacle information, and current lane line information; the historical information includes the historical trajectory of the current vehicle, the historical trajectory of the current obstacle, and multiple segments of the current lane line within a historical time period in the current scene; the current scene feature determination module includes a current feature determination unit, a first interactive feature determination unit, and a second interactive feature determination unit.

[0142] The current feature determination unit is used to determine the current vehicle features, current obstacle features, and current lane line features based on multiple segments of information from the current vehicle's historical trajectory, the current obstacle's historical trajectory, and the current lane line.

[0143] The first interaction feature determination unit is used to determine the first interaction feature between the current vehicle and the current lane line based on the current vehicle features and the current lane line features.

[0144] The second interaction feature determination unit is used to determine the second interaction feature between the current vehicle and the current obstacle based on the first interaction feature and the current obstacle feature, and use it as the feature of the current scene.

[0145] The current feature determination unit is used to determine multiple segment features of the current lane line based on multiple segment information of the current lane line; and to determine the current lane line features based on the multiple segment features of the current lane line.

[0146] The first candidate trajectory determination module includes a first similarity calculation unit, a first reference scene determination unit, and a first candidate trajectory determination unit.

[0147] The first similarity calculation unit is used to calculate the similarity between the features of the current scene and the features of each reference scene in the reference scene database.

[0148] The first reference scene determination unit is used to determine multiple reference scenes from the reference scene database based on similarity.

[0149] The first candidate trajectory determination unit is used to determine the future trajectories of multiple reference scenarios as candidate trajectories for the current vehicle.

[0150] The device 800 for determining the planned trajectory also includes a reference scene feature determination module, which includes a reference feature determination unit, a third interactive feature determination unit, a fourth interactive feature determination unit, and a storage unit.

[0151] The reference feature determination unit is used to determine reference vehicle features, reference obstacle features, and reference lane line features based on multiple segments of information from the historical trajectory of the reference vehicle, the historical trajectory of the reference obstacle, and the reference lane line within a historical time period in the reference scenario.

[0152] The third interaction feature determination unit is used to determine the third interaction feature between the reference vehicle and the reference lane line based on the reference vehicle feature and the reference lane line feature.

[0153] The fourth interaction feature determination unit is used to determine the fourth interaction feature between the reference vehicle and the reference obstacle based on the third interaction feature and the reference obstacle feature, as a feature of the reference scene.

[0154] The storage unit is used to add the features of the reference scene and the future trajectory of the reference scene to the reference scene database.

[0155] The reference feature determination unit is used to determine multiple segment features of the reference lane line based on multiple segment information of the reference lane line; and to determine the reference lane line features based on the multiple segment features of the reference lane line.

[0156] The evaluation value determination module 803 includes a fifth interactive feature determination unit and an evaluation value determination unit.

[0157] The fifth interaction feature determination unit is used to exchange information between the features of the current scene and the features of the candidate trajectory to obtain the fifth interaction feature between the current scene and the candidate trajectory.

[0158] The evaluation value determination unit is used to determine the evaluation value of the candidate trajectory based on the fifth interaction feature between the current scene and the candidate trajectory.

[0159] The target trajectory determination module 804 includes a sorting unit and a target trajectory determination unit.

[0160] The sorting unit is used to sort multiple candidate trajectories based on the evaluation values.

[0161] The target trajectory determination unit is used to perform safety calculations on multiple candidate trajectories in sequence according to the order of their arrangement, until a candidate trajectory that passes the safety calculation is obtained, which is then used as the target trajectory.

[0162] Figure 9 This is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure.

[0163] like Figure 9 As shown, the training device 900 of the deep learning model includes a second candidate trajectory determination module 901, a processing module 902, a loss determination module 903, and an adjustment module 904.

[0164] The second candidate trajectory determination module 901 is used to determine the future trajectories of multiple reference scenarios from the reference scenario database based on the characteristics of the sample scenario, as candidate trajectories of the sample scenario. The future trajectory includes the trajectory information of the reference vehicle in the future time period under the reference scenario.

[0165] The processing module 902 is used to input the features of the sample scene and the features of each of the multiple candidate trajectories into the deep learning model to obtain the evaluation values ​​of each of the multiple candidate trajectories.

[0166] The loss determination module 903 is used to determine the loss of the deep learning model based on the evaluation value.

[0167] The adjustment module 904 is used to adjust the parameters of the deep learning model based on the loss.

[0168] The sample scenario includes labeled trajectories, which contain trajectory information for future time periods within the sample scenario. The loss determination module 903 includes a first probability determination unit, a second probability determination unit, and a loss determination unit.

[0169] The first probability determination unit is used to determine the first probability that a candidate trajectory is the optimal trajectory for the sample scene for each candidate trajectory, based on the distance between the label trajectory and the candidate trajectory.

[0170] The second probability determination unit is used to determine the second probability that a candidate trajectory is the optimal trajectory for the sample scene for each candidate trajectory, based on the evaluation value of the candidate trajectory.

[0171] The loss determination unit is used to determine the loss of the deep learning model based on a first probability and a second probability.

[0172] The loss determination unit is used to determine the loss function with the goal of minimizing the difference between the first probability and the second probability; and to determine the loss of the deep learning model by inputting the first probability and the second probability into the loss function.

[0173] The second candidate trajectory determination module 901 includes a second similarity calculation unit, a second reference scene determination unit, and a second candidate trajectory determination unit.

[0174] The second similarity calculation unit is used to calculate the similarity between the features of the sample scene and the features of each reference scene in the reference scene database.

[0175] The second reference scene determination unit is used to determine multiple reference scenes from the reference scene database based on similarity.

[0176] The second candidate trajectory determination unit is used to determine the future trajectories of multiple reference scenarios as candidate trajectories in the sample scenario.

[0177] According to embodiments of this disclosure, this disclosure also provides an electronic device, an autonomous vehicle, a readable storage medium, and a computer program product.

[0178] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0179] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0180] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0181] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs at least one of the methods and processes described above, such as methods for determining a planned trajectory and methods for training a deep learning model. For example, in some embodiments, at least one of the methods for determining a planned trajectory and methods for training a deep learning model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of at least one of the methods for determining a planned trajectory and methods for training a deep learning model described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured by any other suitable means (e.g., by means of firmware) to perform at least one of a method for determining a planned trajectory and a method for training a deep learning model.

[0182] The autonomous vehicles provided in the embodiments of this disclosure may include scene acquisition devices and such as Figure 10 The electronic device shown is an example of a scene acquisition device, such as a camera. During the operation of the autonomous vehicle, a coordinate system is constructed with the vehicle's location as the origin, the vehicle's direction of travel as the y-axis, and a direction perpendicular to the y-axis as the x-axis. Scene data is acquired frame by frame within this coordinate system. Each frame of scene data may include the scene image at the given time, as well as the autonomous vehicle's position information, obstacle (pedestrian, motor vehicle, non-motor vehicle, etc.) position information, and road lane line information within the scene image.

[0183] The scene acquisition device can send the acquired scene data to the electronic device, which can, for example, implement the method for determining the planned trajectory described above to obtain the planned trajectory of the autonomous vehicle in the future time period (e.g., the next 6 seconds) under the current scene. Continuous frames of scene data are sent to the electronic device, which can continuously output the planned trajectory for the future time period for each frame of scene data, thereby forming the planned trajectory of the autonomous vehicle.

[0184] Autonomous vehicles can control themselves to drive automatically along a planned trajectory determined by electronic devices.

[0185] This embodiment places the electronic device inside the autonomous vehicle, which saves data transmission time and improves data processing efficiency compared to placing it on an external server, thereby further improving the trajectory planning efficiency of the autonomous vehicle.

[0186] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0188] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0190] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0191] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0192] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0193] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining a planned trajectory, comprising: Based on the current scene data of the current vehicle and the historical information associated with the current scene data, the characteristics of the current scene are determined. The current scene data includes current vehicle information, current obstacle information and current lane line information, and the historical information includes multiple segments of current lane line information collected by the current vehicle during historical time periods. Based on the characteristics of the current scene, the future trajectories of multiple reference scenes are determined from the reference scene database as candidate trajectories of the current vehicle, wherein the future trajectory includes the trajectory information of the reference vehicle in the future time period under the reference scene; For each candidate trajectory, the features of the candidate trajectory are determined, and a deep learning model is used to interact the features of the current scene with the features of multiple candidate trajectories to obtain the interaction features between the current scene and each candidate trajectory. Based on the interaction features, the evaluation value of the candidate trajectory is determined; and Based on the evaluation value, a target trajectory is determined from the plurality of candidate trajectories and used as the planned trajectory of the current vehicle in the current scenario; The features for determining the current scene include: Feature extraction is performed on each segment of the current lane line to obtain multiple segment features; Information is exchanged among multiple segment features to make the features of each segment related to each other, and the segment features with the same features are spliced ​​together to obtain the features of the current lane line. The characteristics of the current scene are determined based on the current vehicle characteristics, current obstacle characteristics, and current lane line characteristics.

2. The method according to claim 1, wherein, The historical information also includes the historical trajectory of the current vehicle and the historical trajectory of the current obstacle within a historical time period in the current scene; determining the characteristics of the current scene based on the current scene data of the current vehicle and the historical information associated with the current scene data includes: Based on the current vehicle historical trajectory, the current obstacle historical trajectory, and multiple segments of current lane line information, determine the current vehicle characteristics, current obstacle characteristics, and current lane line characteristics; Based on the current vehicle features and the current lane line features, a first interaction feature between the current vehicle and the current lane line is determined; Based on the first interaction feature and the current obstacle feature, a second interaction feature between the current vehicle and the current obstacle is determined as a feature of the current scene.

3. The method according to claim 1 or 2, wherein, The step of determining the future trajectories of multiple reference scenarios from a reference scenario database based on the characteristics of the current scenario, as candidate trajectories for the current vehicle, includes: Calculate the similarity between the features of the current scene and the features of each reference scene in the reference scene database; Based on the similarity, the plurality of reference scenes are determined from the reference scene database; and The future trajectories of each of the multiple reference scenarios are determined as candidate trajectories for the current vehicle.

4. The method according to claim 1, further comprising: Based on multiple segments of information from the historical trajectories of reference vehicles, historical trajectories of reference obstacles, and reference lane lines within a historical time period in the reference scenario, the characteristics of reference vehicles, reference obstacles, and reference lane lines are determined. Based on the reference vehicle features and the reference lane line features, a third interaction feature between the reference vehicle and the reference lane line is determined; Based on the third interaction feature and the reference obstacle feature, a fourth interaction feature between the reference vehicle and the reference obstacle is determined as a feature of the reference scene; as well as The features of the reference scene and the future trajectory of the reference scene are added to the reference scene database.

5. The method according to claim 4, wherein, The process of determining the characteristics of the reference vehicle, the reference obstacle, and the reference lane line based on multiple segments of information from the historical trajectory of the reference vehicle, the historical trajectory of the reference obstacle, and the reference lane line within a historical time period in the reference scenario includes: Based on multiple segment information of the reference lane line, multiple segment features of the reference lane line are determined; The features of the reference lane line are determined based on multiple segment features of the reference lane line.

6. The method according to any one of claims 1 to 5, wherein, The step of determining the target trajectory from the plurality of candidate trajectories based on the evaluation value, as the planned trajectory of the current vehicle in the current scenario, includes: Based on the evaluation values, the plurality of candidate trajectories are sorted; and According to the order in which the multiple candidate trajectories are arranged, the multiple candidate trajectories are sequentially subjected to security calculations until a candidate trajectory that passes the security calculation is obtained, which is then used as the target trajectory.

7. The method according to claim 1, further comprising: Based on the characteristics of the sample scene, the future trajectories of multiple reference scenes are determined from the reference scene database as candidate trajectories of the sample scene, wherein the future trajectory includes the trajectory information of the reference vehicle in the future time period under the reference scene; The features of the sample scene and the features of each of the multiple candidate trajectories are input into a deep learning model to obtain the evaluation values ​​of each of the multiple candidate trajectories. Based on the evaluation value, determine the loss of the deep learning model; and The parameters of the deep learning model are adjusted based on the loss.

8. The method according to claim 7, wherein, The sample scenario includes a label trajectory, which includes trajectory information for future time periods within the sample scenario. The step of determining the loss of the deep learning model based on the evaluation value includes: For each candidate trajectory, based on the distance between the label trajectory and the candidate trajectory, determine the first probability that the candidate trajectory is the optimal trajectory for the sample scene; For each candidate trajectory, based on the evaluation value of the candidate trajectory, a second probability is determined that the candidate trajectory is the optimal trajectory for the sample scene; and The loss of the deep learning model is determined based on the first probability and the second probability.

9. The method according to claim 8, wherein, Determining the loss of the deep learning model based on the first probability and the second probability includes: The loss function is determined with the objective of minimizing the difference between the first probability and the second probability; and The first probability and the second probability are input into the loss function to determine the loss of the deep learning model.

10. The method according to claim 7, wherein, The step of determining the future trajectories of multiple reference scenes from a reference scene database based on the characteristics of the sample scene, including the following candidate trajectories for the sample scene: Calculate the similarity between the features of the sample scene and the features of each reference scene in the reference scene database; Based on the similarity, the plurality of reference scenes are determined from the reference scene database; and The future trajectories of each of the multiple reference scenarios are determined as candidate trajectories in the sample scenario.

11. An apparatus for determining a planned trajectory, comprising: The current scene feature determination module is used to determine the features of the current scene based on the current scene data of the current vehicle and the historical information associated with the current scene data. The current scene data includes current vehicle information, current obstacle information and current lane line information, and the historical information includes multiple segments of current lane line information collected by the current vehicle during a historical period. The first candidate trajectory determination module is used to determine the future trajectories of multiple reference scenarios from the reference scenario database based on the characteristics of the current scenario, as candidate trajectories of the current vehicle, wherein the future trajectory includes trajectory information of the reference vehicle in the future time period under the reference scenario; An evaluation value determination module is used to determine the features of each candidate trajectory, and to use a deep learning model to interact the features of the current scene with the features of multiple candidate trajectories to obtain the interaction features between the current scene and each candidate trajectory. Based on these interaction features, the evaluation value of the candidate trajectory is determined. The target trajectory determination module is used to determine a target trajectory from the plurality of candidate trajectories based on the evaluation value, as the planned trajectory of the current vehicle in the current scenario; The current scene feature determination module is further used to extract features for each segment of the current lane line to obtain multiple segment features; to perform information interaction on the multiple segment features so that the segment features are related to each other; to splice together segment features with consistent features to obtain the features of the current lane line; and to determine the features of the current scene based on the current vehicle features, the current obstacle features, and the current lane line features.

12. The apparatus according to claim 11, wherein, The historical information includes the historical trajectory of the current vehicle and the historical trajectory of the current obstacle within the historical time period of the current scene. The current scene feature determination module includes: The current feature determination unit is used to determine the current vehicle features, current obstacle features, and current lane line features based on multiple segments of information from the current vehicle historical trajectory, the current obstacle historical trajectory, and the current lane line. The first interaction feature determination unit is used to determine the first interaction feature between the current vehicle and the current lane line based on the current vehicle feature and the current lane line feature. The second interaction feature determination unit is used to determine a second interaction feature between the current vehicle and the current obstacle based on the first interaction feature and the current obstacle feature, as a feature of the current scene.

13. The apparatus according to claim 11 or 12, wherein, The first candidate trajectory determination module includes: The first similarity calculation unit is used to calculate the similarity between the features of the current scene and the features of each reference scene in the reference scene database. The first reference scene determination unit is configured to determine the plurality of reference scenes from the reference scene database based on the similarity; and The first candidate trajectory determination unit is used to determine the future trajectories of the multiple reference scenarios as candidate trajectories of the current vehicle in the current scenario.

14. The apparatus according to claim 11, further comprising a reference scene feature determination module, the reference scene feature determination module comprising: The reference feature determination unit is used to determine the reference vehicle features, reference obstacle features, and reference lane line features based on multiple fragments of information from the historical trajectory of the reference vehicle, the historical trajectory of the reference obstacle, and the reference lane line within a historical time period in the reference scenario. The third interaction feature determination unit is used to determine the third interaction feature between the reference vehicle and the reference lane line based on the reference vehicle feature and the reference lane line feature. The fourth interaction feature determination unit is used to determine a fourth interaction feature between the reference vehicle and the reference obstacle based on the third interaction feature and the reference obstacle feature, as a feature of the reference scene; as well as A storage unit is used to add the features of the reference scene and the future trajectory of the reference scene to the reference scene database.

15. The apparatus according to claim 14, wherein the reference feature determining unit is configured to determine multiple segment features of the reference lane line based on multiple segment information of the reference lane line; and to determine the reference lane line features based on the multiple segment features of the reference lane line.

16. The apparatus according to any one of claims 11 to 15, wherein, The target trajectory determination module includes: A sorting unit is configured to sort the plurality of candidate trajectories according to the evaluation values; and The target trajectory determination unit is used to perform safety calculations on the multiple candidate trajectories sequentially according to the arrangement order of the multiple candidate trajectories, until a candidate trajectory that passes the safety calculation is obtained, which is then used as the target trajectory.

17. The apparatus of claim 11, further comprising: The second candidate trajectory determination module is used to determine the future trajectories of multiple reference scenarios from the reference scenario database based on the characteristics of the sample scenario, as candidate trajectories of the sample scenario, wherein the future trajectory includes the trajectory information of the reference vehicle in the reference scenario within a future time period. The processing module is used to input the features of the sample scene and the features of each of the multiple candidate trajectories into a deep learning model to obtain the evaluation values ​​of each of the multiple candidate trajectories; A loss determination module is used to determine the loss of the deep learning model based on the evaluation value; and An adjustment module is used to adjust the parameters of the deep learning model based on the loss.

18. The apparatus according to claim 17, wherein, The sample scenario includes a labeled trajectory, which includes trajectory information for a future time period within the sample scenario; the loss determination module includes: The first probability determination unit is used to determine, for each candidate trajectory, the first probability that the candidate trajectory is the optimal trajectory of the sample scene based on the distance between the label trajectory and the candidate trajectory; The second probability determination unit is used, for each candidate trajectory, to determine a second probability that the candidate trajectory is the optimal trajectory of the sample scene based on the evaluation value of the candidate trajectory; and The loss determination unit is used to determine the loss of the deep learning model based on the first probability and the second probability.

19. The apparatus according to claim 18, wherein, The loss determination unit is configured to determine a loss function with the objective of minimizing the difference between the first probability and the second probability; and to input the first probability and the second probability into the loss function to determine the loss of the deep learning model.

20. The apparatus according to claim 17, wherein, The second candidate trajectory determination module includes: The second similarity calculation unit is used to calculate the similarity between the features of the sample scene and the features of each reference scene in the reference scene database. The second reference scene determination unit is configured to determine the plurality of reference scenes from the reference scene database based on the similarity; and The second candidate trajectory determination unit is used to determine the future trajectories of the multiple reference scenarios as candidate trajectories under the sample scenario.

21. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1 to 10 when executed by a processor.

24. An autonomous vehicle, including the electronic equipment as claimed in claim 21.

Images