Automatic driving trajectory prediction model training method and device, equipment and storage medium

Abstract

The present disclosure provides an automatic driving trajectory prediction model training method and device, equipment and a storage medium, relates to the technical field of computers, and particularly relates to the technical fields of large models, maps, automatic driving, intelligent transportation and the like. The specific implementation scheme is as follows: map information, obstacle information and hard isolation information are input into an encoding module of a trajectory prediction model, the encoding module encodes the map information, the obstacle information and the hard isolation information respectively to obtain map features, obstacle features and hard isolation features; a decoding module of the trajectory prediction model decodes the prediction request vector, the map features, the obstacle features and the hard isolation features based on the prediction request vector, the map features, the obstacle features and the hard isolation features to obtain a predicted trajectory; a loss function is calculated based on the predicted trajectory, the hard isolation information and a trajectory label, and parameters of the trajectory prediction model are adjusted based on the loss function.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of large models, maps, autonomous driving, and intelligent transportation. Background Technology

[0002] In autonomous driving systems, trajectory prediction is a core component for predicting vehicle behavior and planning safe paths. By accurately predicting the vehicle's trajectory, the autonomous driving system can anticipate the state of surrounding obstacles, thereby planning a driving strategy that conforms to actual traffic conditions and improving driving safety and efficiency. However, current deep learning models may produce trajectory predictions that do not conform to real-world constraints, such as predicting trajectories that cross impassable areas. Such trajectories that violate basic constraints of road structure and traffic rules not only reduce the safety and reliability of the autonomous driving system but also hinder its iterative optimization. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for training an autonomous driving trajectory prediction model.

[0004] According to one aspect of this disclosure, a method for training an autonomous driving trajectory prediction model is provided, comprising: Map information, obstacle information, and hard isolation information are input into the encoding module of the trajectory prediction model. The encoding module encodes the map information, obstacle information, and hard isolation information respectively to obtain map features, obstacle features, and hard isolation features. The trajectory prediction model's decoding module decodes the predicted trajectory based on the prediction request vector, map features, obstacle features, and hard isolation features. The loss function is calculated based on the predicted trajectory, hard isolation information, and trajectory labels, and the parameters of the trajectory prediction model are adjusted based on the loss function.

[0005] According to another aspect of this disclosure, an apparatus for training an autonomous driving trajectory prediction model is provided, comprising: The information input module is used to input map information, obstacle information and hard isolation information into the encoding module of the trajectory prediction model. The encoding module encodes the map information, obstacle information and hard isolation information respectively to obtain map features, obstacle features and hard isolation features. The trajectory prediction module is used to decode the trajectory prediction model based on the prediction request vector, map features, obstacle features, and hard isolation features to obtain the predicted trajectory. The model training module is used to calculate the loss function based on the predicted trajectory, hard isolation information, and trajectory labels, and to adjust the parameters of the trajectory prediction model based on the loss function.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] This disclosure inputs map information, obstacle information, and hard isolation information into the encoding module for targeted encoding, which can extract key features of various types of information; the decoding module decodes based on the prediction request vector and multiple features, which can generate a predicted trajectory, and then calculates the loss function and adjusts the model parameters based on the predicted trajectory, hard isolation information, and trajectory labels, which can continuously optimize the model performance, reduce the occurrence of trajectory prediction errors, improve the rationality and accuracy of the predicted trajectory, and thus improve driving safety and efficiency.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure; Figure 2 This is a flowchart illustrating the implementation of an autonomous driving trajectory prediction model training method according to an embodiment of the present disclosure. Figure 3 This is a flowchart of the overall process of a training trajectory prediction model according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating the extraction of hard isolation occupancy semantic features according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of ViT according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of a decoding module according to an embodiment of the present disclosure; Figure 7 This is a schematic diagram of gradient information for determining occupancy gradient loss according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of distance information for determining occupancy distance loss according to an embodiment of the present disclosure; Figure 9 This is a flowchart illustrating the construction of hard-isolation coded supervision loss according to an embodiment of the present disclosure; Figure 10 This is a flowchart illustrating the implementation of a scan line algorithm according to an embodiment of the present disclosure; Figure 11 This is a schematic diagram of road occupancy information according to an embodiment of the present disclosure; Figure 12 This is a schematic diagram of hard isolation information according to an embodiment of the present disclosure; Figure 13 This is a technical flowchart of a training trajectory prediction model according to an embodiment of the present disclosure; Figure 14 This is a schematic diagram of the structure of an autonomous driving trajectory prediction model training device 1400 according to an embodiment of the present disclosure; Figure 15 This is a schematic diagram of the structure of an autonomous driving trajectory prediction model training device 1500 according to an embodiment of the present disclosure; Figure 16 A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] The term "and / or" in this disclosure indicates that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document means any combination of at least two of a plurality of options, such as including at least one of A, B, and C, which can mean including any one or more elements selected from the set of A, B, and C. The terms "first" and "second" in this document refer to and distinguish multiple similar technical terms, and do not imply a specific order or a limitation to only two. For example, "first feature" and "second feature" refer to two types / two features; the first feature can be one or more, and the second feature can also be one or more.

[0014] In autonomous driving systems, trajectory prediction is a core component for predicting vehicle behavior and planning safe paths. However, existing deep learning models often fail to meet real-world constraints during trajectory prediction, potentially predicting trajectories that cross impassable areas (or hard-barrier areas, or occupied areas). Such unreasonable trajectories violate fundamental constraints of road structure and traffic rules. Inappropriate trajectories generated by the trajectory prediction or planning modules reduce the safety and reliability of autonomous driving systems, hinder their iterative optimization, and may even pose potential safety risks.

[0015] Currently, common solutions to this type of problem mainly fall into two categories: (1) Post-processing rule method: This method is applied after trajectory generation is completed. Specifically, after the deep learning model generates all candidate trajectories, these trajectories are geometrically checked. Once a predicted trajectory is found to have crossed a hard barrier, it will be directly deleted.

[0016] However, this method is prone to erroneous deletion of potential modes. In extreme cases, all trajectories may be deleted, resulting in no usable trajectories. For example, in real-world traffic scenarios, when a traffic participant turns right, their trajectory may be misjudged as crossing the right-hand curb, and thus deleted. Consequently, the predicted trajectory can only maintain the straight-ahead mode, affecting the accuracy, diversity, and completeness of trajectory prediction.

[0017] (2) Implicit Modeling: This method incorporates map or mask information of hard-isolated regions into the input data of the deep learning model, hoping that the model will "automatically learn" the constraint relationships of impassable regions (i.e., hard-isolated or occupied regions) during training. In other words, this method embeds hard-isolation features as input features into the deep learning model without direct supervision, allowing the model to understand impassable regions through the hidden layer structure and thus predict trajectories that conform to the constraints.

[0018] However, this method requires a large capacity from the deep learning model, and small models struggle to fully learn the geometric and semantic features of hard-isolated regions. Even with this method, many unreasonable trajectories are still generated. Furthermore, this method relies on the coverage of training data, and may not consistently generate reasonable and effective trajectories in different traffic scenarios, resulting in poor robustness.

[0019] Combining the two methods described above, existing technologies have two prominent problems in terms of trajectory rationality constraints: (1) Insufficient constraints on the rationality of the trajectory space; the deep learning model lacks the ability to actively avoid impassable areas, which makes the generated trajectory prone to violating traffic rules and road structure constraints; (2) Trajectory filtering carries the risk of modal deletion, which not only affects the prediction accuracy of the model in complex traffic environments, but also reduces the interpretability and stability of the model, posing potential risks to the safe operation of autonomous driving systems.

[0020] To address the aforementioned issues, this disclosure proposes a training method for a trajectory prediction model. This method introduces constraints related to hard road barriers during the training process, jointly modeling the movement trends of traffic participants with road structure information. Consequently, the trajectories determined by the trajectory prediction model are more reasonable both physically and in terms of compliance with traffic rules. This reduces the likelihood of predicted trajectories crossing hard barrier areas during the trajectory generation stage, enhancing the reliability and interpretability of the trajectory prediction model.

[0021] In terms of applications, this disclosure has broad applicability and can be applied to trajectory prediction models, path planning modules, and intelligent obstacle simulation in autonomous driving systems. The method proposed in this disclosure can improve the trajectory prediction quality and robustness of autonomous vehicles in complex road environments, thereby enhancing the safety and efficiency of vehicle operation.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure, such as... Figure 1As shown, the application scenario diagram of this disclosure embodiment may include, but is not limited to, a model training device 110 and a trajectory prediction model 120, which can communicate with each other via any type of wired or wireless network. Specifically, the model training device 110 can be used to receive a training dataset, which may include obstacle trajectory information and map information, etc. Further, the model training device 110 can use the training dataset to enable the initial model to generate predicted trajectories, and calculate a loss function based on the predicted trajectories and pre-labeled trajectory samples. Even further, the model training device 110 trains the initial model based on the loss function to obtain the trajectory prediction model 120. Furthermore, this disclosure embodiment does not impose a specific limitation on the number of model training devices 110; for example, the application scenario diagram of this disclosure embodiment may include one or more model training devices 110.

[0023] Figure 2 This is a flowchart illustrating the implementation of an autonomous driving trajectory prediction model training method according to an embodiment of the present disclosure, including: S210. Input map information, obstacle information and hard isolation information into the encoding module of the trajectory prediction model. The encoding module encodes the map information, obstacle information and hard isolation information respectively to obtain map features, obstacle features and hard isolation features. S220. The decoding module of the trajectory prediction model decodes the predicted trajectory based on the prediction request vector, map features, obstacle features, and hard isolation features. S230. Calculate the loss function based on the predicted trajectory, hard isolation information, and trajectory label, and adjust the parameters of the trajectory prediction model based on the loss function.

[0024] In this embodiment of the disclosure, the map information may include one or more of the following: lane geometry information, traffic light information, stop line information, and pedestrian crossing information. Lane geometry information may include the shape, width, length, curvature of the lanes, and the relative positional relationships between different lanes; traffic light information may include the location, type, and switching time patterns of different traffic light colors; stop line information may include the specific location of the stop line on the road and its distance from the traffic light; pedestrian crossing information may include the location, width, and length of the pedestrian crossing.

[0025] Obstacle information can be related information used to describe objects (such as vehicles, pedestrians, etc.) that may affect the driving of the main vehicle in an autonomous driving scenario. It can include one or more of the following time-series data: obstacle position information, speed information, acceleration information, and heading angle information.

[0026] Hard barrier information refers to a dataset used to describe the characteristics of hard-barrier areas in a road environment that are impassable, fixed, and clearly separated. In one example, hard barrier information can be presented as a binary image, in which white areas can be considered non-hard-barrier areas, and black areas can be considered hard-barrier areas. Furthermore, hard barrier information may include one or more of the following: type of hard barrier facility, location information, shape characteristics, etc.

[0027] In the embodiments of this disclosure, the encoding module of the trajectory prediction model comprises multiple sub-modules, each encoding information for a different modality. The encoding module independently encodes the information for each input modality, aiming to transform the raw information into a feature representation that the trajectory prediction model can understand and process. After encoding, map features, obstacle features, and hard isolation features can be obtained respectively. Details regarding feature encoding will be described in detail later. Furthermore, the decoding module of the trajectory prediction model decodes the prediction request vector, map features, obstacle features, and hard isolation features to generate a predicted trajectory. In one example, the predicted trajectory consists of a series of discrete predicted trajectory points. After post-processing the predicted trajectory (such as kinematic constraint processing), a constrained trajectory can be obtained, which can be used to characterize the predicted trajectory for the target vehicle. The structure of the decoding module of the trajectory prediction model and the process of determining the predicted trajectory will be described in detail later.

[0028] In one example, the prediction request vector can serve as the initiation signal for the trajectory prediction model to begin the trajectory prediction task, typically consisting of a set of tokens. Each token carries specific information, which could be candidate modalities that traffic participants might adopt, such as different driving modes a vehicle might take at an intersection, like going straight or turning; or it could be an instance of the trajectory to be predicted. This set of tokens works together to provide the trajectory prediction model with information about the possible motion states of the target vehicle, guiding the model to generate a predicted trajectory that accurately reflects the actual situation.

[0029] In this embodiment, a loss function can be calculated based on the generated predicted trajectory, hard isolation information, and pre-prepared trajectory labels. The trajectory labels can be the actual movement trajectory of the target vehicle in real-world conditions, serving as a standard to measure the accuracy of the predicted trajectory. Simultaneously, hard isolation information is utilized in the calculation of the loss function to provide constraints on the relationship between the predicted trajectory and the hard isolation region. The calculation process of the loss function will be described in detail later.

[0030] Furthermore, the parameters of the trajectory prediction model are adjusted based on the calculated loss function. In one example, this disclosure uses a backpropagation algorithm to calculate the gradient value of the loss function with respect to the model parameters of each layer, starting from the output layer of the trajectory prediction model. Based on the calculated gradient values ​​of the model parameters of each layer, the model parameters are updated using an optimization algorithm (such as stochastic gradient descent). This optimization algorithm can determine the adjustment magnitude and direction of the model parameters based on the magnitude and direction of the gradient values.

[0031] In this example, the trajectory prediction model is trained iteratively multiple times by repeating the steps of calculating the loss function, calculating the gradient value, and adjusting the model parameters. As the iterations proceed, the loss function gradually decreases, and the performance of the trajectory prediction model continuously improves. The training process ends when the loss function converges to a preset value or when the preset number of iterations is reached, yielding the final trajectory prediction model.

[0032] Using the above method, map information, obstacle information, and hard isolation information are input into the encoding module for targeted encoding, which can extract key features of various types of information. The decoding module decodes based on the prediction request vector and multiple features, which can generate a predicted trajectory. Then, based on the predicted trajectory, hard isolation information, and trajectory labels, the loss function is calculated and the model parameters are adjusted. This can continuously optimize the model performance, reduce trajectory prediction errors, improve the rationality and accuracy of the predicted trajectory, and thus improve driving safety and efficiency.

[0033] This disclosure proposes a training method for a trajectory prediction model based on interaction modeling between traffic participants and hard road barriers, aiming to improve the rationality of trajectory prediction by autonomous driving systems in complex scenarios, with a focus on enhancing the rationality of trajectory prediction and path planning. The core idea of ​​this method is to jointly model the motion information of traffic participants with the occupancy information of hard road barriers (i.e., hard barrier information, Occupancy Map, Occ Map), and introduce a hard barrier information constraint supervision mechanism in the trajectory prediction stage, thereby enabling the predicted trajectory to actively avoid hard barrier areas in the road.

[0034] Figure 3 This is a flowchart of the overall process of a training trajectory prediction model according to an embodiment of the present disclosure. Figure 3 As shown, training the trajectory prediction model includes the following steps: S301. Input training data and preprocess the training data.

[0035] In this embodiment of the disclosure, the training data may include map information, obstacle information and road occupancy information, and the preprocessing process may refer to the process of determining hard isolation information after processing the road occupancy information.

[0036] In some implementations, before inputting map information, obstacle information, and hard isolation information into the encoding module of the trajectory prediction model, the following method is also included: Determine road occupancy information and transform the coordinate system of the road occupancy information to the main vehicle coordinate system; The road occupancy information after coordinate system transformation is normalized into intermediate occupancy information of a preset size. The intermediate occupancy information includes at least one closed region used to characterize the occupancy of a hard-isolated object. The closed region includes a passable grid and different passable grids. Adjust passable grids within a closed area to impassable grids to determine hard isolation information.

[0037] In this embodiment of the disclosure, road occupancy information can be a depiction of the three-dimensional spatial usage status of a road. In one example, a voxel grid can be used to divide the three-dimensional space into small cubic units (i.e., voxel grids), and each grid can be defined as a passable grid and an impassable grid to obtain road occupancy information. Thus, in the road occupancy information, it is possible to identify whether each grid is occupied by an object (i.e., occupied space) or is in an vacant state (i.e., unoccupied space). Furthermore, the objects occupying space in the road occupancy information are classified to distinguish between static targets (such as roads, trees, fixed buildings, etc.) or dynamic targets in motion (such as vehicles, pedestrians, etc.), thereby determining hard isolation areas based on the grids occupied by static targets.

[0038] After determining the road occupancy information, in order to unify the perspective of the training data for the input trajectory prediction model, this disclosure requires transforming the coordinate system of the road occupancy information to the vehicle coordinate system. In one example, the road occupancy information is usually obtained by collecting three-dimensional spatial information from various sensors (such as LiDAR, vision sensors, etc.). These sensors each have their own orientation and angle, and the data they collect is based on the sensor's own coordinate system.

[0039] In this example, the translation matrix between the road occupancy information coordinate system and the vehicle coordinate system is determined; then, the rotation matrix is ​​constructed based on the heading angle, pitch angle, and roll angle of the vehicle coordinate system; and finally, the coordinate system of the road occupancy information, the translation matrix, and the rotation matrix are used to transform the road occupancy information coordinate system to the vehicle coordinate system.

[0040] Furthermore, this disclosure can normalize the road occupancy information after coordinate system transformation into occupancy intermediate information of a preset size. This occupancy intermediate information can be presented in the form of an image, and the size can be [w, h, f], where w is the width of the occupancy intermediate information, h is the height of the occupancy intermediate information, and f is the number of channels of the occupancy intermediate information (e.g., including whether it is occupied, occupancy type, and occupancy height).

[0041] In this embodiment, a hard-isolated object can refer to a static target within the occupies intermediate information, and a closed region can be a region with a clear boundary in a two-dimensional image that completely surrounds the space occupied by the hard-isolated object. Within the closed region, an impassable grid can be a region completely blocked and impassable by the hard-isolated object; since the occupies intermediate information is generally a sparse point cloud, in addition to impassable grids, there may also be passable grids within the closed region. To enhance the understanding of the trajectory prediction model, this disclosure requires processing the passable grids within the closed region to obtain dense hard-isolated information (i.e., an Occ Map).

[0042] In one example, this disclosure may set a scan line, which is used to traverse the occupancy intermediate information row by row or column by column, identify passable grids in closed regions of the occupancy intermediate information, and adjust the passable grids to be impassable grids to determine hard isolation information. Here, the scan line algorithm will be described in detail later.

[0043] By employing the above method, through steps such as coordinate system transformation, size normalization, and grid adjustment, the representation of hard isolation areas in road occupancy information is corrected, making the hard isolation information conform to the actual hard isolation area outline in the real scene, improving the accuracy of hard isolation information, providing a reliable data foundation for subsequent trajectory prediction model processing, helping the model to accurately capture road structure information, and enhancing the model's understanding and reasoning ability in complex road scenes.

[0044] like Figure 3 As shown, training the trajectory prediction model also includes the following steps.

[0045] S302. Perform multimodal feature encoding on the training data.

[0046] In this embodiment of the disclosure, the encoding module of the trajectory prediction model can perform multimodal encoding of map information, obstacle information and hard isolation information to obtain map features, obstacle features and hard isolation features respectively, so that the trajectory prediction model can understand and utilize the input training data.

[0047] S303. Decode based on multimodal features and reconstruct hard isolation information.

[0048] In this embodiment of the disclosure, the decoding module of the trajectory prediction model can be used to decode based on multimodal features (i.e., map features, obstacle features, and hard isolation features) in conjunction with the prediction request vector to obtain the predicted trajectory. In one example, interactive modeling of hard isolation information is added to this decoding stage, causing the predicted trajectory to detour around the hard isolation area.

[0049] Furthermore, this disclosure can also utilize an external decoding model to reconstruct hard-isolation information based on hard-isolation features, obtaining hard-isolation reconstructed values. These hard-isolation reconstructed values ​​have the same structure as the hard-isolation information; that is, they can also be images of size [w, h, f]. In one example, these hard-isolation reconstructed values ​​can be used for subsequent supervision of a trajectory prediction model.

[0050] S304. Calculate the loss function and use the loss function to train the trajectory prediction model.

[0051] In this embodiment, the loss function can be calculated based on the predicted trajectory, hard isolation information, trajectory labels, and hard isolation reconstruction values. Using the loss function, the trajectory prediction capability of the trajectory generation model can be supervised and optimized, enabling the predicted trajectory to bypass hard isolation areas.

[0052] The following content details the process of encoding map information, obstacle information, and hard isolation information separately.

[0053] In this embodiment of the disclosure, the encoding module of the trajectory prediction model may include multiple different encoding layers to encode multimodal information separately, thereby extracting semantic information of the environment.

[0054] In some implementations, the encoding module encodes map information, including: The encoding module extracts features from map elements in the map information to obtain the first semantic features; The encoding module determines the first relative relationship between the requested location indicated by the predicted request vector and the location of the map feature, and determines the first relative location feature based on the first relative relationship; The map features include first semantic features and first relative position features.

[0055] In this embodiment of the disclosure, when processing map information, the encoding module converts map elements such as lane lines and center lines into semantic representations that the trajectory prediction model can understand, so as to obtain the first semantic feature.

[0056] Furthermore, the encoding module can calculate the first relative relationship between the requested location and map features, such as spatial association information like distance, direction, and angle, based on the requested location indicated by the predicted request vector, and generate a first relative position feature based on the first relative relationship. This first relative position feature can reflect the spatial interaction relationship between the requested location and map features, such as the distance between the vehicle's current position and the intersection ahead, or the relative orientation of a pedestrian's position and a zebra crossing.

[0057] In this embodiment of the disclosure, the map features are composed of a first semantic feature and a first relative position feature, which includes both the attribute information of the map features themselves and the association between the requested location and the map features.

[0058] By adopting the above approach, this comprehensive map feature provides more comprehensive and accurate data support for the subsequent trajectory prediction model processing, thereby improving the model's ability to understand the map scene and enhancing the reliability and practicality of trajectory prediction.

[0059] In some implementations, the encoding module encodes obstacle information, including: The encoding module extracts features from obstacle information to obtain second semantic features; The encoding module determines a second relative relationship between the request position indicated by the predicted request vector and the obstacle position, and determines a second relative position feature based on the second relative relationship; Obstacle features include second semantic features and second relative position features.

[0060] In this embodiment, the encoding module can receive obstacle information, such as vehicles, pedestrians, and other objects affecting the driving of the main vehicle. By extracting features from the obstacle information, the encoding module can capture multiple key attributes of the obstacles and convert them into an encoded form that the trajectory prediction model can understand, i.e., second semantic features.

[0061] For example, for surrounding vehicles, the second semantic feature can include historical state information such as vehicle type (car, truck, bus, etc.), speed, direction of travel, and whether it is turning or changing lanes; for pedestrians, the second semantic feature can include the pedestrian's walking speed, direction of travel, etc.

[0062] In this embodiment of the disclosure, the requested position indicated by the predicted request vector can represent the current or future expected location of the main vehicle. The encoding module can calculate a second relative relationship between the requested position and the positions of various obstacles. This second relative relationship can include distance, angle, direction, etc. Based on the second relative relationship, the encoding module can determine a second relative position feature. This second relative position feature can express the distance, orientation, and angle relationship between the main vehicle and the obstacles.

[0063] In one example, this disclosure combines a second semantic feature and a second relative position feature to form an obstacle feature.

[0064] Using the above method, second semantic features are obtained by extracting features from obstacle information, which can deeply explore the inherent attributes and state information of obstacles, providing a basis for the trajectory generation model to understand obstacle features. By determining the second relative relationship between the requested position indicated by the prediction request vector and the obstacle position, and generating second relative position features accordingly, the relative spatial layout of the vehicle and obstacles is characterized. The second semantic features and the second relative position features are fused to form obstacle features, enabling the model to not only understand the obstacle itself when dealing with interaction scenarios with obstacles, but also accurately grasp the relative geometric relationship between the vehicle and obstacles, enhancing the relative geometric perception in the trajectory prediction process.

[0065] In this embodiment of the disclosure, the encoding module may include one or more of the following structures: VectorNet and Lane Graph Convolutional Network (LaneGCN). Thus, this disclosure can utilize the encoding module to extract features from map information and obstacle information to obtain map features and obstacle features.

[0066] In some implementations, the encoding module encodes the hard isolation information, including: The encoding module extracts hard-isolation occupancy semantic features from the hard-isolation information; Determine the position information of each word in the semantic features occupied by hard isolation within a predetermined spatial range, and determine the hard isolation position encoding based on the position information; the predetermined spatial range is indicated by the hard isolation information. Hard isolation features include hard isolation occupancy semantic features and hard isolation location encoding.

[0067] In this embodiment, the encoding module can perform in-depth analysis and processing of hard isolation information to extract key semantic features of hard isolation occupancy. Here, the semantic features of hard isolation occupancy can be an abstract expression of the attributes of the hard isolation area, such as information representing the type of the hard isolation area (e.g., metal fence) and the continuity of the hard isolation area.

[0068] In this embodiment of the disclosure, the predetermined spatial range can be the actual road space area indicated by the hard isolation information. Within the predetermined spatial range, the semantic features of hard isolation are subdivided into multiple tokens, and each token has corresponding location information within the predetermined spatial range. This location information may include the coordinate position of the hard isolation area corresponding to the token in the road, etc.

[0069] Furthermore, based on the location information, the encoding module can determine the hard-isolation location code. Here, the hard-isolation location code records the location information of each word in a specific encoding format. For example, in a two-dimensional coordinate system, the hard-isolation location code can be a vector containing horizontal and vertical coordinate information. Through this vector, the trajectory prediction model can determine the location information of the corresponding word within a predetermined range.

[0070] In this embodiment of the disclosure, the hard isolation occupancy semantic feature and the hard isolation position encoding are combined to form a complete hard isolation feature.

[0071] Using the above method, the encoding module extracts hard isolation occupancy semantic features from hard isolation information, enabling it to uncover the essential attributes of hard isolation areas and providing a semantic foundation for the trajectory prediction model. By determining the positional information of each lexical unit within a predetermined spatial range and generating hard isolation position codes, the semantic information is anchored in the actual space, allowing the model to clearly grasp the specific distribution of hard isolation areas within the road. The fusion of hard isolation occupancy semantic features and hard isolation position codes constitutes hard isolation features, achieving an organic unity of semantic and spatial information, thereby enhancing the model's ability to perceive and understand hard isolation information.

[0072] In some implementations, the encoding module extracts hard isolation occupancy semantic features from the hard isolation information, including: The hard isolation information is segmented to obtain multiple image patches; The feature vector and position vector of each image patch are determined, and the feature vector and the corresponding position vector of each image patch are concatenated to obtain the spatial feature vector; Multiple spatial feature vectors are input into the encoding module, which then extracts the semantic features of hard isolation occupancy.

[0073] Figure 4 This is a schematic diagram illustrating the extraction of hard isolation occupancy semantic features according to an embodiment of the present disclosure.

[0074] like Figure 4 As shown, this disclosure divides the hard isolation information into multiple fixed-size image patches, which can be numbered from 1 to 9. These multiple image patches are then flattened and uniformly input into the linear mapping module 410, which determines the feature vector of each image patch. In one example, the linear mapping module 410 can be a linear projection of flattened patches.

[0075] In this embodiment of the disclosure, a relative position encoding method can be used to determine the position vector of each image block. This method focuses on the relative positional relationship (such as distance, direction, etc.) between image blocks. Specifically, the relative row and column difference between image block i and image block j is calculated using a two-dimensional relative position offset matrix and mapped to a position vector.

[0076] Furthermore, this disclosure allows the feature vectors of each image patch to be concatenated with their corresponding position vectors to form a spatial feature vector. For example... Figure 4 As shown, the white graphic represents the feature vector of each image block, and the black graphic represents the position vector of each image block. By concatenating the feature vector and the position vector, the spatial feature vector of each image block can be determined.

[0077] In this embodiment of the disclosure, multiple spatial feature vectors are input into an encoding module, which can extract hard isolation occupancy semantic features.

[0078] In one example, the encoding module may include a Vision Transformer (ViT) 420, which can perform self-attention encoding on the input spatial feature vector to extract hard-isolated occupancy semantic features from local regions. Specifically, this disclosure utilizes the self-attention mechanism of ViT 420 to capture long-range dependencies between different image patches, such as the continuity of hard-isolated regions, thereby extracting hard-isolated occupancy semantic features.

[0079] Figure 5 This is a schematic diagram of the structure of ViT according to an embodiment of the present disclosure. Figure 5 As shown, the structure of ViT may include an L-layer network, and each layer may include a first normalization submodule 510, a multi-head attention submodule 520, a second normalization submodule 530, and a multilayer perceptron (MLP) 540.

[0080] In one example, assume the size of the input hard isolation information is 224 pixels (px). 224px, this disclosure can divide hard isolation information into fixed sizes of 16px. A 16px image patch means each hard isolation information can be divided into 196 image patches, meaning the length of the flattened sequence of image patches is 196. In this example, each image patch can have a dimension of 16. 16 3-dimensional (i.e., 768-dimensional). This is because the linear mapping module has a dimension of 768. N (N=768), therefore the total number of feature vectors output by the linear mapping module is 196. 768-dimensional, meaning the linear mapping module outputs 196 feature vectors, each with a dimension of 768.

[0081] In this embodiment, the process of determining hard-isolation occupancy semantic features using ViT requires the introduction of position vectors. These position vectors can be analogized to a parameter table containing multiple rows, the number of which is the same as the number of feature vectors output by the linear mapping module. Each row represents a position vector, and the dimension of the position vectors is the same as the dimension of the feature vectors output by the linear mapping module, i.e., 768 dimensions. Further, the feature vectors of each image patch are concatenated with their corresponding position vectors to obtain a spatial feature vector of 196... 1536 dimensions.

[0082] In the embodiments disclosed herein, such as Figure 5 As shown, when the spatial feature vector is input into the first normalization submodule 510, the dimension of the output of the first normalization submodule 510 is still 196. 1536 dimensions.

[0083] In the process of inputting the output of the first normalization submodule 510 into the multi-head attention submodule 520, the input content can first be mapped into a query vector, a key vector, and a value vector. If there is only one attention head, the dimensions of the query vector, key vector, and value vector are all 196. 1536; If there are multiple attention heads (e.g., 12), then the dimensions of the query vector, key vector, and value vector are all 196. 1536 / Number of attention heads (e.g., 196) 1536 / 12 dimensions, or 196 (128 dimensions), which yields multiple sets of query vectors, key vectors, and value vectors. Then, these multiple sets of query vectors, key vectors, and value vectors are concatenated to obtain the output of the multi-head attention submodule 520, which has a dimension of 196. 1536 dimensions.

[0084] Furthermore, the output of the multi-head attention submodule 520 is input into the second normalization submodule 530, and the dimension of the output of the second normalization submodule 530 remains 196. 1536 dimensions.

[0085] like Figure 5 As shown, the output of the second normalization submodule 530 can be used as the input of MLP 540. MLP 540 can scale up and down the dimensions of the input content, that is, MLP 540 can scale the dimensions of the input content from 196... 1536 dimensions magnified to 196 6144 dimensions (i.e., magnified 4 times), then 196 The 6144-dimensional content was reduced to 196. 1536 dimensions.

[0086] In this embodiment, the spatial feature vector of the input ViT and the content output by ViT have the same dimension. This disclosure can stack multiple ViTs to extract hard-isolation occupancy semantic features. It should be noted that the specific dimensions provided above are only for illustrating the algorithm flow, and this disclosure does not explicitly limit the size or dimension of the hard-isolation information.

[0087] By segmenting the hard isolation information into multiple image blocks using the above method, the complexity of information processing can be reduced. Determining the feature vector accurately captures the visual content information of each image block, while the position vector provides the spatial location information of the image block. The concatenated spatial feature vector achieves an organic fusion of visual content and spatial location. Furthermore, the encoding module can mine the semantic features of hard isolation occupancy based on the spatial feature vector, improving the trajectory prediction model's ability to understand hard isolation information.

[0088] It should be noted that this disclosure can also embed deep learning feature extraction networks such as SwinTransformer and Residual Net (RseNet) into the encoding module to achieve the extraction of semantic features of hard isolation occupancy.

[0089] The following content details the structure of the decoding module of the trajectory prediction model and the process of determining the predicted trajectory.

[0090] In this embodiment of the disclosure, a transformer-based decoding module is used to perform cross attention interaction between obstacle features, map features and hard isolation features, so as to inject hard isolation features into the trajectory prediction token, thereby realizing the active avoidance of hard isolation areas during the trajectory prediction process.

[0091] Figure 6 This is a schematic diagram of the structure of a decoding module according to an embodiment of the present disclosure.

[0092] In some implementations, the decoding module includes multiple decoding layers connected in sequence, with the last decoding layer outputting the predicted trajectory; Each decoding layer performs decoding based on the first input data, map features, obstacle features, and hard isolation features; wherein, the first input data of the first decoding layer includes the prediction request vector, and the first input data of other decoding layers includes the output data of the previous decoding layer.

[0093] In this embodiment of the disclosure, the decoding module of the trajectory prediction model can be constructed based on multiple (e.g., N) sequentially connected decoding layers of the Transformer architecture. This decoding module takes the prediction request vector (or trajectory query vector) as initial input, and combines it with pre-defined anchor points and the positional encoding information of the predicted trajectory (or predicted trajectory points) output by the previous decoding layer. It then uses a self-attention module to perform preliminary modeling of the prediction request vector.

[0094] In one example, the first input data of the first decoding layer includes a prediction request vector, which may be determined using preset initialization data (Zero Init); the first input data of other decoding layers may include the output data of the previous decoding layer, namely the prediction request vector and prediction trajectory generated by the previous decoding layer.

[0095] This disclosure allows for cross-attention calculation with map features, obstacle features, and hard isolation features, based on the prediction request vector output from the previous decoding layer. During this process, this disclosure can utilize the prediction request vector to determine the query vector used for cross-attention calculation (i.e., the self-attention result, such as...). Figure 6 As shown in q'), and based on map features, obstacle features, and hard isolation features, determine the key vector for cross-attention computation (i.e., Figure 6 The k1, k2, and k3 shown) and the value vector (i.e. Figure 6 (as shown in v1, v2, and v3). In this way, the trajectory prediction model can explicitly integrate the behavioral information of traffic participants around the main vehicle, road topology information, and occupancy information of hard-barrier areas during the trajectory prediction process.

[0096] It should be noted that this disclosure utilizes a cross-attention mechanism, which can directly inject the semantic information of impassable areas in hard isolation information into the token of the prediction request vector. This enables the trajectory prediction model to perceive the location of hard isolation areas during the trajectory prediction stage and actively adjust the trajectory generation direction, thereby suppressing the trajectory from crossing hard isolation areas such as central guardrails, curbs, and medians.

[0097] Furthermore, this disclosure concatenates multiple cross-attention results obtained through multiple cross-attention calculations and fuses the concatenated results using a multilayer perceptron (MLP). Then, this disclosure inputs the fused result into a trajectory decoder to generate a predicted trajectory. This disclosure uses a multi-layer stacking approach to iteratively refine the predicted trajectory.

[0098] In the output stage of the last decoding layer of the decoding module, kinematic constraint units can be further introduced to ensure that the output predicted trajectory meets both the accessibility requirements of the environment and the principle of physical realizability.

[0099] By employing the above approach, through the ordered stacking of multiple decoding layers, the trajectory prediction model can perform progressive and in-depth decoding processing of the input information, which helps to more accurately uncover complex features related to the trajectory prediction task. Furthermore, each decoding layer decodes based on the first input data, map features, obstacle features, and hard isolation features, achieving effective fusion of multi-source heterogeneous information. Specifically, the first decoding layer uses the prediction request vector as the first input data, providing initial task-oriented information for the entire decoding process; subsequent decoding layers use the output data of the previous layer as input, forming a mechanism for information transmission and progression. This allows the model to utilize the output results of the previous layer to progressively optimize the inference results, improving the accuracy and reliability of the inference results.

[0100] In some implementations, each decoding layer includes a self-attention unit and a cross-attention unit; The decoding layer decodes based on the first input data, map features, obstacle features, and hard isolation features, including: The self-attention unit performs self-attention calculations based on the first input data to obtain the self-attention result; The cross-attention unit performs cross-attention calculations based on self-attention results, map features, obstacle features, and hard isolation features to obtain cross-attention results; and determines the output data of the decoding layer based on the cross-attention results.

[0101] like Figure 6 As shown, each decoding layer may include a self-attention unit 610 and a cross-attention unit 640. During the self-attention calculation stage, the self-attention unit 610 can capture the internal dependencies of the first input data through a self-attention mechanism, thereby obtaining the self-attention result. Specifically, this disclosure can generate an initial query vector, an initial key vector, and an initial value vector based on the first input data to determine the input content of the self-attention unit 610, and then determine the self-attention result through self-attention calculation.

[0102] In this embodiment of the disclosure, the cross-attention unit 640 can use the self-attention result as the query vector (i.e., q') for cross-attention calculation, and combine it with external environmental features (i.e., map features, obstacle features and hard isolation features) to perform cross-modal information fusion and form a cross-attention result.

[0103] Furthermore, this disclosure can determine the output data of the decoding layer (i.e., the prediction request vector and prediction trajectory output by the decoding layer) based on the cross-attention results.

[0104] Using the above method, the self-attention unit performs self-attention calculation based on the first input data, which can uncover the complex relationships and potential patterns between elements within the first input data, capture key information, and suppress irrelevant noise, thereby generating accurate self-attention results. Subsequently, the cross-attention unit, guided by the self-attention results, performs cross-attention calculation in conjunction with external environmental features, realizing the organic integration of internal features and external environmental information. This enables the trajectory prediction model to understand the relationship between trajectory generation and map topology, obstacle distribution, and hard isolation locations, thereby improving the accuracy, robustness, and adaptability of trajectory prediction.

[0105] The following content details the specific methods for calculating self-attention (i.e., determining the results of self-attention).

[0106] In some implementations, the self-attention unit performs self-attention calculations based on the first input data to obtain a self-attention result, including: The self-attention unit of the first decoding layer performs self-attention calculation based on the prediction request vector and pre-set anchor information to obtain the first intermediate result; The self-attention unit of the first decoding layer integrates the first intermediate result and concatenates the integrated result with the pre-set anchor information to obtain the self-attention result of the first decoding layer.

[0107] In the embodiments disclosed herein, such as Figure 6 As shown, the prediction request vector in the first decoding layer can be determined based on the initialization data (i.e., Zero Init). In one example, the initialization data may refer to the zero initialization or fixed initialization assignment set in advance when the decoding layer processes the trajectory prediction task. During the decoding layer operation, this initialization data can be continuously updated iteratively.

[0108] In this embodiment of the disclosure, the prediction request vector (or trajectory query vector) may be a set of initial tokens used by the decoding layer to predict future trajectories, wherein each token corresponds to a candidate modality or a trajectory instance to be predicted.

[0109] In this embodiment of the disclosure, the pre-set anchor points can refer to pre-set initial trajectory points. The anchor points can provide the decoding layer with initial spatial prior knowledge, so that when the decoding layer performs trajectory prediction, it does not start from an unconstrained state, but refines the work around some reasonable initial candidate positions.

[0110] In one example, such as Figure 6As shown, the self-attention unit 610 may include a self-attention calculation subunit 611 and a gated residual subunit 612, and each decoding layer may also include a first sine position coding unit 620.

[0111] In this example, the first sinusoidal position encoding unit 620 may include a sinusoidal position encoding subunit and a Transformer Multilayer Perceptron (TRMLP). Before performing self-attention calculation, this disclosure requires encoding pre-defined anchor information. Specifically, the sinusoidal position encoding subunit can perform sinusoidal position encoding on the pre-defined anchor information; then, this disclosure allows the position-encoded anchor information to pass through the TRMLP of the first sinusoidal position encoding unit 620 for mapping processing, forming anchor position features that can be used for attention calculation. The core function of the first sinusoidal position encoding unit 620 is to transform the anchor information into an embedded representation that the decoding layer can compute, thereby providing guidance for the subsequent self-attention alignment process.

[0112] Furthermore, such as Figure 6 As shown, this disclosure can perform self-attention calculation based on the predicted request vector and pre-defined anchor point information. Specifically, this disclosure can determine the initial query vector (i.e., based on the predicted request vector and the anchor point position features obtained after encoding the pre-defined anchor point information) Figure 6 q in the initial key vector (i.e. Figure 6 k) and the initial value vector (i.e. Figure 6 The formula for self-attention calculation is as follows: (q, k, and v are then input into the self-attention calculation subunit 611 for self-attention calculation.) (1) Where softmax(.) represents the normalized exponential function; q is the initial query vector, k is the initial key vector, v is the initial value vector; dk is the dimension of the initial key vector; Attention(.) represents the result of the self-attention calculation, i.e., the first intermediate result.

[0113] In this embodiment, after the self-attention calculation subunit 611 performs self-attention calculation based on the prediction request vector and pre-set anchor point information, it inputs the result of the self-attention calculation (i.e., the first intermediate result) into the gated residual network (GRN) subunit 612 to integrate the first intermediate result. Specifically, the gated residual network subunit 612 is located after the self-attention calculation subunit 611. The gated residual network subunit 612 filters effective information through a gating mechanism and retains some original information by combining residual connections, thereby realizing a nonlinear update of the first intermediate result and achieving the integration effect of the first intermediate result.

[0114] In this embodiment of the disclosure, the decoding layer may further include a second sine position encoding unit 630, which has the same structure and function as the first sine position encoding unit 620, and will not be described in detail here.

[0115] In one example, the present disclosure may input pre-set anchor information into a second sine position encoding unit 630, which converts the anchor information into an embedded representation (i.e., anchor position features) that the decoding layer can compute.

[0116] It should be noted that in the first decoding layer, the content input to the second sine position encoding unit 630 can be pre-set anchor point information; in other decoding layers, the content input to the second sine position encoding unit 630 can be the predicted trajectory output by the previous decoding layer. In other words, in the first decoding layer, the predicted trajectory is the same as the pre-set anchor point information.

[0117] Furthermore, this disclosure can concatenate the result after integrating the first intermediate result, the anchor position features output by the second sinusoidal position encoding unit 630, and the initial position vector (or zero position embedding (Zero PE)) to obtain the self-attention result of the first decoding layer (i.e., Figure 6 (as shown in q'). In one example, the initial position vector can be a zero vector form of the initial position vector, which serves to provide a uniform benchmark for the self-attention results when the decoding layer starts operating.

[0118] By employing the above method, self-attention computation based on the prediction request vector and pre-defined anchor point information can uncover the intrinsic relationships between elements in the prediction request vector and between the vector and the anchor point information, thereby generating a first intermediate result rich in information. Furthermore, integrating this first intermediate result can optimize the information structure, strengthen key features, and reduce potential redundancy and noise. Finally, the integrated result is concatenated with the anchor point information, providing guiding input for subsequent trajectory prediction and improving the accuracy and stability of trajectory prediction.

[0119] In some implementations, the self-attention unit performs self-attention calculations based on the first input data to obtain a self-attention result, including: The self-attention units of the decoding layers other than the first decoding layer perform self-attention calculations based on the cross-attention results output by the previous decoding layer and the pre-set anchor point information to obtain the second intermediate result; The self-attention units of other decoding layers integrate the second intermediate results and concatenate the integrated results with the predicted trajectory output by the previous decoding layer to obtain the self-attention results of other decoding layers.

[0120] In this embodiment of the disclosure, for decoding layers other than the first decoding layer, the cross-attention result and anchor point position features output by the previous decoding layer can be input into the self-attention calculation subunit 611 to perform self-attention calculation, thereby obtaining the second intermediate result. The self-attention calculation process can refer to the above formula (1).

[0121] It is understood that this disclosure can use the cross-attention result output by the previous decoding layer as the prediction request vector of the current decoding layer to achieve iterative optimization of the input data of the decoding layer.

[0122] Here, the anchor point position features can be determined by inputting pre-defined anchor point information into the first sine position encoding unit 620. In one example, this disclosure can determine the output data of the previous decoding layer (i.e., the prediction request vector and prediction trajectory of the current decoding layer need to be input) based on the cross-attention result of the previous decoding layer. The process of determining the output data of the previous decoding layer will be described in detail later.

[0123] Furthermore, the gated residual subunits 612 of the self-attention units 610 of other decoding layers can integrate the second intermediate results to obtain the integrated result. Moreover, this disclosure can obtain the self-attention result (i.e., q') of other decoding layers based on the integrated result, the predicted trajectory, and the initial position vector. It is understood that during the concatenation process, this disclosure can utilize the second sine position encoding unit 630 to encode the predicted trajectory output by the previous decoding layer to obtain the encoded predicted trajectory, and then concatenate based on the encoded predicted trajectory, the second intermediate result, and the initial position vector to obtain the self-attention result (q') of other decoding layers.

[0124] Using the above method, self-attention calculation is performed based on the cross-attention result output from the previous decoding layer and pre-defined anchor point information. This leverages the interaction information captured by preceding processing and pre-defined spatial prior knowledge, ensuring that the second intermediate result contains both contextual features and reasonable initial position guidance. Furthermore, the integration operation on the second intermediate result optimizes the feature representation, improving its robustness. Concatenating the integrated result with the predicted trajectory output from the previous decoding layer achieves information fusion at different levels. This preserves the continuity of historical predictions while incorporating the processing results of the current decoding layer, providing comprehensive and accurate input for subsequent trajectory prediction.

[0125] The following content details the specific methods for calculating cross-attention (i.e., determining the cross-attention result).

[0126] In some implementations, the cross-attention unit performs cross-attention calculations based on self-attention results, map features, obstacle features, and hard isolation features to obtain cross-attention results, including: The cross-attention unit performs cross-attention calculations on the self-attention result and map features, the self-attention result and obstacle features, and the self-attention result and hard isolation features to obtain three intermediate results. The cross-attention unit splices and merges the three intermediate results to obtain the cross-attention result.

[0127] In the embodiments disclosed herein, such as Figure 6 As shown, the cross attention unit 640 may include a map attention subunit 641, an obstacle attention subunit 642, a hard isolation attention subunit 643, and a stitching and fusion subunit 644.

[0128] In one example, the map attention subunit 641 can be used to perform cross-attention calculation on the self-attention result (q') and map features. Specifically, this disclosure can utilize the first relative position feature and the first semantic feature in the map features to construct a first key-value pair (i.e., k1v1), and then the map attention subunit 641 can perform cross-attention calculation based on the first key-value pair (i.e., k1v1) and the self-attention result (q') to obtain an intermediate result (which can be called the map cross-attention calculation result).

[0129] In another example, the obstacle attention subunit 642 can be used to perform cross-attention calculation on the self-attention result (q') and obstacle features. Specifically, this disclosure can utilize the second relative position feature and the second semantic feature in the obstacle features to construct a second key-value pair (i.e., k2v2), and then the obstacle attention subunit 642 can perform cross-attention calculation based on the second key-value pair (i.e., k2v2) and the self-attention result (q') to obtain an intermediate result (which may be called the obstacle cross-attention calculation result).

[0130] In another example, the hard-isolation attention subunit 643 can be used to perform cross-attention calculation on the self-attention result (q') and the hard-isolation features. Specifically, this disclosure can utilize the hard-isolation position encoding and hard-isolation occupancy semantic features in the hard-isolation features to construct a third key-value pair (i.e., k3v3). Then, the hard-isolation attention subunit 643 can perform cross-attention calculation based on the third key-value pair (i.e., k3v3) and the self-attention result (q') to obtain an intermediate result (which can be called the hard-isolation cross-attention calculation result).

[0131] It is understandable that the above process of cross-attention calculation can be implemented using formula (1).

[0132] In this embodiment of the disclosure, the cross-attention unit 640 may further include a splicing and fusion subunit 644. In one example, the splicing and fusion subunit 644 may include a splicing (Concat) function and a fusion function, the fusion function being implemented via an MLP.

[0133] In this embodiment of the disclosure, the splicing and fusion subunit 644 can splice and fuse three intermediate results (i.e., map cross attention calculation result, obstacle cross attention calculation result, and hard isolation cross attention calculation result) to obtain the cross attention result.

[0134] In one example, such as Figure 6As shown, the cross-attention result may include the prediction request vector output by the current decoding layer. Furthermore, this disclosure may also determine the prediction trajectory output by the current decoding layer based on the cross-attention result. The prediction request vector output by the current decoding layer (i.e., the cross-attention result) and the prediction trajectory output by the current decoding layer together constitute the output data of the current decoding layer.

[0135] Using the above method, this disclosure performs cross-attention calculations on the self-attention results with map features, obstacle features, and hard isolation features, respectively, enabling the decoding layer of the trajectory prediction model to explore the correlation between the self-attention results and features of different types of environments, thus enriching the dimensions of feature representation. Furthermore, by splicing and fusing the three intermediate results generated by the cross-attention calculation, the effective aggregation of multi-source heterogeneous data is achieved, improving the perception and trajectory prediction functions of the trajectory prediction model in complex environments.

[0136] In some implementations, the cross-attention unit determines the output data of the decoding layer based on the cross-attention result, including: The cross-attention unit performs Gaussian mixture decoding on the cross-attention results to obtain the predicted trajectory output by the decoding layer; The output data of the decoding layer includes cross-attention results and predicted trajectories.

[0137] In the embodiments disclosed herein, such as Figure 6 As shown, each decoding layer may also include a Gaussian Mixture Model Decoder 650 (GMM Decoder). This disclosure can utilize the Gaussian Mixture Model Decoder 650 to perform Gaussian mixture decoding on the cross-attention results to obtain the predicted trajectory output by the decoding layer.

[0138] Specifically, this disclosure allows the cross-attention results to be input into a Gaussian mixture decoder 650. The Gaussian mixture decoder 650 can perform in-depth analysis and processing on the cross-attention results to obtain the predicted trajectory output by the decoder. In this process, the Gaussian mixture decoder 650 can mine the multimodal characteristics of the cross-attention results based on the theoretical foundation of the Gaussian mixture model.

[0139] In one example, the Gaussian mixture decoder 650 can generate multimodal predicted trajectories, meaning it can obtain multiple predicted trajectories with different orientations and characteristics. These predicted trajectories reflect the possible motion paths of the target object under different situations or conditions. Furthermore, the Gaussian mixture decoder 650 can assign corresponding probabilities to each generated predicted trajectory. These probability values ​​reflect the likelihood of each predicted trajectory. In practical applications, this disclosure allows the predicted trajectory with the highest probability value to be used as the predicted trajectory output by the decoder.

[0140] Furthermore, the Gaussian mixture decoder 650 can also output position distribution parameters for future moments. These position distribution parameters can describe the range and probability distribution of the target object's possible positions at various future moments. By analyzing these position distribution parameters, the uncertainties and changing trends of the target object during its future movement can be understood.

[0141] In this embodiment of the disclosure, the predicted trajectory output by the decoder is obtained after the Gaussian mixture decoder 650 processes the cross-attention result. The cross-attention result and the predicted trajectory output by the decoder together constitute the output data of the decoder.

[0142] By adopting the above method, Gaussian mixture decoding can utilize the probabilistic modeling capability of Gaussian mixture models to transform abstract cross-attention results into specific predicted trajectories, and can capture the multimodal characteristics of predicted trajectories, thereby improving the accuracy of the prediction model and providing an information foundation for the subsequent training process of the trajectory prediction model.

[0143] The following content details the calculation process of the loss function.

[0144] In some implementations, a loss function is calculated based on the predicted trajectory, hard isolation information, and trajectory labels, including: The predicted trajectory is subjected to kinematic constraints to obtain the constrained trajectory; Based on the constrained trajectory and trajectory label, the trajectory correctness supervision loss is determined; and based on the constrained trajectory and hard isolation information, the trajectory rationality supervision loss is determined. The loss function is calculated based on the trajectory correctness supervision loss and the trajectory rationality supervision loss.

[0145] In the embodiments disclosed herein, such as Figure 6 As shown, the decoding module may also include kinematic constraint units 660. In one example, the kinematic constraint unit 660 can be used to receive the predicted trajectory output from the last decoding layer and apply kinematic constraints to the predicted trajectory to obtain a constrained trajectory. This constrained trajectory conforms to both the prediction logic of the trajectory prediction model and the motion laws of the real world.

[0146] In one example, the kinematic constraint processing may include one or more of the following: velocity continuity, acceleration smoothness, rotation angle encoding range, and curvature constraints. After the kinematic constraint unit 660 performs kinematic constraint processing on the predicted trajectory, components that do not conform to actual kinematic laws can be removed to obtain a constrained trajectory. This constrained trajectory not only theoretically meets the prediction requirements of the trajectory prediction model, but also physically conforms to the motion characteristics of vehicles or traffic participants.

[0147] Furthermore, this disclosure can determine the loss function based on the constraint trajectory, trajectory label, and hard isolation information. In one example, this disclosure can design a multi-dimensional loss function structure, which can mainly include trajectory correctness supervision loss and trajectory rationality supervision loss, etc. The various parts of the loss function are co-optimized to ensure that the trajectory output by the trajectory prediction model (i.e., the final predicted trajectory, such as the constraint trajectory) conforms to the laws of physical motion, avoids hard isolation regions, and maintains the accuracy and motion continuity of the trajectory points.

[0148] In this embodiment of the disclosure, the trajectory correctness supervision loss can be determined based on the constraint trajectory and trajectory label. The trajectory label can include true and correct trajectory information.

[0149] In one example, the trajectory correctness supervision loss aims to ensure the trajectory output by the trajectory prediction model has basic correctness. It can consist of two parts: trajectory classification loss and trajectory regression loss. The trajectory classification loss distinguishes between different motion modes (such as straight, left turn, right turn, etc.), while the trajectory regression loss constrains the deviations in the spatiotemporal coordinates of the trajectory output by the trajectory prediction model. In this example, the trajectory correctness supervision loss can be represented using Euclidean distance (or L2 distance), Gaussian mixture model loss (GMM loss), etc.

[0150] In this embodiment of the disclosure, the trajectory rationality loss can be determined based on the constraint trajectory and hard isolation information.

[0151] Furthermore, this disclosure allows for the calculation of a loss function based on trajectory correctness supervision loss and trajectory rationality supervision loss. The specific calculation methods for trajectory rationality supervision loss and the loss function will be described in detail later.

[0152] By employing the above method, a constrained trajectory is obtained by applying kinematic constraints to the predicted trajectory. This ensures that the constrained trajectory follows the actual kinematic laws of vehicles or traffic participants, enhancing its realism. Furthermore, based on the constrained trajectory, trajectory labels, and hard isolation information, the correctness supervision loss and the reasonableness supervision loss of the trajectory are determined, and the loss function is calculated accordingly. This improves the inference performance of the trajectory prediction model, enhances the correctness and reasonableness of the model's output, and improves the model's trajectory prediction ability and generalization performance in complex traffic scenarios.

[0153] The following content describes in detail how the loss of trajectory rationality supervision is determined.

[0154] In some implementations, hard isolation information is used to indicate the occupied area within a predetermined spatial range that is occupied by a hard-isolated object; Based on constrained trajectories and hard isolation information, the trajectory rationality supervision loss is determined, including: Based on the relative position information of the constrained trajectory and the occupied area, the trajectory rationality monitoring loss is determined.

[0155] In this embodiment of the disclosure, the hard isolation information indicates the occupied area (i.e., hard isolation area) within a predetermined space that is occupied by hard isolation objects (such as guardrails or median strips in the middle of a road). These occupied areas are inaccessible to traffic participants and are considered impassable areas. It is understood that in this disclosure, the concepts of impassable area, hard isolation area, and occupied area are the same.

[0156] In one example, the trajectory rationality supervision loss can consist of occupancy gradient loss and occupancy distance loss, used to guide the trajectory prediction model to actively avoid hard-isolated regions (i.e., occupancy regions within a predetermined spatial range occupied by hard-isolated objects). In this example, this disclosure can determine the occupancy gradient loss and occupancy distance loss based on the relative position information between the constrained trajectory and the occupancy region, thereby determining the trajectory rationality supervision loss.

[0157] In some implementations, the trajectory rationality supervision loss is determined based on the relative position information of the constraint trajectory and the occupied area, including at least one of the following: When a trajectory point in a constrained trajectory is located within an occupied area, the trajectory rationality supervision loss is determined based on the distance between the trajectory point and the edge of the occupied area. When the endpoint of the constrained trajectory is located in an occupied area, the value of the trajectory rationality supervision loss is set to a preset value.

[0158] In this embodiment of the disclosure, when a trajectory point in a constrained trajectory is located within an occupied region, the occupancy gradient loss can be determined based on the distance between the trajectory point and the edge of the occupied region. The core idea of ​​this occupancy gradient loss is to measure the depth to which the constrained trajectory traverses the occupied region. The greater the traversal depth, the greater the distance between the trajectory point and the edge of the occupied region, and consequently, the greater the trajectory rationality supervision loss. In other words, in some implementations, the value of the trajectory rationality supervision loss is positively correlated with this distance.

[0159] Thus, based on the relationship between the trajectory rationality supervision loss and the distance value, the trajectory rationality supervision loss can be determined. This trajectory rationality supervision loss can guide the trajectory prediction model to be penalized more when predicting trajectories that cross occupied areas, thereby enabling the trajectory prediction model to develop the behavior of bypassing occupied areas during the training phase.

[0160] Specifically, in one example, for calculating the trajectory rationality supervision loss, this disclosure can employ gradient algorithms based on morphological erosion or expanded signed distance field (ESDF) to perform gradient processing on the hard isolation information, obtaining a gradient region for the occupied area in the hard isolation information. In the gradient region, the loss gradient at the edge of the region is lower, and the loss gradient at the center of the region is higher, thus forming a spatial penalty for trajectory points located in the occupied region.

[0161] Figure 7 This is a schematic diagram of gradient information for determining occupancy gradient loss according to an embodiment of the present disclosure.

[0162] like Figure 7 As shown, the hard isolation information indicates that the occupied area within a predetermined space, defined by a hard-isolated object, comprises three parts. After gradient processing of the hard isolation information, the gradient information for each occupied area can be determined. Here, the gradient information can be represented by different numbers: smaller numbers indicate a smaller gradient in the occupied area (1 represents the edge of the occupied area), larger numbers indicate a larger gradient in the occupied area, and the part without a number represents the non-occupied area.

[0163] This disclosure can quantify the distance between a trajectory point and the edge of an occupied region from the start to the end point based on the numerical values ​​in the gradient information of the occupancy gradient loss, thereby determining the occupancy gradient loss. That is, the farther the trajectory point is from the edge of the occupied region, the larger the gradient of the trajectory point within the occupied region, and the larger the corresponding occupancy gradient loss. Furthermore, this disclosure can determine the rationality supervision loss based on the occupancy gradient loss.

[0164] In this embodiment of the disclosure, the occupancy distance loss in the trajectory rationality supervision loss can be determined based on the position of the endpoint of the constraint trajectory. This occupancy distance loss can control the endpoint of the constraint trajectory, so that the endpoint of the constraint trajectory can simultaneously meet the two requirements of "approaching the target endpoint (such as the endpoint of the trajectory label)" and "avoiding the occupied area".

[0165] Specifically, this disclosure can determine the coordinates of the endpoint of a constrained trajectory, and based on the coordinates of the endpoint and the coordinates of the occupied area, determine whether the endpoint is located within the occupied area. For example, if the coordinates of the endpoint satisfy the boundary conditions of the occupied area, then the endpoint is inside or at the boundary of the occupied area, and thus it is determined that the endpoint of the constrained trajectory is located within the occupied area.

[0166] Furthermore, once it is determined that the endpoint of the constraint trajectory is located within the occupied area, this disclosure can set the occupancy distance loss to a preset value, and then determine the value of the trajectory rationality loss based on the occupancy distance loss. In one example, if the endpoint of the constraint trajectory is located within the occupied area, the preset value can be infinity.

[0167] In another example, if the endpoint of the constrained trajectory is not within the occupied area, this disclosure can determine the occupancy distance loss based on the endpoint of the constrained trajectory and the target endpoint (such as the endpoint of the trajectory label), and then determine the value of the trajectory rationality loss based on the occupancy distance loss. In this example, the closer the distance between the endpoint of the constrained trajectory and the target endpoint, the smaller the occupancy distance loss.

[0168] In another example, if the endpoint of the constraint trajectory is not within the occupied area, this disclosure can also determine the occupancy distance loss based on the distance between the endpoint of the constraint trajectory and the occupied area, and then determine the value of the trajectory rationality loss based on the occupancy distance loss. In this example, the closer the distance between the endpoint of the constraint trajectory and the boundary of the occupied area, the greater the occupancy distance loss.

[0169] Understandably, the occupancy distance loss can be determined based on the positional relationship between the endpoint of the constraint trajectory and the occupancy area (i.e., whether the endpoint of the constraint trajectory is located within the occupancy area or outside the occupancy area), and / or the distance between the endpoint of the constraint trajectory and the target endpoint.

[0170] Figure 8 This is a schematic diagram of distance information for determining occupancy distance loss according to an embodiment of the present disclosure.

[0171] like Figure 8 As shown, this disclosure can utilize the Breadth First Search (BFS) algorithm to construct an occupancy distance mapping for hard isolation information, thereby realizing the distance measurement between the endpoint of the constrained trajectory and each grid in the hard isolation information. Figure 8 The numbers in the white area represent the occupancy distance loss from each grid cell to the end point of the constraint trajectory; the black area represents the occupancy area. If the end point of the constraint trajectory is located in this area, the occupancy distance loss can be a preset value (such as infinity).

[0172] By using the above method, the trajectory rationality supervision loss is determined by constraining the relative position information of the trajectory and the occupied area. This can guide the trajectory prediction model to learn the irrationality of constraining the trajectory to enter the occupied area during the training process, prompting the model to apply a higher loss to the behavior of crossing the occupied area during inference. In this way, an active "detour" decision-making mode is formed during the training phase, which improves the rationality and security of trajectory prediction.

[0173] In this embodiment, the trajectory rationality loss can be obtained by a linear combination of the occupancy gradient loss and the occupancy distance loss. By adjusting the weights corresponding to these two parts, the "detour intensity" and "trajectory stability" of the constrained trajectory can be balanced, thereby achieving robust control under different scenarios. It should be noted that... Figure 7 and Figure 8 The figures shown are only for illustrating the relative magnitudes of the trajectory rationality supervision loss and are not actual calculated values. This trajectory rationality supervision loss depends on how it is constructed and can be scaled linearly or non-linearly according to actual needs.

[0174] In some implementations, a loss function is calculated based on trajectory correctness supervision loss and trajectory reasonableness supervision loss, including: The loss function is obtained by weighting and summing the trajectory correctness supervision loss, trajectory rationality supervision loss, and hard isolation coding supervision loss; The hard-isolation coding supervision loss characterizes the coding module's semantic understanding ability of hard-isolation information.

[0175] In this embodiment of the disclosure, semantic understanding capability can refer to the encoding module's ability to extract meaningful information from hard-isolated information and understand the actual meaning represented by this information. In one example, the semantic understanding capability of the encoding module can be reflected in the following aspects: (1) Information recognition capability: The encoding module needs to be able to accurately identify the hard isolation area (or occupied area) from the hard isolation information. (2) Feature extraction capability: After identifying the hard isolation region, the encoding module must extract information that can accurately describe the hard isolation features; (3) Semantic association capability: The encoding module also needs to understand the semantic association between hard-isolated information and other information.

[0176] In one example, this disclosure can perform a deconvolution decoding operation on hard-isolated features, ultimately outputting hard-isolated reconstructed values ​​of size [w, h, f]. This step prepares for subsequent calculation of the hard-isolation decoding supervision loss, thereby achieving effective supervision of the hard-isolated features.

[0177] In this embodiment, the trajectory correctness supervision loss, trajectory rationality supervision loss, and hard isolation coding supervision loss are weighted and summed to obtain the loss function. In one example, the formula for calculating the loss function can be: (2) In the above formula, Loss The loss function; Loss can be used to supervise trajectory accuracy. It can be used as the weight for monitoring the trajectory correctness loss; Loss can be monitored to assess the rationality of the trajectory. It can be used as the weight for monitoring the loss of trajectory rationality; Hard isolation coding can be used to supervise the loss. These can be used as weights for hard-isolation encoding supervision loss. Among them, In one example, this disclosure can adjust the above three weights according to actual needs.

[0178] By employing the above method, a loss function is obtained by weighted summation of the trajectory correctness supervision loss, trajectory rationality supervision loss, and hard-isolation coding supervision loss, thus achieving the organic integration of multi-dimensional losses. Using this comprehensive loss function to guide the training of the trajectory generation model can improve the quality of trajectory prediction, thereby enhancing the applicability and reliability of the autonomous driving system.

[0179] In some implementations, it also includes: Decode the hard isolation features to obtain the hard isolation reconstruction values; The hard isolation coding supervision loss is determined based on the difference between the hard isolation reconstructed value and the hard isolation information.

[0180] In this embodiment, to further enhance the trajectory prediction model's understanding of hard isolation semantics, a decoding supervision mechanism is introduced for the encoding results of hard isolation information. During the decoding stage of hard isolation features, a hard isolation reconstruction value of size [w, h, f] is reconstructed and compared pixel-level with the hard isolation information. The hard isolation encoding supervision loss is determined by calculating the difference between the hard isolation reconstruction value and the hard isolation information. Here, the hard isolation encoding supervision loss can be different types of loss values ​​such as cross-entropy loss and focal loss, to effectively constrain the quality of hard isolation feature extraction, thereby improving the encoding module's ability to represent hard isolation regions (i.e., occupied regions) in the hard isolation information.

[0181] Figure 9 This is a flowchart illustrating the construction of hard isolation coded supervision loss according to an embodiment of the present disclosure.

[0182] In one example, this disclosure can add a hard isolation reconstruction supervision branch based on hard isolation features. For example... Figure 9 As shown, this disclosure can input hard isolation information (with dimensions [w, h, f]) into the encoding module to obtain hard isolation features; then, through deconvolution decoding operation, the hard isolation features are decoded into hard isolation reconstructed values ​​(with dimensions also [w, h, f]); then, the hard isolation information and the hard isolation reconstructed values ​​are compared, and the hard isolation encoding supervision loss is determined based on the difference between the two.

[0183] By using the above method, the hard isolation features are decoded to obtain the hard isolation reconstruction value, and the hard isolation encoding supervision loss is determined based on the difference between the reconstruction value and the hard isolation information. This ensures that the hard isolation features retain the occupied boundary, occupied area and spatial distribution information during the compressed representation process, thereby providing accurate feature input for the subsequent trajectory generation process. This improves the ability of the trajectory prediction model to perceive and respond to hard isolation objects in complex road scenarios, and enhances the rationality and security of trajectory prediction.

[0184] It should be noted that the decoding network used to decode hard-isolated features may include multi-layer convolution, deconvolution, upsampling or other reconstruction structures, and this disclosure does not limit the form of the decoding network.

[0185] The following content details the scanline algorithm.

[0186] Figure 10 This is a flowchart illustrating the implementation of a scan line algorithm according to an embodiment of the present disclosure.

[0187] In this embodiment of the disclosure, to improve the continuity and completeness of hard isolation information, a scanline algorithm can be used to complete the road occupancy information. In one example, the specific operation of the scanline algorithm is as follows: First, the polygon of the road hard isolation object is projected onto a sparse grid coordinate system to determine the closed area occupied by the hard isolation object; then, scanlines are constructed one by one along the row or column direction of the road occupancy information, and the intersection point of each scanline with the closed area is calculated, and then these intersection points are sorted according to the coordinate order; finally, based on the principle that intersection points appear in pairs, the grid between each pair of intersection points is determined as a closed area, and it is assigned an occupancy status (that is, the passable grids in the closed area are adjusted to impassable grids), thereby obtaining dense hard isolation information.

[0188] like Figure 10 As shown, the scan line can start from the bottom of the polygon and scan upwards line by line. When each scan line intersects the polygon, the line segments within the polygon area undergo a raster type adjustment operation pixel by pixel. When the scan line reaches the highest point of the polygon and completes the raster type adjustment, the hard isolation information can be determined.

[0189] Since polygons can be defined by a sequence of vertices, their upper and lower bounds on the y-axis can be obtained. Scan lines within these bounds will intersect the polygon. Intuitively, the number of intersections between the scan line and the polygon is even. From left to right, the line segment between an even-numbered point and its next adjacent point belongs to the area within the polygon and requires grid adjustment.

[0190] Figure 11This is a schematic diagram of road occupancy information according to an embodiment of the present disclosure. Figure 11 As shown, the road occupancy information includes three closed regions. After coordinate system transformation and normalization, the intermediate occupancy information also includes these three closed regions. Each closed region includes passable grids (i.e., white grids) and impassable grids (i.e., black grids).

[0191] Figure 12 This is a schematic diagram of hard isolation information according to an embodiment of the present disclosure. Figure 12 As shown, this disclosure can utilize a scanline algorithm to traverse each row or column occupying intermediate information, thereby locating a closed region, adjusting the passable grids in the closed region to impassable grids (or filling the impassable grids completely in the closed region) to determine hard isolation information.

[0192] Based on the scan line algorithm, this disclosure can adjust the grid type of the closed region for hard-isolated objects. After the adjustment is completed, the inside of the closed region of the hard-isolated information is a solid region.

[0193] Figure 13 This is a technical flowchart of a training trajectory prediction model according to an embodiment of the present disclosure. Figure 13 As shown, the process of training a trajectory prediction model can include four stages: input and preprocessing stage, multimodal feature encoding stage, trajectory prediction and feature reconstruction stage, and model training stage.

[0194] During the input and preprocessing stage, this disclosure can obtain obstacle information, map information, and road occupancy information, and then preprocess the road occupancy information to obtain hard isolation information.

[0195] In the multimodal feature encoding stage, this disclosure can input obstacle information into obstacle information encoder 1301, map information into map information encoder 1302, and hard isolation information into hard isolation information encoder 1303 to obtain obstacle features, map features, and hard isolation features respectively. It is understood that obstacle information encoder 1301, map information encoder 1302, and hard isolation information encoder 1303 can collectively constitute the encoding module described above in this disclosure.

[0196] In the trajectory prediction and feature reconstruction stage, this disclosure can input obstacle features, map features and hard isolation features into the decoding module 1304 to obtain the constrained trajectory; at the same time, this disclosure can input hard isolation features into the hard isolation reconstruction decoder 1305 to obtain the hard isolation reconstruction value.

[0197] During the model training phase, this disclosure can construct a trajectory correctness supervision loss based on constrained trajectories and trajectory labels, and a trajectory reasonableness supervision loss based on constrained trajectories and hard isolation information. Simultaneously, a hard isolation encoding supervision loss is constructed based on hard isolation information and hard isolation reconstruction values. In one example, this disclosure can perform a weighted summation of the trajectory correctness supervision loss, trajectory reasonableness supervision loss, and hard isolation encoding supervision loss to obtain the loss function used to train the trajectory prediction model.

[0198] It should be noted that the above-mentioned encoding modules (i.e., obstacle information encoder 1301, map information encoder 1302 and hard isolation information encoder 1303) and decoding module 1304 together constitute the trajectory prediction model.

[0199] In the embodiments of this disclosure, the trajectory prediction model can output a set of trajectories. Specifically, for the trajectory prediction function in an autonomous driving system, the trajectory prediction model can output information such as the multimodal trajectory of each traffic participant over a future period and the probability of each trajectory; for the path planning function in an autonomous driving system, the trajectory prediction model can output data such as the temporal coordinates, speed, acceleration, and curvature of the vehicle's future trajectory; and for intelligent driving system simulation, it can output multi-obstacle interaction trajectories for closed-loop simulation. Through the trajectory prediction model training method proposed in this disclosure, the number of trajectories crossing hard-isolated areas can be reduced in trajectory prediction tasks, improving the rationality and safety of the trajectories.

[0200] Compared with the prior art, this disclosure has the following advantages: (1) A hard isolation interactive modeling mechanism is introduced in the trajectory prediction stage, which can actively achieve detour and reduce the situation where the trajectory crosses the hard isolation area; (2) By combining the occupancy gradient loss and occupancy distance loss, the spatial rationality of the predicted trajectory is improved; (3) This disclosure supports a multi-model general structure from prediction to planning to simulation, which can flexibly adapt to different task requirements; (4) This disclosure improves the safety, stability and interpretability of trajectory prediction models in complex road scenarios.

[0201] The prediction model trained using the model training method proposed in this application can be applied to various scenarios. For example, it can be applied to the prediction unit of an autonomous vehicle to output the multimodal trajectory and probability of each traffic participant over a future period; it can also be applied to the planning unit of an autonomous vehicle to output the temporal coordinates, speed, acceleration, and curvature of the vehicle's future trajectory; and it can also be used in intelligent obstacle simulation scenarios to output multi-obstacle interaction trajectories for closed-loop simulation.

[0202] This disclosure also proposes an autonomous driving trajectory prediction model training device. Figure 14 This is a schematic diagram of the structure of an autonomous driving trajectory prediction model training device 1400 according to an embodiment of the present disclosure, including: The information input module 1410 is used to input map information, obstacle information and hard isolation information into the encoding module of the trajectory prediction model. The encoding module encodes the map information, obstacle information and hard isolation information respectively to obtain map features, obstacle features and hard isolation features. The trajectory prediction module 1420 is used to decode the trajectory prediction model based on the prediction request vector, map features, obstacle features and hard isolation features to obtain the predicted trajectory. The model training module 1430 is used to calculate the loss function based on the predicted trajectory, hard isolation information and trajectory label, and to adjust the parameters of the trajectory prediction model based on the loss function.

[0203] In some implementations, the model training module 1430 is used for: The predicted trajectory is subjected to kinematic constraints to obtain the constrained trajectory; Based on the constrained trajectory and trajectory label, the trajectory correctness supervision loss is determined; and based on the constrained trajectory and hard isolation information, the trajectory rationality supervision loss is determined. The loss function is calculated based on the trajectory correctness supervision loss and the trajectory rationality supervision loss.

[0204] In some implementations, hard isolation information is used to indicate the occupied area within a predetermined spatial range that is occupied by a hard-isolated object; Model training module 1430 is used for: Based on the relative position information of the constrained trajectory and the occupied area, the trajectory rationality monitoring loss is determined.

[0205] In some implementations, the model training module 1430 is used for at least one of the following: When a trajectory point in a constrained trajectory is located within an occupied area, the trajectory rationality supervision loss is determined based on the distance between the trajectory point and the edge of the occupied area. When the endpoint of the constrained trajectory is located in an occupied area, the value of the trajectory rationality supervision loss is set to a preset value.

[0206] In some implementations, the value of the trajectory rationality supervision loss is positively correlated with the value of the distance.

[0207] In some implementations, the model training module 1430 is used for: The loss function is obtained by weighting and summing the trajectory correctness supervision loss, trajectory rationality supervision loss, and hard isolation coding supervision loss; The hard-isolation coding supervision loss characterizes the coding module's semantic understanding ability of hard-isolation information.

[0208] In some implementations, the model training module 1430 is further configured to: Decode the hard isolation features to obtain the hard isolation reconstruction values; The hard isolation coding supervision loss is determined based on the difference between the hard isolation reconstructed value and the hard isolation information.

[0209] In some implementations, the information input module 1410 is used for: The encoding module extracts features from map elements in the map information to obtain the first semantic features; The encoding module determines the first relative relationship between the requested location indicated by the predicted request vector and the location of the map feature, and determines the first relative location feature based on the first relative relationship; The map features include first semantic features and first relative position features.

[0210] In some implementations, the information input module 1410 is used for: The encoding module extracts features from obstacle information to obtain second semantic features; The encoding module determines a second relative relationship between the request position indicated by the predicted request vector and the obstacle position, and determines a second relative position feature based on the second relative relationship; Obstacle features include second semantic features and second relative position features.

[0211] In some implementations, the information input module 1410 is used for: The encoding module encodes the hard isolation information, including: The encoding module extracts hard-isolation occupancy semantic features from the hard-isolation information; Determine the position information of each word in the semantic features occupied by hard isolation within a predetermined spatial range, and determine the hard isolation position encoding based on the position information; the predetermined spatial range is indicated by the hard isolation information. Hard isolation features include hard isolation occupancy semantic features and hard isolation location encoding.

[0212] In some implementations, the information input module 1410 is used for: The hard isolation information is segmented to obtain multiple image patches; The feature vector and position vector of each image patch are determined, and the feature vector and the corresponding position vector of each image patch are concatenated to obtain the spatial feature vector; Multiple spatial feature vectors are input into the encoding module, which then extracts the semantic features of hard isolation occupancy.

[0213] In some implementations, each decoding layer includes a self-attention unit and a cross-attention unit; Trajectory prediction module 1420, used for: The self-attention unit performs self-attention calculations based on the first input data to obtain the self-attention result; The cross-attention unit performs cross-attention calculations based on self-attention results, map features, obstacle features, and hard isolation features to obtain cross-attention results; and determines the output data of the decoding layer based on the cross-attention results.

[0214] In some implementations, the trajectory prediction module 1420 is used for: The cross-attention unit performs cross-attention calculations on the self-attention result and map features, the self-attention result and obstacle features, and the self-attention result and hard isolation features to obtain three intermediate results. The cross-attention unit splices and merges the three intermediate results to obtain the cross-attention result.

[0215] In some implementations, the trajectory prediction module 1420 is used for: The cross-attention unit performs Gaussian mixture decoding on the cross-attention results to obtain the predicted trajectory output by the decoding layer; The output data of the decoding layer includes cross-attention results and predicted trajectories.

[0216] In some implementations, the trajectory prediction module 1420 is used for: The self-attention unit of the first decoding layer performs self-attention calculation based on the prediction request vector and pre-set anchor information to obtain the first intermediate result; The self-attention unit of the first decoding layer integrates the first intermediate result and concatenates the integrated result with the pre-set anchor information to obtain the self-attention result of the first decoding layer.

[0217] In some implementations, the trajectory prediction module 1420 is used for: The self-attention units of the decoding layers other than the first decoding layer perform self-attention calculations based on the cross-attention results output by the previous decoding layer and the pre-set anchor point information to obtain the second intermediate result; The self-attention units of other decoding layers integrate the second intermediate results and concatenate the integrated results with the predicted trajectory output by the previous decoding layer to obtain the self-attention results of other decoding layers.

[0218] In some embodiments, this disclosure also proposes an autonomous driving trajectory prediction model training device. Figure 15 This is a schematic diagram of the structure of an autonomous driving trajectory prediction model training device 1500 according to an embodiment of the present disclosure, which further includes: Preprocessing module 1540 is used to determine road occupancy information and transform the coordinate system of the road occupancy information to the main vehicle coordinate system; The road occupancy information after coordinate system transformation is normalized into intermediate occupancy information of a preset size. The intermediate occupancy information includes at least one closed region used to characterize the occupancy of objects that are hard-isolated. The closed region includes passable grids and different passable grids. Adjust passable grids within a closed area to impassable grids to determine hard isolation information.

[0219] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0220] The acquisition, storage, and application of personal information by users involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0221] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0222] Figure 16 A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0223] like Figure 16As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1602 or a computer program loaded into random access memory (RAM) 1603 from storage unit 1608. The RAM 1603 may also store various programs and data required for the operation of device 1600. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0224] Multiple components in device 1600 are connected to I / O interface 1605, including: input unit 1606, such as keyboard, mouse, etc.; output unit 1607, such as various types of monitors, speakers, etc.; storage unit 1608, such as disk, optical disk, etc.; and communication unit 1609, such as network card, modem, wireless transceiver, etc. Communication unit 1609 allows device 1600 to exchange / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0225] The computing unit 1601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as the autonomous driving trajectory prediction model training method. For example, in some embodiments, the autonomous driving trajectory prediction model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1600 via ROM 1602 and / or communication unit 1609. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, one or more steps of the autonomous driving trajectory prediction model training method described above can be performed. Alternatively, in other embodiments, computing unit 1601 may be configured to perform an autonomous driving trajectory prediction model training method by any other suitable means (e.g., by means of firmware).

[0226] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0227] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0228] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0229] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0230] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0231] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0232] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0233] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training an autonomous driving trajectory prediction model, comprising: Map information, obstacle information, and hard isolation information are input into the encoding module of the trajectory prediction model. The encoding module encodes the map information, obstacle information, and hard isolation information respectively to obtain map features, obstacle features, and hard isolation features. The decoding module of the trajectory prediction model decodes the predicted trajectory based on the prediction request vector, the map features, the obstacle features, and the hard isolation features. A loss function is calculated based on the predicted trajectory, the hard isolation information, and the trajectory label, and the parameters of the trajectory prediction model are adjusted based on the loss function.

2. The method according to claim 1, wherein, The calculation of the loss function based on the predicted trajectory, the hard isolation information, and the trajectory label includes: The predicted trajectory is subjected to kinematic constraints to obtain a constrained trajectory; Based on the constrained trajectory and the trajectory label, determine the trajectory correctness supervision loss; and based on the constrained trajectory and the hard isolation information, determine the trajectory rationality supervision loss. The loss function is calculated based on the trajectory correctness supervision loss and the trajectory rationality supervision loss.

3. The method according to claim 2, wherein, The hard isolation information is used to indicate the occupied area within a predetermined space that is occupied by a hard-isolated object; The step of determining the trajectory rationality supervision loss based on the constrained trajectory and the hard isolation information includes: The trajectory rationality monitoring loss is determined based on the relative position information of the constraint trajectory and the occupied area.

4. The method according to claim 3, wherein, The step of determining the trajectory rationality supervision loss based on the relative position information of the constraint trajectory and the occupied area includes at least one of the following: If a trajectory point in the constrained trajectory is located within the occupied area, the trajectory rationality supervision loss is determined based on the distance between the trajectory point and the edge of the occupied area. If the endpoint of the constrained trajectory is located in the occupied area, the value of the trajectory rationality supervision loss is set to a preset value.

5. The method according to claim 4, wherein, The value of the trajectory rationality supervision loss is positively correlated with the value of the distance.

6. The method according to any one of claims 2-5, wherein, The calculation of the loss function based on the trajectory correctness supervision loss and the trajectory reasonableness supervision loss includes: The loss function is obtained by weighted summation of the trajectory correctness supervision loss, the trajectory rationality supervision loss, and the hard isolation coding supervision loss; The hard isolation coding supervision loss characterizes the semantic understanding ability of the coding module for the hard isolation information.

7. The method according to claim 6, further comprising: The hard isolation features are decoded to obtain the hard isolation reconstruction values; The hard isolation coding supervision loss is determined based on the difference between the hard isolation reconstructed value and the hard isolation information.

8. The method according to any one of claims 1-7, wherein, The encoding module encodes the map information, including: The encoding module extracts features from the map elements in the map information to obtain the first semantic features; The encoding module determines a first relative relationship between the requested location indicated by the predicted request vector and the location of the map element, and determines a first relative location feature based on the first relative relationship; The map features include the first semantic feature and the first relative position feature.

9. The method according to any one of claims 1-7, wherein, The encoding module encodes the obstacle information, including: The encoding module extracts features from the obstacle information to obtain second semantic features; The encoding module determines a second relative relationship between the request position indicated by the predicted request vector and the obstacle position, and determines a second relative position feature based on the second relative relationship; The obstacle features include the second semantic features and the second relative position features.

10. The method according to any one of claims 1-7, wherein, The encoding module encodes the hard isolation information, including: The encoding module extracts hard isolation occupancy semantic features from the hard isolation information; The position information of each word in the hard isolation semantic feature is determined within a predetermined spatial range, and the hard isolation position code is determined based on the position information; the predetermined spatial range is indicated by the hard isolation information. The hard isolation feature includes the hard isolation occupancy semantic feature and the hard isolation location encoding.

11. The method according to claim 10, wherein, The encoding module extracts hard isolation occupancy semantic features from the hard isolation information, including: The hard isolation information is segmented to obtain multiple image blocks; The feature vector and position vector of each image block are determined, and the feature vector and the corresponding position vector of each image block are concatenated to obtain a spatial feature vector; Multiple spatial feature vectors are input into the encoding module, which then extracts the hard isolation occupancy semantic features.

12. The method according to any one of claims 1-11, wherein, The decoding module includes multiple decoding layers connected in sequence, with the last decoding layer outputting the predicted trajectory. Each of the decoding layers performs decoding based on the first input data, the map features, the obstacle features, and the hard isolation features; wherein, the first input data of the first decoding layer includes the prediction request vector, and the first input data of the other decoding layers includes the output data of the previous decoding layer.

13. The method according to claim 12, wherein, Each of the aforementioned decoding layers includes a self-attention unit and a cross-attention unit; The decoding layer performs decoding based on the first input data, the map features, the obstacle features, and the hard isolation features, including: The self-attention unit performs self-attention calculation based on the first input data to obtain the self-attention result; The cross-attention unit performs cross-attention calculations based on the self-attention result, the map features, the obstacle features, and the hard isolation features to obtain the cross-attention result; and determines the output data of the decoding layer based on the cross-attention result.

14. The method according to claim 13, wherein, The cross-attention unit performs cross-attention calculations based on the self-attention result, the map features, the obstacle features, and the hard isolation features to obtain the cross-attention result, including: The cross-attention unit performs cross-attention calculations on the self-attention result and the map features, the self-attention result and the obstacle features, and the self-attention result and the hard isolation features to obtain three intermediate results. The cross-attention unit splices and merges the three intermediate results to obtain the cross-attention result.

15. The method according to claim 13 or 14, wherein, The cross-attention unit determines the output data of the decoding layer based on the cross-attention result, including: The cross-attention unit performs Gaussian mixture decoding on the cross-attention result to obtain the predicted trajectory output by the decoding layer; The output data of the decoding layer includes the cross-attention result and the predicted trajectory.

16. The method according to any one of claims 13-15, wherein, The self-attention unit performs self-attention calculations based on the first input data to obtain a self-attention result, including: The self-attention unit of the first decoding layer performs self-attention calculation based on the prediction request vector and the pre-set anchor information to obtain a first intermediate result; The self-attention unit of the first decoding layer integrates the first intermediate result and concatenates the integrated result with the pre-set anchor information to obtain the self-attention result of the first decoding layer.

17. The method according to any one of claims 13-15, wherein, The self-attention unit performs self-attention calculations based on the first input data to obtain a self-attention result, including: The self-attention units of the other decoding layers besides the first decoding layer perform self-attention calculations based on the cross-attention results output by the previous decoding layer and the pre-set anchor point information to obtain the second intermediate result; The self-attention units of the other decoding layers integrate the second intermediate result and concatenate the integrated result with the predicted trajectory output by the previous decoding layer to obtain the self-attention result of the other decoding layers.

18. The method according to any one of claims 1-17, further comprising, before inputting the map information, obstacle information, and hard isolation information into the encoding module of the trajectory prediction model: Determine road occupancy information and transform the coordinate system of the road occupancy information to the vehicle coordinate system; The road occupancy information after coordinate system transformation is normalized into intermediate occupancy information of a preset size. The intermediate occupancy information includes at least one closed region used to characterize the occupancy of a hard-isolated object. The closed region includes a passable grid and different passable grids. The passable grids within the closed area are adjusted to impassable grids to determine the hard isolation information.

19. An autonomous driving trajectory prediction model training device, comprising: The information input module is used to input map information, obstacle information and hard isolation information into the encoding module of the trajectory prediction model. The encoding module encodes the map information, obstacle information and hard isolation information respectively to obtain map features, obstacle features and hard isolation features. The trajectory prediction module is used to decode the trajectory prediction model based on the prediction request vector, the map features, the obstacle features, and the hard isolation features to obtain the predicted trajectory. The model training module is used to calculate a loss function based on the predicted trajectory, the hard isolation information, and the trajectory label, and to adjust the parameters of the trajectory prediction model based on the loss function.

20. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-18.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-18.

22. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-18.

Images