Trajectory prediction method and device, electronic equipment and computer readable storage medium

Abstract

The application relates to a trajectory prediction method and device, electronic equipment and a computer readable storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a semantic map corresponding to a prediction target and a historical observation trajectory sequence corresponding to the prediction target, the historical observation trajectory sequence representing the trajectory position of the prediction target at each historical moment; then, the historical observation trajectory sequence corresponding to the prediction target is represented by sequence coding to obtain sequence coding features; a function representation corresponding to the semantic map is generated based on the semantic map corresponding to the prediction target, the function representation corresponding to the semantic map representing the mapping relationship between semantics and trajectory positions; then, trajectory prediction is performed based on the sequence coding features and by using the function representation corresponding to the semantic map to obtain a future prediction trajectory sequence. The application relates to a trajectory prediction method and device, electronic equipment and a computer readable storage medium, which can improve the vehicle trajectory prediction accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a trajectory prediction method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Trajectory prediction is currently mainly used in autonomous driving. When the perception system completes its work, the acquired information is needed as prior knowledge to guide subsequent planning tasks. Perception information is collected from historical data and the current moment, while planning tasks require planning vehicle operations over a future period. Therefore, perception information cannot be used directly and effectively; higher-level algorithms are needed to understand existing prior knowledge and transform perception information into other forms to help the autonomous driving system better complete its planning tasks. Trajectory prediction algorithms are one such high-level algorithm.

[0003] Currently, trajectory prediction methods primarily utilize deep learning techniques, such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and Gated Recurrent Units (GRUs), to extract sequence features. These methods are then embedded as components into the encoder-decoder network structure to achieve future trajectory prediction. Meanwhile, high-definition maps or road semantic maps, as important prior knowledge, can provide more feature inputs and are also crucial factors in trajectory prediction. Therefore, effectively extracting and encoding features from high-definition maps is of great significance for improving the accuracy of vehicle trajectory prediction. Summary of the Invention

[0004] The purpose of this application is to provide a trajectory prediction method, apparatus, electronic device, and computer-readable storage medium to solve at least one of the above-mentioned technical problems.

[0005] The above-mentioned objective of this application is achieved through the following technical solution:

[0006] Firstly, a trajectory prediction method is provided, including:

[0007] Obtain the semantic map corresponding to the predicted target and the historical observation trajectory sequence corresponding to the predicted target, wherein the historical observation trajectory sequence represents the trajectory position of the predicted target at each historical moment;

[0008] The historical observation trajectory sequence corresponding to the predicted target is represented by sequence encoding to obtain sequence encoding features;

[0009] Based on the semantic map corresponding to the predicted target, a semantic map corresponding function representation is generated, wherein the semantic map corresponding function representation characterizes the mapping relationship between semantics and trajectory position;

[0010] Based on the sequence encoding features and the function representation corresponding to the semantic map, trajectory prediction is performed to obtain the future predicted trajectory sequence.

[0011] In one possible implementation, the step of performing sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features includes:

[0012] Based on the historical observation trajectory sequence corresponding to the predicted target, a position sequence, an acceleration sequence, a velocity sequence, and an anomaly indication sequence are determined. The anomaly indication sequence is used to characterize the frame loss situation in the historical observation trajectory sequence.

[0013] The position sequence, acceleration sequence, velocity sequence, and anomaly indication sequence are each represented by spatial embedding to obtain their respective spatial embedding representations;

[0014] Position encoding is performed based on the corresponding spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features;

[0015] Sequence coding features are obtained by extracting sequence features based on multiple attention heads, the position coding features, the acceleration coding features, the velocity coding features, and the anomaly indication coding features.

[0016] In another possible implementation, the position encoding based on the respective spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features includes:

[0017] Obtain the position encoding parameter matrix;

[0018] Based on the position encoding parameter matrix and the corresponding spatial embedding representation, the position encoding feature, the acceleration encoding feature, the velocity encoding feature, and the anomaly indication encoding feature are determined.

[0019] In another possible implementation, the sequence encoding features are obtained by extracting sequence features using multiple attention heads and based on the position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features, including:

[0020] Each encoded feature is mapped through various attention heads to obtain the query feature, value feature, and key feature corresponding to each attention head.

[0021] Based on the query features, value features, and key features corresponding to each attention head, the attention features corresponding to each encoded feature under each attention head are generated.

[0022] The attention features corresponding to each encoded feature under each attention head are matrix-connected;

[0023] The features after matrix concatenation are processed through a fully connected layer to obtain the temporal latent features corresponding to each encoded feature;

[0024] The temporal hidden features corresponding to each coding feature are concatenated to obtain the sequence coding features.

[0025] In another possible implementation, the semantic map corresponding to the predicted target is represented by a vector representation of the semantic map;

[0026] Based on the semantic map corresponding to the predicted target, a semantic map corresponding to a function representation is generated, including:

[0027] The vector representation corresponding to the semantic map is embedded and positionally encoded.

[0028] The position-encoded vector representation is linearly mapped to the value channel and the key channel to obtain the key feature and the value feature;

[0029] Based on the key features and the value features, a function representation corresponding to the semantic map is generated.

[0030] In another possible implementation, generating the functional representation corresponding to the semantic map based on the key features and value features includes:

[0031] The key features are standardized to obtain standardized key features;

[0032] Based on the value features and the standard post-processed key features, the location semantics are determined, and the location semantics represent the semantics of each location in the semantic map;

[0033] Based on the value features, semantic locations are determined, whereby each semantic location represents the position of each semantic term in the semantic map.

[0034] Based on the location semantics and the semantic location, the function representation corresponding to the semantic map is determined.

[0035] In another possible implementation, the step of generating a semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target further includes:

[0036] In the semantic map corresponding to the predicted target, obtain the orientation information of the predicted target at the last moment of historical observation;

[0037] Based on the orientation information of the predicted target at the last moment of historical observation, the semantic map corresponding to the predicted target is rotated and cropped.

[0038] The step of generating a semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target includes:

[0039] Based on the cropped semantic map, a function representation corresponding to the semantic map is generated.

[0040] In another possible implementation, the step of predicting the future trajectory sequence based on the sequence-encoded features and the function representation corresponding to the semantic map includes:

[0041] Based on the sequence encoding features and through the function representation corresponding to the semantic map, the map interaction features are determined;

[0042] Trajectory prediction is performed based on the sequence encoding features and the map interaction features to obtain the future predicted trajectory sequence.

[0043] In another possible implementation, determining map interaction features based on the sequence encoding features and through the function representation corresponding to the semantic map includes:

[0044] The sequence-encoded features are mapped to the input of the function representation corresponding to the semantic map to obtain the mapped features;

[0045] The mapped features are input into the function representation corresponding to the semantic map to obtain the map interaction features.

[0046] In another possible implementation, the step of predicting the trajectory based on the sequence encoding features and the map interaction features to obtain the future predicted trajectory sequence includes:

[0047] The sequence encoding features and the map interaction features are concatenated to obtain the concatenated features.

[0048] Based on the connected features, a future predicted trajectory sequence is obtained through multiple iterative processes.

[0049] In a second aspect, a trajectory prediction device is provided, comprising: a first acquisition module, configured to acquire a semantic map corresponding to a prediction target and a historical observation trajectory sequence corresponding to the prediction target, wherein the historical observation trajectory sequence characterizes the trajectory position of the prediction target at each historical moment;

[0050] The sequence encoding module is used to perform sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features;

[0051] The generation module is used to generate a semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target. The semantic map corresponding to the semantic map represents the mapping relationship between semantics and trajectory position.

[0052] The trajectory prediction module is used to predict the future trajectory sequence based on the sequence encoding features and the function representation corresponding to the semantic map.

[0053] In one possible implementation, when the sequence encoding module performs sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features, it is specifically used for:

[0054] Based on the historical observation trajectory sequence corresponding to the predicted target, a position sequence, an acceleration sequence, a velocity sequence, and an anomaly indication sequence are determined. The anomaly indication sequence is used to characterize the frame loss situation in the historical observation trajectory sequence.

[0055] The position sequence, acceleration sequence, velocity sequence, and anomaly indication sequence are each represented by spatial embedding to obtain their respective spatial embedding representations;

[0056] Position encoding is performed based on the corresponding spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features;

[0057] Sequence coding features are obtained by extracting sequence features based on multiple attention heads, the position coding features, the acceleration coding features, the velocity coding features, and the anomaly indication coding features.

[0058] In another possible implementation, when the sequence encoding module performs position encoding based on the respective corresponding spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features, it is specifically used for:

[0059] Obtain the position encoding parameter matrix;

[0060] Based on the position encoding parameter matrix and the corresponding spatial embedding representation, the position encoding feature, the acceleration encoding feature, the velocity encoding feature, and the anomaly indication encoding feature are determined.

[0061] In another possible implementation, the sequence coding module extracts sequence features by using multiple attention heads and based on the position coding features, acceleration coding features, velocity coding features, and anomaly indication coding features to obtain sequence coding features, specifically used for:

[0062] Each encoded feature is mapped through various attention heads to obtain the query feature, value feature, and key feature corresponding to each attention head.

[0063] Based on the query features, value features, and key features corresponding to each attention head, the attention features corresponding to each encoded feature under each attention head are generated.

[0064] The attention features corresponding to each encoded feature under each attention head are matrix-connected;

[0065] The features after matrix concatenation are processed through a fully connected layer to obtain the temporal latent features corresponding to each encoded feature;

[0066] The temporal hidden features corresponding to each coding feature are concatenated to obtain the sequence coding features.

[0067] In another possible implementation, the semantic map corresponding to the predicted target is represented by a vector representation of the semantic map;

[0068] When the generation module generates a semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target, it is specifically used for:

[0069] The vector representation corresponding to the semantic map is embedded and positionally encoded.

[0070] The position-encoded vector representation is linearly mapped to the value channel and the key channel to obtain the key feature and the value feature;

[0071] Based on the key features and the value features, a function representation corresponding to the semantic map is generated.

[0072] In another possible implementation, when the generation module generates the function representation corresponding to the semantic map based on the key features and value features, it is specifically used for:

[0073] The key features are standardized to obtain standardized key features;

[0074] Based on the value features and the standard post-processed key features, the location semantics are determined, and the location semantics represent the semantics of each location in the semantic map;

[0075] Based on the value features, semantic locations are determined, whereby each semantic location represents the position of each semantic term in the semantic map.

[0076] Based on the location semantics and the semantic location, the function representation corresponding to the semantic map is determined.

[0077] In another possible implementation, the device further includes: a second acquisition module and a rotational cutting module, wherein,

[0078] The second acquisition module is used to acquire the orientation information of the predicted target at the last moment of historical observation in the semantic map corresponding to the predicted target;

[0079] The rotation and cropping module is used to rotate and crop the semantic map corresponding to the predicted target based on the orientation information of the predicted target at the last moment of historical observation.

[0080] Specifically, when the generation module generates the semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target, it is used for:

[0081] Based on the cropped semantic map, a function representation corresponding to the semantic map is generated.

[0082] In another possible implementation, when the trajectory prediction module performs trajectory prediction based on the sequence encoding features and through the function representation corresponding to the semantic map to obtain the future predicted trajectory sequence, it is specifically used for:

[0083] Based on the sequence encoding features and through the function representation corresponding to the semantic map, the map interaction features are determined;

[0084] Trajectory prediction is performed based on the sequence encoding features and the map interaction features to obtain the future predicted trajectory sequence.

[0085] In another possible implementation, when the trajectory prediction module determines map interaction features based on the sequence encoding features and through the function representation corresponding to the semantic map, it is specifically used for:

[0086] The sequence-encoded features are mapped to the input of the function representation corresponding to the semantic map to obtain the mapped features;

[0087] The mapped features are input into the function representation corresponding to the semantic map to obtain the map interaction features.

[0088] In another possible implementation, when the trajectory prediction module performs trajectory prediction based on the sequence encoding features and the map interaction features to obtain a future predicted trajectory sequence, it is specifically used for:

[0089] The sequence encoding features and the map interaction features are concatenated to obtain the concatenated features.

[0090] Based on the connected features, a future predicted trajectory sequence is obtained through multiple iterative processes.

[0091] Thirdly, an electronic device is provided, the electronic device comprising:

[0092] One or more processors;

[0093] Memory;

[0094] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, the applications being configured to: perform operations corresponding to trajectory predictions as shown in any possible implementation of the first aspect.

[0095] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the trajectory prediction method as shown in any possible implementation of the first aspect.

[0096] In summary, this application includes at least one of the following beneficial technical effects:

[0097] This application provides a trajectory prediction method, apparatus, electronic device, and computer-readable storage medium. Compared with related technologies, in this application, when predicting future trajectories, prediction is performed based on the functional representation corresponding to the semantic map (the functional relationship between semantics and trajectory position) and the sequence encoding representation corresponding to the historical trajectory sequence. Since the functional relationship can cover spatial location and road semantic information, and the sequence encoding feature characterizes the trend of historical trajectories, that is, trajectory prediction is performed based on the sequence encoding corresponding to the historical trajectory sequence and the functional representation corresponding to the semantic map, so that the predicted trajectory satisfies the spatial location and road semantic constraints, thereby improving the accuracy of trajectory prediction for the target. Attached Figure Description

[0098] Figure 1 This is a schematic flowchart of a trajectory prediction method according to an embodiment of this application;

[0099] Figure 2 This is a schematic diagram of the encoder processing flow according to an embodiment of this application;

[0100] Figure 3 This is a schematic diagram illustrating the process of the decoder predicting future trajectories according to an embodiment of this application;

[0101] Figure 4 This is a schematic diagram of the structure of a trajectory prediction device according to an embodiment of this application;

[0102] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0103] The present application will be further described in detail below with reference to the accompanying drawings.

[0104] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0105] The trajectory prediction methods involved in related technologies mainly suffer from the following problems:

[0106] (1) Lack of efficient semantic representation and feature extraction methods: Mainstream methods use network models with many parameters and low forward inference efficiency, which cannot complete efficient road semantic feature extraction in trajectory prediction tasks with high real-time requirements.

[0107] (2) Lack of interactive feature representation between map semantics and trajectory trend: Map semantic features need to be interactively modeled with trajectory trend features to highlight the different impacts of map semantics at different locations on future predictions.

[0108] This application aims to design a general and highly generalizable vehicle trajectory prediction method that can accurately predict the future trajectory of vehicles in complex road semantic environments. Firstly, addressing problem (1), this application proposes using a multi-channel high-precision semantic map as prior knowledge, applicable to various types of semantic map inputs, thus providing more input features for the model. By designing a high-performance road semantic feature extraction module, fewer model parameters and forward inference computations are used to map high-precision map information into a fixed-size matrix function, reducing the time complexity of forward computation to a minimum. To address the problem (2) lack of interactive feature representation between map semantics and trajectory trends, this application proposes mapping trajectory trend features to addressing features and using the addressing features as input to the matrix function representing the high-precision map. Finally, the function output yields the interactive feature output between trajectory features and map semantics.

[0109] Furthermore, this application also provides a vehicle trajectory prediction method based on high-performance map interactive coding. By efficiently encoding a high-precision map into a function representation, and using historical trajectory trend features extracted through a multi-head attention mechanism as function input, high-performance interactive coding between the high-precision map and historical trajectories is achieved. This method reduces model parameters while ensuring feature extraction efficiency when encoding complex high-precision maps, thus improving the training and inference efficiency of the model. The entire model, based on the encoder-decoder network, uses a multi-head attention mechanism and a gated recurrent unit (GRU) as the core of sequence encoding and decoding, while adding dynamic model constraints, ultimately forming a complete, highly generalizable, and computationally efficient vehicle trajectory prediction method.

[0110] Specifically, in the embodiments of this application, the influence of various types of semantic maps can be considered when predicting vehicle trajectories. High-precision semantic maps can provide a large amount of prior knowledge in vehicle trajectory prediction tasks, ensuring that the predicted trajectories are distributed within the specified areas, thus guaranteeing the feasibility of the predicted trajectories. Furthermore, since alarm semantic maps can be used to consider the influence of the surrounding road environment, they guide the model to generate more accurate predicted trajectories.

[0111] Furthermore, this application also proposes a map coding method with higher computational efficiency. The high-precision map coding process usually requires a feature extraction module with many parameters, and such methods have low computational efficiency.

[0112] Furthermore, this application embodiment also proposes to address and map multi-channel semantic maps, and to summarize map features through linear transformation, encoding high-precision map information within a fixed area into a function representation, thereby achieving higher feature extraction computation efficiency.

[0113] Furthermore, this application also proposes an interactive representation method for high-precision maps and historical trajectories. In this application embodiment, the high-precision map is efficiently encoded into a function representation, while the trend features of the historical trajectory are used as function input, thereby completing the interactive representation between the high-precision map and the historical trajectory in a way that saves memory and has higher computational efficiency.

[0114] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0115] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0116] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0117] This application provides a trajectory prediction method, which can be executed by an electronic device, such as a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0118] It should be noted that the electronic devices used to perform trajectory prediction methods may also include: in-vehicle devices installed on various intelligent vehicles and intelligent robots, etc.

[0119] Furthermore, such as Figure 1 As shown, the trajectory prediction method may include:

[0120] Step S101: Obtain the semantic map corresponding to the preset target and the historical observation trajectory sequence corresponding to the predicted target.

[0121] In the embodiments of this application, the preset target may include target vehicles and robots, etc.

[0122] Specifically, the semantic map in this application embodiment can be a raster-style semantic map, a semantic map mapped from a first-view perspective to a bird's-eye view perspective, or other forms of semantic map, which are not limited in this application embodiment.

[0123] Specifically, the historical observation trajectory sequence includes: the location information of the target at each historical moment and the location information at the current moment. In the embodiments of this application, the target may include: vehicles and robots, and other objects that require trajectory prediction.

[0124] Among them, the historical observation trajectory sequence can To characterize, for example, It can correspond to the number of trajectory sampling frames within 2 seconds.

[0125] Furthermore, in the embodiments of this application, a semantic map can be obtained first, followed by a historical observation trajectory sequence; alternatively, a historical observation trajectory sequence can be obtained first, followed by a semantic map; or both a semantic map and a historical observation trajectory sequence can be obtained simultaneously. No limitation is imposed in the embodiments of this application.

[0126] Step S102: Perform sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features.

[0127] Among them, sequence encoding features are used to characterize the historical movement trend of the predicted target.

[0128] Step S103: Based on the semantic map corresponding to the predicted target, generate the semantic map corresponding to the function representation.

[0129] Among them, the function representation corresponding to the semantic map represents the functional mapping between semantics and location.

[0130] It should be noted that step S102 can be executed before step S103, after step S103, or simultaneously with step S103. Figure 1 This is merely an example and is not intended to limit the embodiments of this application.

[0131] Step S104: Based on sequence encoding features and using the function representation corresponding to the semantic map, trajectory prediction is performed to obtain the future predicted trajectory sequence. For the embodiments of this application, through... , indicating the time step Two-dimensional trajectory coordinates, for example, The number of trajectory sampling frames corresponding to the next 6 seconds.

[0132] This application provides a trajectory prediction method. Compared with related technologies, in this application, when predicting future trajectories, prediction is performed based on the functional representation corresponding to the semantic map (the functional relationship between semantics and trajectory position) and the sequence encoding representation corresponding to the historical trajectory sequence. Since the functional relationship can cover spatial location and road semantic information, and the sequence encoding feature represents the trend of historical trajectories, that is, trajectory prediction is performed based on the sequence encoding corresponding to the historical trajectory sequence and the functional representation corresponding to the semantic map, so that the predicted trajectory satisfies the spatial location and road semantic constraints, thereby improving the accuracy of trajectory prediction for the predicted target.

[0133] Specifically, in the embodiments of this application, the semantic map corresponding to the predicted target can be obtained through sensors currently installed on the predicted target (e.g., a vehicle), such as a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), and wheel speed sensors; the historical observation trajectory sequence corresponding to the predicted target can be obtained from local storage, and further, the historical observation trajectory sequence corresponding to the predicted target can also be measured by sensors currently installed on the predicted target (e.g., a vehicle). It should be noted that any method of obtaining the semantic map corresponding to the predicted target and the historical trajectory sequence corresponding to the predicted target is within the protection scope of the embodiments of this application.

[0134] Specifically, after obtaining the historical observation trajectory corresponding to the predicted target through the above embodiments, the historical observation trajectory sequence is represented by sequence encoding to obtain sequence encoding features. This can specifically include: steps S1021 (not shown in the figure), S1022 (not shown in the figure), S1023 (not shown in the figure), and S1024 (not shown in the figure), wherein...

[0135] Step S1021: Based on the historical observation trajectory sequence corresponding to the predicted target, determine the position sequence, acceleration sequence, velocity sequence, and anomaly indication sequence.

[0136] Specifically, the historical observation trajectory sequence is determined as a position sequence, the first derivative of the historical observation trajectory sequence is obtained as a velocity sequence, and the second derivative of the historical observation trajectory sequence is obtained as an acceleration sequence.

[0137] The anomaly indication sequence is used to characterize the loss of frames in the historical observation trajectory sequence. In this embodiment, at certain moments during the observation process, trajectory frames are lost; the lost trajectory frame moment corresponds to 1, and the non-lost trajectory frame moment corresponds to 0.

[0138] Step S1022: Perform spatial embedding representation on the position sequence, acceleration sequence, velocity sequence and anomaly indication sequence respectively to obtain their respective spatial embedding representations.

[0139] Specifically, after obtaining the position sequence, acceleration sequence, velocity sequence and anomaly indication sequence through the above embodiments, these sequences are spatially embedded. That is, these sequences are extended from two dimensions (or one dimension) to higher dimensions through a fully connected layer to obtain denser embedding features (spatial embedding representation of position sequence, spatial embedding representation of acceleration sequence, spatial embedding representation of velocity sequence and spatial embedding representation of anomaly indication sequence).

[0140] Specifically, the position sequence is spatially embedded using formula (1) to obtain the spatial embedding representation of the position sequence, where formula (1) is as follows:

[0141] Formula (1);

[0142] in, Indicates a fully connected layer. This represents the weight parameters of the fully connected layer. Spatial embedding representation of position sequences Represents a position sequence.

[0143] Specifically, the velocity sequence is spatially embedded using formula (2) to obtain the spatially embedded representation of the velocity sequence, where formula (2) is as follows:

[0144] Formula (2);

[0145] in, Spatial embedding representation of velocity sequences, This represents a velocity sequence.

[0146] Specifically, the acceleration sequence is spatially embedded using formula (3) to obtain the spatially embedded representation of the acceleration sequence, where formula (3) is as follows:

[0147] Formula (3);

[0148] Wherein, represents the spatial embedding representation of the acceleration sequence, This represents an acceleration sequence.

[0149] Specifically, the anomaly indicator sequence is spatially embedded using formula (4) to obtain the spatially embedded representation of the anomaly indicator sequence, where formula (4) is as follows:

[0150] Formula (4);

[0151] in, Spatial embedding representation of anomaly indication sequences, This indicates an abnormality indicator sequence.

[0152] Step S1023: Perform position encoding based on their respective spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features and anomaly indication encoding features.

[0153] In the embodiments of this application, after obtaining the spatial embedding representations corresponding to the position sequence, acceleration sequence, velocity sequence and anomaly indication sequence respectively, fixed position encoding is used to obtain position encoding features, acceleration encoding features, velocity encoding features and anomaly indication encoding features.

[0154] Specifically, position encoding is performed based on their respective spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features. This can specifically include: obtaining a position encoding parameter matrix; and determining the position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features based on the position encoding parameter matrix and their respective spatial embedding representations. In this embodiment, the spatial embedding representations corresponding to the position sequence, the acceleration sequence, the velocity sequence, and the anomaly indication sequence are position encoded using the following formulas (5), (6), (7), and (8) to obtain the position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features.

[0155] in,

[0156] , formula (5);

[0157] , formula (6);

[0158] Formula (7);

[0159] Formula (8);

[0160] in, , , as well as These represent position coding features, acceleration coding features, velocity coding features, and anomaly indication coding features, respectively.

[0161] in, , For the position encoding parameter matrix, The relative position index of the trajectory sequence is represented by , and i represents the dimension index of the embedding vector.

[0162] It should be noted that the position encoding parameter matrix PE and the spatial embedding representation of the position sequence are related. Spatial embedding representation of acceleration sequences Spatial embedding representation of velocity sequences and the spatial embedding representation of anomaly indicator sequences They have the same dimensions.

[0163] Step S1024: Extract sequence features by using multiple attention heads and based on position coding features, acceleration coding features, velocity coding features and anomaly indication coding features to obtain sequence coding features.

[0164] Specifically, sequence features are extracted using multiple attention heads and based on position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features to obtain sequence encoding features. This can specifically include steps S10241 (not shown in the figure), S10242 (not shown in the figure), and S10243 (not shown in the figure), wherein...

[0165] Step S10241: Map each encoded feature through each attention head to obtain the query feature, value feature and key feature corresponding to each attention head.

[0166] Specifically, the location coding feature, acceleration coding feature, velocity coding feature, and anomaly indication coding feature are respectively mapped through each attention head to obtain the query feature, value feature, and key feature corresponding to each coding feature in each attention head.

[0167] Specifically, in this embodiment of the application, four attention heads can be used to perform feature mapping on position coding features, acceleration coding features, velocity coding features and anomaly indication coding features. The feature mapping is achieved by the following formulas (9), (10), (11) and (12), respectively. The four attention heads are used as an example for description.

[0168] in,

[0169] Formula (9);

[0170] Formula (10);

[0171] Formula (11);

[0172] Formula (12);

[0173] in, The labels representing fully connected layers with four self-attention heads. In the l-th attention head, it is used for mapping to The weights of the fully connected layer, This represents the weights of the fully connected layer used to map to the Key in the l-th attention head. This represents the weights of the fully connected layer used to map to Value in the l-th attention head.

[0174] Step S10242: Based on the query features, value features, and key features corresponding to each attention head, generate the attention features corresponding to each encoding feature under each attention head.

[0175] In the embodiments of this application, based on the query features, value features and key features corresponding to each attention head, and through dot product attention calculation, the attention features corresponding to each encoding feature under each attention head are obtained.

[0176] Specifically, the attention features corresponding to the position coding features, the attention features corresponding to the acceleration coding features, the attention features corresponding to the velocity coding features, and the attention features corresponding to the anomaly indication coding features under each attention head are generated by the following formulas (13), (14), (15) and (16).

[0177] in, , formula (13);

[0178] , formula (14);

[0179] , formula (15);

[0180] , formula (16);

[0181] in, , , as well as These represent the attention features corresponding to the position encoding feature, acceleration encoding feature, velocity encoding feature, and anomaly indication encoding feature under each attention head, respectively. Attention() represents the dot product attention calculation.

[0182] in, ;

[0183] ;

[0184] ;

[0185] ;

[0186] Step S10243: Perform matrix concatenation on the attention features corresponding to each encoding feature under each attention head, and process the matrix concatenated features through a fully connected layer to obtain the temporal latent features corresponding to each encoding feature.

[0187] Specifically, the temporal implicit features corresponding to the position coding feature, the temporal implicit features corresponding to the acceleration coding feature, the temporal implicit features corresponding to the velocity coding feature, and the temporal implicit features corresponding to the anomaly indication coding feature are obtained by formulas (17), (18), (19) and (20), respectively.

[0188] in, ; Formula (17);

[0189] Formula (18);

[0190] ; Formula (19);

[0191] ; Formula (20);

[0192] in, , , as well as These represent the temporal implicit features corresponding to the position coding feature, the acceleration coding feature, the velocity coding feature, and the anomaly indication coding feature, respectively. This indicates that it is used to connect multiple matrices or vectors; This represents the weight parameters in the corresponding fully connected network. ( ) is used to characterize the operation of fully connected layers.

[0193] Step S10244: Connect the temporal latent features corresponding to each coding feature to obtain the sequence coding features.

[0194] After obtaining the temporal implicit features corresponding to the position coding feature, acceleration coding feature, velocity coding feature, and anomaly indication coding feature through the above embodiments, these temporal implicit features are then... Processing yields sequence coding features, that is,... .

[0195] Furthermore, the above embodiments detailed the method of sequence encoding the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features. The following embodiments introduce the method of processing the semantic map to obtain the function representation corresponding to the semantic map.

[0196] Specifically, before generating the semantic map corresponding to the predicted target's function representation, the process may further include: obtaining the orientation information of the predicted target at the last historical observation time from the semantic map corresponding to the predicted target; and rotating and cropping the semantic map corresponding to the predicted target based on the orientation information of the predicted target at the last historical observation time. For example, the last historical observation time is... Determine the prediction target in The orientation information at any given time is used to rotate the semantic map corresponding to the predicted target based on the orientation information, and then the map is cropped.

[0197] For example, the semantic map selects the last observation position of the predicted target trajectory as the center, with a length and width of 1000*600 pixels. Taking a semantic map resolution of 10 pixels / meter as an example, the semantic map input is the area within 100m around the predicted target (vehicle). The 60m clipping section, along with the semantic map, only uses the semantics of four channels: drivable area, pedestrian walking area, lane lines, and stop lines.

[0198] Specifically, based on the semantic map corresponding to the predicted target, a function representation corresponding to the semantic map is generated. This can specifically include generating a function representation corresponding to the semantic map based on the cropped semantic map. In this embodiment, after obtaining the cropped semantic map, embedding representation and position encoding are performed on the cropped semantic map to obtain the function representation corresponding to the semantic map.

[0199] It should be noted that the method of embedding and encoding the cropped semantic map in this embodiment is similar to the method of embedding and encoding the sequences (position sequence, acceleration sequence, velocity sequence and anomaly indication sequence) in the above embodiments, and will not be repeated here.

[0200] Furthermore, before embedding and encoding the cropped semantic map, the method may further include: if the cropped semantic map is not horizontally placed, then the semantic map is orientation-transformed to obtain the horizontal function representation corresponding to the semantic map. In this embodiment, the obtained semantic map is obtained through... , indicating vehicle exist The surrounding high-definition map image at any given time, with a dimension of [missing information]. ,in and Indicates the length and width of the map. This indicates the number of map channels, and the semantic map is placed horizontally, meaning the semantic map is positioned from dimension [missing information]. The three-dimensional matrix is ​​transformed into a matrix with dimensions of 1. The C vectors of length W×H represent Furthermore, if the clipped semantic map is a horizontally placed semantic map, then the vector representation of the horizontally placed semantic map is still... .

[0201] It should be noted that, in this embodiment, the direction transformation is not limited to cropping the semantic map beforehand; the direction transformation can be performed first, and then the semantic map after the direction transformation can be cropped in the manner described above. Furthermore, in this embodiment, the semantic map can be cropped in the manner described above without performing a direction transformation, or vice versa; no limitation is made in this embodiment.

[0202] Specifically, the semantic map is represented by the vector representation of the semantic map. In this embodiment, based on the semantic map corresponding to the prediction target, a function representation corresponding to the semantic map is generated, which may include: performing spatial embedding representation and position encoding on the vector representation of the semantic map; linearly mapping the position-encoded vector representation to the value channel and the key channel to obtain key features and value features; and generating a function representation corresponding to the semantic map based on the key features and value features.

[0203] Furthermore, in the above embodiments, the semantic map is cropped and / or its orientation is transformed to obtain a vector representation of the semantic map. Then, the semantic map is represented by vectors. Spatial embedding representation and positional encoding are performed to obtain the position-encoded vector representation. Then, using the following formulas (21) and (22), the position-encoded vector representation is linearly mapped to the value channel and the key channel to obtain the value features and key features. Then, based on the value features and key features, the function representation corresponding to the semantic map is generated.

[0204] in,

[0205] , formula (21);

[0206] , formula (22);

[0207] Where K represents the key feature and V represents the value feature. and Let T be the computational parameter of the linear mapping, and let T represent the matrix transpose. This represents the vector representation after position encoding.

[0208] Specifically, after obtaining the key features and value features, a functional representation corresponding to the semantic map is generated based on the key features and value features. This may include: standardizing the key features to obtain standardized key features; determining the positional semantics based on the value features and the standardized key features; determining the semantic position based on the value features; and determining the functional representation corresponding to the semantic map based on the positional semantics and the semantic position.

[0209] Among them, the location semantics represents the semantics of each location in the semantic map, and the semantic location represents the position of each semantic in the semantic map.

[0210] It should be noted that, in the embodiments of this application, the step of determining the semantic position based on the value features can be performed first, and then the key features can be standardized to obtain the standardized key features, and the position semantics can be determined based on the value features and the standardized post-processed key features. Alternatively, the step of determining the position semantics can be performed first by standardizing the key features to obtain the standardized key features, and the position semantics can be determined based on the value features and the standardized post-processed key features, and then the step of determining the semantic position based on the value features can be performed. Of course, they can also be performed simultaneously.

[0211] Specifically, in this embodiment of the application, the key features are standardized by formula (23) to obtain the standardized key features, and the positional semantics are determined based on the value features and the standardized key features; and the semantic position is determined based on the value features by formula (24). Furthermore, after obtaining the positional semantics by formula (23) and the semantic position by formula (24), the function representation corresponding to the semantic map is obtained based on the positional semantics and the semantic position by formula (25).

[0212] Formulas (23), (24), and (25) are shown below:

[0213] , formula (23);

[0214] , formula (24);

[0215] , formula (25);

[0216] in, Indicates positional semantics, This represents semantic location. In this embodiment, a semantic map function representation is constructed. Specifically, this can be viewed as implementing function passing through matrix multiplication, while simultaneously encoding the location. Each location... n Contribution to map semantics C The final function is expressed as , indicating related map semantics C and corresponding position n Function mapping.

[0217] Furthermore, since Equation (23) performs softmax normalization on the key values ​​before dot product attention compared to traditional attention operations, and incorporates positional encoding using a superposition method, the computational process no longer has linear complexity, significantly improving the algorithm's computational efficiency. Table 1 shows the time complexity analysis of using CNN, ATTENTION, and high-performance map encoding. Indicates the sequence length or image dimensions. This indicates the size of the convolution kernel, while This indicates the dimensionality of the output feature.

[0218] Table 1

[0219] Method type Time complexity CNN Self-attention mechanism High-performance map coding

[0220] Furthermore, the semantic map functional representation obtained through the above embodiments... and sequence coding features Subsequently, in order to further improve the accuracy of the future predicted trajectory sequence, trajectory prediction is performed based on sequence coding features and through the function representation corresponding to the semantic map to obtain the future predicted trajectory sequence. Specifically, this may include: determining map interaction features based on sequence coding features and through the function representation corresponding to the semantic map; and performing trajectory prediction based on sequence coding features and map interaction features to obtain the future predicted trajectory sequence.

[0221] Specifically, based on sequence-encoded features and through the function representation corresponding to the semantic map, map interaction features are determined. This can include: mapping the sequence-encoded features to the input of the function representation corresponding to the semantic map to obtain the mapped features; and inputting the mapped features into the function representation corresponding to the semantic map to obtain the map interaction features. In this embodiment, the sequence-encoded features... Function representation mapped to semantic map Input The function output yields the final map interaction features. .in, , .

[0222] Furthermore, in the obtained historical trajectory coding features and map semantic interaction features The trajectory is decoded for future predictions using a trajectory decoding module. In this embodiment, a recurrent unit network (RNN) is used to decode the trajectory at each future time step. In this embodiment, the RRNN may include a GRU, a Long Short-Term Memory (LSTM) network, or a Recurrent Neural Network (RNN).

[0223] Specifically, based on historical trajectory coding features and map semantic interaction features And through the trajectory decoding module, future trajectory prediction is decoded, which may specifically include: extracting features and The hidden state features are connected to each other as the initial hidden state input of the recurrent unit network. Then, the hidden state features output by the network iteration are used to obtain the coordinates of the future trajectory prediction point through regression of the fully connected layer.

[0224] Specifically, in this embodiment of the application, future trajectory prediction is performed using formulas (26), (27), and (28).

[0225] , formula (26);

[0226] , formula (27);

[0227] , formula (28);

[0228] in, This indicates the predicted trajectory coordinates at time t in the future. This represents the trajectory coordinates at time t-1. and This represents the parameters of the fully connected layer. At the time step... After iterative output, the final predicted trajectory sequence is obtained. .

[0229] The following embodiments illustrate a trajectory prediction method through an example. In this embodiment, after obtaining the historical observation trajectory sequence corresponding to the prediction target... Then, the historical observation trajectory sequence corresponding to the predicted target will be... Divided into positional sequences , acceleration sequence velocity sequence and abnormal indication sequence Then the position sequence , acceleration sequence velocity sequence and abnormal indication sequence Spatial domain embedding representations are performed separately, and then positional encoding is applied to each corresponding spatial domain embedding representation. Feature mapping is then performed on each position-encoded feature (position-encoded feature, acceleration-encoded feature, velocity-encoded feature, and anomaly indication-encoded feature) to obtain the query, key, and value features corresponding to each encoded feature under each attention head. Finally, dot product attention is calculated based on the query, key, and value corresponding to each attention head to obtain the attention features corresponding to each encoded feature under each attention head. Then, based on the attention features corresponding to each encoded feature under each attention head ( The process involves using fully connected layers to obtain the hidden temporal features (Multihead) corresponding to each encoded feature. Then, the Multiheads corresponding to each encoded feature are contacted to obtain the sequence-encoded features. ,like Figure 2 As shown, where Figure 2 The example given uses four attention points, but this is not intended to be limiting.

[0230] Furthermore, the semantic map corresponding to the obtained prediction target is horizontally placed to obtain a dimension of [missing value]. The algorithm generates C vectors of length W×H. These vectors are then spatially embedded and positionally encoded. A linear mapping is applied to the positionally encoded vectors to obtain keys and values. The keys are then normalized using a softmax function to obtain normalized keys. Finally, positional semantics are derived based on the values ​​and normalized keys. And determine semantic location based on value. Then based on positional semantics and semantic location The function is represented as Furthermore, sequence encoding features Mapping to a function is represented as Input Then based on as well as Obtain map interaction features Then based on as well as Perform trajectory prediction to obtain the predicted trajectory coordinates at time t. , specifically Figure 3 As shown.

[0231] Furthermore, in the above embodiments, an encoder-decoder model architecture is used to predict future trajectories. Specifically, before predicting future trajectories using the above model, the initial model needs to be trained to obtain an encoder-decoder model for predicting future trajectories. The specific training method in this embodiment includes: preprocessing the data to meet the model input requirements: dividing the observed trajectory sequence into historical observed trajectory sequences and future predicted trajectory sequences. The historical observed trajectory sequence serves as the input for model training and inference, while the future predicted trajectory sequence serves as the target value during training. Simultaneously, the high-precision map input is preprocessed; at the last observation position of the predicted target, a high-precision map image within a certain range around it is cropped as input.

[0232] The above embodiments describe a trajectory prediction method from the perspective of process flow. The following embodiments describe a trajectory prediction device from the perspective of modules. For details, please refer to the following embodiments.

[0233] This application provides a trajectory prediction device, such as... Figure 4 As shown, the trajectory prediction device 40 may specifically include: a first acquisition module 41, a sequence encoding module 42, a generation module 43, and a trajectory prediction module 44, wherein,

[0234] The first acquisition module 41 is used to acquire the semantic map corresponding to the prediction target and the historical observation trajectory sequence corresponding to the prediction target. The historical observation trajectory sequence represents the trajectory position of the prediction target at each historical moment.

[0235] Sequence encoding module 42 is used to perform sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features;

[0236] The generation module 43 is used to generate a semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target. The semantic map corresponding to the semantic map represents the mapping relationship between semantics and trajectory position.

[0237] The trajectory prediction module 44 is used to predict the future trajectory sequence based on sequence encoding features and the function representation corresponding to the semantic map.

[0238] In one possible implementation of this application embodiment, when the sequence encoding module 42 performs sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features, it is specifically used for:

[0239] Based on the historical observation trajectory sequence corresponding to the predicted target, the position sequence, acceleration sequence, velocity sequence and anomaly indication sequence are determined. The anomaly indication sequence is used to characterize the frame loss situation in the historical observation trajectory sequence.

[0240] The position sequence, acceleration sequence, velocity sequence, and anomaly indication sequence are each represented by spatial embedding to obtain their respective spatial embedding representations.

[0241] Position encoding is performed based on their respective spatial embedding representations to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features;

[0242] Sequence coding features are obtained by using multiple attention heads and extracting sequence features based on position coding features, acceleration coding features, velocity coding features, and anomaly indication coding features.

[0243] In another possible implementation of this application embodiment, when the sequence encoding module 42 performs position encoding based on its respective spatial embedding representation to obtain position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features, it is specifically used for:

[0244] Obtain the position encoding parameter matrix;

[0245] Based on the position coding parameter matrix and its corresponding spatial embedding representation, position coding features, acceleration coding features, velocity coding features and anomaly indication coding features are determined.

[0246] In another possible implementation of this application embodiment, the sequence encoding module 42 extracts sequence features by using multiple attention heads and based on position encoding features, acceleration encoding features, velocity encoding features, and anomaly indication encoding features to obtain sequence encoding features, specifically used for:

[0247] Each encoded feature is mapped through various attention heads to obtain the query feature, value feature, and key feature corresponding to each attention head.

[0248] Based on the query features, value features, and key features corresponding to each attention head, generate the attention features corresponding to each encoding feature under each attention head;

[0249] Connect the attention features corresponding to each encoded feature under each attention head in a matrix;

[0250] The features after matrix concatenation are processed through a fully connected layer to obtain the temporal latent features corresponding to each encoded feature;

[0251] The temporal latent features corresponding to each coding feature are concatenated to obtain the sequence coding features.

[0252] Another possible implementation of this application embodiment is that the semantic map corresponding to the predicted target is represented by the vector representation of the semantic map;

[0253] When generating the semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target, the generation module 43 is specifically used for:

[0254] Embedding representation and position encoding are performed on the vector representation corresponding to the semantic map;

[0255] The position-encoded vector representation is linearly mapped to the value channel and the key channel to obtain the key feature and the value feature;

[0256] Based on key features and value features, a functional representation corresponding to the semantic map is generated.

[0257] In another possible implementation of this application embodiment, when generating the function representation corresponding to the semantic map based on key features and value features, the generation module 43 is specifically used for:

[0258] The key features are standardized to obtain the standardized key features;

[0259] Based on value features and standard post-processed key features, positional semantics are determined, and positional semantics represent the semantics of each position in the semantic map.

[0260] Based on value features, semantic locations are determined, and semantic locations represent the position of each semantic term in the semantic map.

[0261] Based on location semantics and semantic location, determine the function representation corresponding to the semantic map.

[0262] In another possible implementation of this application embodiment, the device 40 further includes: a second acquisition module and a rotational cropping module, wherein...

[0263] The second acquisition module is used to acquire the orientation information of the predicted target at the last moment of historical observation in the semantic map corresponding to the predicted target;

[0264] The rotation and cropping module is used to rotate and crop the semantic map corresponding to the predicted target based on the orientation information of the predicted target at the last moment of historical observation.

[0265] Specifically, when generating the semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target, the generation module 43 is used for:

[0266] Based on the cropped semantic map, a function representation corresponding to the semantic map is generated.

[0267] In another possible implementation of this application embodiment, when the trajectory prediction module 44 performs trajectory prediction based on sequence encoding features and through the function representation corresponding to the semantic map to obtain the future predicted trajectory sequence, it is specifically used for:

[0268] Based on sequence encoding features and using the corresponding function representation of the semantic map, map interaction features are determined;

[0269] Trajectory prediction is performed based on sequence coding features and map interaction features to obtain future predicted trajectory sequences.

[0270] In another possible implementation of this application embodiment, when the trajectory prediction module 44 determines the map interaction features based on sequence encoding features and through the function representation corresponding to the semantic map, it is specifically used for:

[0271] The sequence-encoded features are mapped to the input of the function representation corresponding to the semantic map to obtain the mapped features;

[0272] The mapped features are input into the function representation corresponding to the semantic map to obtain map interaction features.

[0273] In another possible implementation of this application embodiment, when the trajectory prediction module 44 performs trajectory prediction based on sequence encoding features and map interaction features to obtain a future predicted trajectory sequence, it is specifically used for:

[0274] The sequence coding features and map interaction features are concatenated to obtain the concatenated features.

[0275] The future predicted trajectory sequence is obtained through multiple iterations based on the connected features.

[0276] This application provides a trajectory prediction device. Compared with related technologies, in this application, when predicting future trajectories, prediction is performed based on the function representation corresponding to the semantic map (the functional relationship between semantics and trajectory position) and the sequence encoding representation corresponding to the historical trajectory sequence. Since the functional relationship can cover spatial location and road semantic information, and the sequence encoding feature represents the trend of historical trajectories, that is, trajectory prediction is performed based on the sequence encoding corresponding to the historical trajectory sequence and the function representation corresponding to the semantic map, so that the predicted trajectory meets the spatial location and road semantic constraints, thereby improving the accuracy of trajectory prediction for the predicted target.

[0277] This application provides a trajectory prediction device applicable to the above method embodiments, which will not be described in detail here.

[0278] This application provides an electronic device, such as... Figure 5 As shown, Figure 5The illustrated electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of this application.

[0279] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0280] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0281] The memory 503 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0282] The memory 503 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 503 to implement the content shown in the foregoing method embodiments.

[0283] The electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. They can also be servers; in this embodiment, the server can be a cloud server. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0284] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with related technologies, in this application embodiment, when predicting future trajectories, prediction is performed based on the functional representation corresponding to the semantic map (the functional relationship between semantics and trajectory position) and the sequence encoding representation corresponding to the historical trajectory sequence. Since this functional relationship can encompass spatial location and road semantic information, and the sequence encoding features characterize the trend of historical trajectories, that is, trajectory prediction is performed based on the sequence encoding corresponding to the historical trajectory sequence and the functional representation corresponding to the semantic map, so that the predicted trajectory satisfies spatial location and road semantic constraints, thereby improving the accuracy of trajectory prediction for the predicted target.

[0285] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0286] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0287] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0288] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0289] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0290] The above description of the embodiments is only used to provide a detailed introduction to the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this application, and should not be construed as a limitation of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A trajectory prediction method, characterized in that, include: Obtain the semantic map corresponding to the predicted target and the historical observation trajectory sequence corresponding to the predicted target, wherein the historical observation trajectory sequence represents the trajectory position of the predicted target at each historical moment; In the semantic map corresponding to the predicted target, obtain the orientation information of the predicted target at the last moment of historical observation; Based on the orientation information of the predicted target at the last moment of historical observation, the semantic map corresponding to the predicted target is rotated and cropped. Based on the cropped semantic map, a function representation corresponding to the semantic map is generated, and the function representation corresponding to the semantic map represents the mapping relationship between semantics and trajectory position; Based on the historical observation trajectory sequence corresponding to the predicted target, a position sequence, an acceleration sequence, a velocity sequence, and an anomaly indication sequence are determined. The anomaly indication sequence is used to characterize the frame loss situation in the historical observation trajectory sequence. The position sequence, acceleration sequence, velocity sequence, and anomaly indication sequence are each represented by spatial embedding to obtain their respective spatial embedding representations; Obtain the position encoding parameter matrix; Based on the position coding parameter matrix and the corresponding spatial embedding representation, position coding features, acceleration coding features, velocity coding features and anomaly indication coding features are determined; Each encoded feature is mapped through various attention heads to obtain the query feature, value feature, and key feature corresponding to each attention head. Based on the query features, value features, and key features corresponding to each attention head, the attention features corresponding to each encoded feature under each attention head are generated. The attention features corresponding to each encoded feature under each attention head are matrix-connected; The features after matrix concatenation are processed through a fully connected layer to obtain the temporal latent features corresponding to each encoded feature; The temporal latent features corresponding to each encoded feature are concatenated to obtain the sequence encoded features; The sequence-encoded features are mapped to the input of the function representation corresponding to the semantic map to obtain the mapped features; The mapped features are input into the function representation corresponding to the semantic map to obtain map interaction features; The sequence encoding features and the map interaction features are concatenated to obtain the concatenated features. Based on the connected features, a future predicted trajectory sequence is obtained through multiple iterative processes.

2. The method according to claim 1, characterized in that, The semantic map corresponding to the predicted target is represented by a vector representation of the semantic map; Based on the semantic map corresponding to the predicted target, a semantic map corresponding to a function representation is generated, including: The vector representation of the semantic map is subjected to spatial embedding representation and position encoding; The position-encoded vector representation is linearly mapped to the value channel and the key channel to obtain the key feature and the value feature; Based on the key features and the value features, a function representation corresponding to the semantic map is generated.

3. The method according to claim 2, characterized in that, The step of generating the function representation corresponding to the semantic map based on the key features and value features includes: The key features are standardized to obtain standardized key features; Based on the value features and the standardized key features, the location semantics are determined, and the location semantics represent the semantics of each location in the semantic map; Based on the value features, semantic locations are determined, whereby each semantic location represents the position of each semantic term in the semantic map. Based on the location semantics and the semantic location, the function representation corresponding to the semantic map is determined.

4. A trajectory prediction device, characterized in that, Performing the trajectory prediction method as described in any one of claims 1-3 includes: The first acquisition module is used to acquire the semantic map corresponding to the prediction target and the historical observation trajectory sequence corresponding to the prediction target, wherein the historical observation trajectory sequence represents the trajectory position of the prediction target at each historical moment. The sequence encoding module is used to perform sequence encoding representation on the historical observation trajectory sequence corresponding to the predicted target to obtain sequence encoding features; The generation module is used to generate a semantic map corresponding to the semantic map based on the semantic map corresponding to the predicted target. The semantic map corresponding to the semantic map represents the mapping relationship between semantics and trajectory position. The trajectory prediction module is used to predict the future trajectory sequence based on the sequence encoding features and the function representation corresponding to the semantic map.

5. An electronic device, characterized in that, It includes: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to: perform the trajectory prediction method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the trajectory prediction method as described in any one of claims 1 to 3.

Images