Trajectory Prediction Method, System, Device and Storage Medium Based on Hybrid Fourier Encoder

Abstract

The present invention relates to the technical field of trajectory prediction, and specifically discloses a trajectory prediction method, system, device, and storage medium based on a hybrid Fourier encoder. The trajectory prediction method based on the hybrid Fourier encoder includes: obtaining the historical trajectory of an agent and the center line of a lane; performing feature encoding on the historical trajectory of the agent and the center line of the lane through a mixture of Gaussians encoder and an attention mechanism to obtain agent encoding features and lane line encoding features; combining the agent encoding features and the lane line encoding features, and performing prediction through the attention mechanism to obtain a predicted trajectory. The present invention designs a hybrid Fourier trajectory feature map encoder, introduces a Fourier encoder, first extracts the encoding features of the historical trajectory of the agent and the center line of the lane, and then performs trajectory prediction through the attention mechanism, which can extract richer feature information from discrete trajectory points and improve the accuracy of trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of trajectory prediction, and particularly to a trajectory prediction method, system, device and storage medium based on a hybrid Fourier encoder. Background Art

[0002] Trajectory prediction is a subfield of motion prediction, which refers to the task of predicting the future position, speed, direction and other state information of a target given its past or current motion trajectory. In the trajectory prediction task, an encoder is usually used to embed historical trajectories and environmental information into context information. Such information includes, but is not limited to, lane information, historical trajectory data, vehicle interaction information, etc. The encoder encodes this information through its internal mechanisms, such as neural network layers, attention mechanisms, etc., to generate a feature representation with rich context information. These feature representations are then input into a decoder or other prediction models to generate future trajectory predictions. However, currently in the field of deep learning, graph encoders often use fully connected layers to construct adjacency matrices, and the extraction of features is not sufficient. Summary of the Invention

[0003] The main object of the present invention is to propose a trajectory prediction method, system, device and storage medium based on a hybrid Fourier encoder, aiming to enhance the feature extraction ability of the trajectory prediction model for historical information and improve the trajectory prediction accuracy.

[0004] To achieve the above object, the trajectory prediction method based on a hybrid Fourier encoder proposed by the present invention, the trajectory prediction method based on a hybrid Fourier encoder includes:

[0005] Obtain the historical trajectory of the agent and the center line of the lane;

[0006] Through a hybrid Gaussian encoder and an attention mechanism, perform feature encoding on the historical trajectory of the agent and the center line of the lane to obtain an agent encoding feature and a lane line encoding feature;

[0007] Combine the agent encoding feature and the lane line encoding feature, and perform prediction through an attention mechanism to obtain a predicted trajectory.

[0008] In an embodiment, the step of obtaining the historical trajectory of the agent and the center line of the lane includes:

[0009] Obtain the historical trajectory of the agent to be predicted and the historical trajectories of other agents around it;

[0010] Obtain the center line information of all lanes at the position where the agent to be predicted is located.

[0011] In one embodiment, the step of encoding the features of the agent's historical trajectory and the lane centerline through a mixture of Gaussian encoders and an attention mechanism to obtain the agent encoding features and the lane line encoding features includes:

[0012] Use a mixture of Gaussian encoders to extract the features of the agent's historical trajectory and the lane centerline respectively to obtain a feature matrix;

[0013] Use the attention mechanism to extract the relevant information between the agent's historical trajectory and the lane centerline, and combine the feature matrix to obtain the agent encoding features and the lane line encoding features.

[0014] In one embodiment, the step of using a mixture of Gaussian encoders to extract the features of the agent's historical trajectory and the lane centerline respectively to obtain a feature matrix includes:

[0015] Multiply the trainable embedding encoder weights with the agent's historical trajectory and the lane centerline to obtain the upsampled features;

[0016] Use the Fourier encoding method to extract features from the upsampled features to obtain the extracted features;

[0017] Use a multi-layer perceptron mechanism to establish a fully connected layer for the extracted features to obtain a representation adjacency matrix;

[0018] Extract the representation adjacency matrix in the time dimension through a mathematical model to obtain the feature matrix of the mixed Fourier encoding result.

[0019] In one embodiment, before the step of combining the agent encoding features and the lane line encoding features and making a prediction through the attention mechanism to obtain the predicted trajectory, it includes:

[0020] Obtain the central mode query password and use the central mode query password as the query value of the attention mechanism.

[0021] In one embodiment, the step of combining the agent encoding features and the lane line encoding features and making a prediction through the attention mechanism to obtain the predicted trajectory includes:

[0022] Fuse the agent encoding features and the lane line encoding features through a cross-attention mechanism, and extract the relevant features of the agent encoding features and the lane line encoding features;

[0023] Use the relevant features and combine the self-attention mechanism to make a prediction to obtain the predicted trajectory.

[0024] In one embodiment, after the step of using the relevant features and combining the self-attention mechanism to make a prediction to obtain the predicted trajectory, it includes:

[0025] Decode the predicted trajectory through a multi-layer perceptron to generate and output the future trajectory of the agent and the predicted distance probability distribution.

[0026] The present invention also proposes a trajectory prediction device based on a hybrid Fourier encoder. The automatic parking path planning system of the intelligent vehicle is used to execute the above-mentioned trajectory prediction method based on the hybrid Fourier encoder. The automatic parking path planning system of the intelligent vehicle includes:

[0027] A trajectory prediction input module, configured to obtain the historical trajectory of the agent and the center line of the lane;

[0028] A trajectory prediction model module, configured to perform feature encoding on the historical trajectory of the agent and the center line of the lane through a mixture of Gaussian encoders and an attention mechanism to obtain the encoded features of the agent and the encoded features of the lane line;

[0029] A trajectory prediction output module, configured to combine the encoded features of the agent and the encoded features of the lane line, and perform prediction through the attention mechanism to obtain the predicted trajectory.

[0030] The present invention also proposes a trajectory prediction device based on a hybrid Fourier encoder. The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the above-mentioned trajectory prediction method based on the hybrid Fourier encoder.

[0031] The present invention also proposes a storage medium. The storage medium is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the trajectory prediction method based on the hybrid Fourier encoder as described above are implemented.

[0032] The present invention relates to the technical field of trajectory prediction, and specifically discloses a trajectory prediction method, system, device, and storage medium based on a hybrid Fourier encoder. The trajectory prediction method based on the hybrid Fourier encoder includes: obtaining the historical trajectory of the agent and the center line of the lane; performing feature encoding on the historical trajectory of the agent and the center line of the lane through a mixture of Gaussian encoders and an attention mechanism to obtain the encoded features of the agent and the encoded features of the lane line; combining the encoded features of the agent and the encoded features of the lane line, and performing prediction through the attention mechanism to obtain the predicted trajectory. The present invention designs a hybrid Fourier trajectory feature map encoder, introduces a Fourier encoder, first extracts the encoded features of the historical trajectory of the agent and the center line of the lane, and then performs trajectory prediction through the attention mechanism, which can extract richer feature information from discrete trajectory points and improve the accuracy of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.

[0034] Figure 1 Schematic flow chart of the first embodiment of the trajectory prediction method based on a hybrid Fourier encoder provided by the present invention;

[0035] Figure 2 Schematic flow chart of the trajectory prediction method based on a hybrid Fourier encoder provided by the present invention;

[0036] Figure 3 Schematic flow chart of the second embodiment of the trajectory prediction method based on a hybrid Fourier encoder provided by the present invention;

[0037] Figure 4 Encoder diagram of the trajectory prediction method based on a hybrid Fourier encoder provided by the present invention;

[0038] Figure 5 Schematic flow chart of the third embodiment of the trajectory prediction method based on a hybrid Fourier encoder provided by the present invention;

[0039] Figure 6 Decoder diagram of the trajectory prediction method based on a hybrid Fourier encoder provided by the present invention;

[0040] Figure 7 Schematic structural diagram of an embodiment of the trajectory prediction system based on a hybrid Fourier encoder provided by the present invention;

[0041] Figure 8 Schematic structural diagram of the trajectory prediction device based on the hardware operating environment involved in the trajectory prediction method based on a hybrid Fourier encoder in the embodiments of the present application.

[0042] The realization, functional features, and advantages of the objectives of the present invention will be further described in conjunction with the embodiments with reference to the drawings. Detailed implementation manners

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0044] It should be noted that if there are directional indications involved in the embodiments of the present invention (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative positional relationship and movement conditions between components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0045] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, if "and / or" or "and / or" appears throughout the text, its meaning includes three parallel scenarios. Taking "A and / or B" as an example, it includes Scenario A, or Scenario B, or the scenario where both A and B are satisfied simultaneously. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on what can be achieved by those of ordinary skill in the art. When the combination of technical solutions results in contradictions or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0046] Currently, in the field of deep learning, graph encoders often use fully connected layers to construct adjacency matrices, and the extraction of features is not sufficient. The present invention designs a hybrid Fourier trajectory feature graph encoder. A Fourier encoder is introduced before the fully connected graph encoder. First, the high-frequency features of the trajectory history information are extracted, and then the fully connected graph encoding is performed, which can extract richer feature information from discrete trajectory points and improve the trajectory prediction accuracy. The present invention also designs a decoder architecture centered on a trainable modality weight for feature extraction and generating predicted trajectories. Using the embedding layer weight as the decoder center, the correlation between it and the encoder features is extracted respectively through the attention mechanism, and finally the predicted trajectory is obtained through a multi-layer perceptron.

[0047] The present invention relates to the technical field of trajectory prediction, and specifically discloses a trajectory prediction method, system, device, and storage medium based on a hybrid Fourier encoder. The trajectory prediction method based on the hybrid Fourier encoder includes: obtaining the historical trajectory of an agent and the center line of a lane; through a mixture of Gaussian encoders and an attention mechanism, performing feature encoding on the historical trajectory of the agent and the center line of the lane to obtain agent encoding features and lane line encoding features; combining the agent encoding features and the lane line encoding features, and performing prediction through the attention mechanism to obtain a predicted trajectory. The present invention designs a hybrid Fourier trajectory feature map encoder, introduces a Fourier encoder, first extracts the encoding features of the historical trajectory of the agent and the center line of the lane, and then performs trajectory prediction through the attention mechanism, which can extract richer feature information from discrete trajectory points and improve the accuracy of trajectory prediction.

[0048] It can be understood that this application can relatively accurately predict the future trajectory of an agent. Since a Fourier encoder is introduced before the fully connected graph encoder, it can extract richer feature information from discrete trajectory points. Therefore, it has a good prediction effect on the future trajectory of an agent, especially the trajectory of pedestrians with a relatively discrete distribution; this application has strong interpretability, and compared with the virtual interaction force calculation method, it has a smaller amount of calculation and higher efficiency.

[0049] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a trajectory prediction device based on a hybrid Fourier encoder that can implement the above functions. Hereinafter, taking the trajectory prediction model as an example, this embodiment and the following embodiments will be described.

[0050] Based on this, an embodiment of this application provides a trajectory prediction method based on a hybrid Fourier encoder, referring to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the trajectory prediction method based on the hybrid Fourier encoder of this application.

[0051] It can be understood that as Figure 2 shown, trajectory prediction includes three parts: trajectory prediction input, trajectory prediction model, and trajectory prediction output.

[0052] In this embodiment, the trajectory prediction method based on the hybrid Fourier encoder includes steps S10 to S30.

[0053] Step S10: Obtain the historical trajectory of the agent and the center line of the lane.

[0054] It can be understood that the agent historical trajectory refers to the path record of a vehicle, pedestrian, or obstacle traveled over a past period of time, usually including information such as the position, speed, and direction of the vehicle or pedestrian. The lane center line refers to the characteristic line formed by connecting the centers of the road surface widths from the starting point to the ending point of the road surface.

[0055] Step S20: Through a mixture of Gaussian encoder and attention mechanism, perform feature encoding on the agent historical trajectory and the lane center line to obtain an agent encoded feature and a lane line encoded feature.

[0056] It can be understood that the mixture of Gaussian encoder is a type of mixture of Fourier encoder. In the mixture of Gaussian encoder, a mixture of Gaussian models is introduced into the decoder network of the variational autoencoder to simulate the ability to generate multi-modal data. The decoder network no longer directly reconstructs the input data from samples in the latent space, but generates the latent representations of each modality and reconstructs the modality data from the latent representations through an inverse transformation. The present invention introduces a mixture of Fourier models, which can enhance the decoder architecture of the trajectory prediction model for historical information trajectories and improve the generalization of the trajectory prediction model.

[0057] It should be noted that the encoder performs feature encoding on the historical trajectory information through a mixture of Gaussian encoder and attention mechanism to obtain an agent encoded feature and a lane line encoded feature respectively.

[0058] Step S30: Combine the agent encoded feature and the lane line encoded feature, and perform prediction through the attention mechanism to obtain a predicted trajectory.

[0059] It can be understood that the lane line encoded feature and the agent encoded feature are respectively input into a cross-attention mechanism to extract relevant features, and then prediction is performed through a self-attention mechanism to estimate the future trajectory.

[0060] In this embodiment, obtaining the agent historical trajectory and the lane center line as the input for trajectory prediction, the trajectory prediction model includes an encoder and a decoder. The encoder introduces a mixture of Fourier encoder, combines with the attention mechanism, and performs feature encoding on the agent historical trajectory and the lane center line to obtain an agent encoded feature and a lane line encoded feature. The decoder includes an attention mechanism and a multi-layer perceptron. The agent encoded feature and the lane line encoded feature obtain a predicted trajectory through the attention mechanism. Introducing the mixture of Fourier encoder can improve the generalization of the trajectory prediction model and predict the distance probability distribution while predicting the future trajectory of the agent.

[0061] Based on the first embodiment of the present application, in the second embodiment of the present application, for the same or similar content as in the above-mentioned embodiment one, reference can be made to the above introduction and will not be elaborated hereinafter. On this basis, please refer to Figure 3 , Figure 3This is a schematic flowchart provided for the second embodiment of the trajectory prediction method based on a hybrid Fourier encoder in this application. In the above steps, the trajectory prediction method based on a hybrid Fourier encoder includes:

[0062] Step S201: Obtain the historical trajectory of the agent to be predicted and the historical trajectories of other agents in its vicinity.

[0063] It should be noted that the agent historical trajectory information includes the target-level historical trajectory of the agent to be predicted (including the position and speed at historical moments) and the target-level historical trajectories of other agents in its vicinity:

[0064]

[0065] Among them, are the position and speed in the central coordinate system with the current position of the agent to be predicted as the center and the direction of the agent to be predicted as the positive direction. t is the time information, and its range can be selected according to the actual situation.

[0066] Step S202: Obtain the centerline information of all lanes at the position where the agent to be predicted is located.

[0067] It should be noted that the lane centerline refers to the centerline information of all lanes near the agent to be predicted, that is, a set of road center points discretely sampled at a certain distance:

[0068]

[0069] Among them, x and y are the position coordinates of the lane centerline sampling points in the central coordinate system with the current position of the agent to be predicted as the center and the direction of the agent to be predicted as the positive direction, and i is the lane centerline sampling point number.

[0070] Step S203: Use a mixture of Gaussian encoders to extract the features of the agent historical trajectory and the lane centerline respectively, and obtain a feature matrix.

[0071] It can be understood that, as Figure 4 shown, Q (Query), K (Key), and V (Value) are three core feature vectors in the attention mechanism, which are used to calculate the correlation between elements in a sequence and perform weighted processing accordingly. Among them:

[0072] The first feature vector Query (query), which represents the trajectory information at the current time step, is used to query the relevant information at other time steps.

[0073] The second eigenvector, Key, is used as a benchmark for comparison with the Query. That is, in the encoding sequence given by the lane line mixture Gaussian encoder, the Query will be compared with the Keys of all elements to determine which elements are most relevant to the current Query.

[0074] The third eigenvector, Value, is used in the final weighted calculation. Once the relevance between the Query and the Keys of the elements in the sequence is determined, these relevance weights are used to perform a weighted sum on the Value to generate the final output.

[0075] Step S203 includes steps S231 to S234:

[0076] Step S231: Multiply the trainable embedding encoder weights with the agent's historical trajectory and the lane centerline to obtain the dimensionality - raised features.

[0077] Step S232: Use the Fourier encoding method to extract features from the dimensionality - raised features to obtain the extracted features.

[0078] It can be understood that the mixture Gaussian encoder multiplies the trainable embedding encoder weights with the input features to achieve dimensionality - raising of the input features, and uses the Fourier encoding method to extract features from the dimensionality - raised features:

[0079]

[0080] Among them, V is the input agent's historical trajectory or lane centerline, and its dimension is denoted as [num, input_dim]. num represents the number of samples, that is, how many agent's historical trajectory or lane centerline data there are. input_dim is the feature dimension of each sample, which contains information used to describe the agent's historical position or other relevant features. is the embedding encoder weight, whose input dimension is equal to input_dim and the output dimension is hidden_dim. That is, each input feature is mapped to a new hidden space, and the dimension of this space is hidden_dim.

[0081] Among them, for The result of applying the sine and cosine functions is to introduce periodic features and increase the nonlinear ability of the model. Here, the trigonometric function operations are performed element - by - element.

[0082] Step S233: Use a multi - layer perceptron mechanism to establish a fully - connected layer for the extracted features to obtain the adjacency matrix representation.

[0083] It is understandable that the MLP is a multi-layer perceptron, which is a basic layer in the neural network for performing non-linear transformations. It is usually composed of multiple fully connected layers for further processing and decoding of the already extracted features. In this application, since and each will generate an output with a dimension of [num, hidden_dim], in the MLP, these two outputs are concatenated with the original input V to form a new matrix with a dimension of [num, hidden_dim*2+1]. Each row of this new matrix is a feature vector that contains the original features, the features after sine transformation, and the features after cosine transformation. Therefore, the input dimension of the MLP is hidden_dim*2+1, and the output dimension is hidden_dim.

[0084] After using Fourier encoding, time encoding is performed through a fully connected graph, that is, a fully connected layer is constructed using a multi-layer perceptron to represent the adjacency matrix:

[0085]

[0086] where represents the node features after Fourier encoding. represents the i-th layer of the multi-layer perceptron, which receives the features after Fourier encoding as input. represents the features after being processed by the i-th layer of the fully connected layer, which combines the output of with the original Fourier-encoded features to retain the original frequency-domain information and the deeper feature representations learned through the MLP.

[0087] Step S234: Extract the adjacency matrix representing in the time dimension through a mathematical model to obtain the feature matrix of the mixed Fourier encoding result.

[0088] It is understandable that the important features in the time dimension are extracted through max pooling to obtain the mixed Fourier encoding result, that is represented as:

[0089]

[0090] Step S204: Use the attention mechanism to extract the relevant information between the historical trajectory of the agent and the center line of the lane, and combine the feature matrix to obtain the agent encoding feature and the lane line encoding feature.

[0091] It should be noted that cross-attention involves the attention interaction between features from two different sources: the lane line features and the agent features cross each other, and the model determines the correlation between the lane and the agent through the attention mechanism. In this process, the lane centerline may serve as the Query (Q), while the agent's historical trajectory serves as the Key (K) and Value (V), or vice versa, depending on the requirements of the scenario. Through cross-attention, the model can capture the interaction information between the lane environment and the agent.

[0092] It can be understood that the self-attention mechanism is used to mutually focus on each part within the same input sequence. That is, the features of each agent here focus on itself, which can be features such as its own past historical trajectory, speed, acceleration, etc., to extract relevant spatio-temporal information. In this mechanism, Q, K, and V all come from the same agent feature set. In this way, the model can perform self-correlation between the same features to generate new representations.

[0093] Specifically, a mixture of Gaussian encoders is used to separately extract the features of the agent's historical trajectory and the lane centerline, obtaining a feature matrix with a dimension of hidden_dim. After that, according to Figure 4 As shown, the encoder uses the cross-attention mechanism and the self-attention mechanism respectively to extract the correlation between the agent's historical trajectory and the lane line, obtaining the agent encoded features and the lane line encoded features.

[0094] In this embodiment, the encoder performs feature encoding on the historical trajectory information through a mixture of Gaussian encoders and the attention mechanism, respectively obtaining the agent encoded features and the lane line encoded features; among them, the mixture of Gaussian encoders separately extracts the features of the agent's historical trajectory and the lane centerline, obtaining a feature matrix with a dimension of hidden_dim. After that, the encoder uses the cross-attention mechanism and the self-attention mechanism respectively to extract the correlation between the agent's historical trajectory and the lane line, obtaining the agent encoded features and the lane line encoded features.

[0095] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the content that is the same as or similar to the above-mentioned first and second embodiments can be referred to the above introduction and will not be elaborated hereinafter. On this basis, please refer to Figure 5 , Figure 5 which is the flow diagram provided for the third embodiment of the trajectory prediction method based on the mixture of Fourier encoders of the present application.

[0096] Step S301: Obtain the central modal query password and use the central modal query password as the query value of the attention mechanism.

[0097] It can be understood that as Figure 6As shown, the decoder queries the password with the embedded encoder weights as the central mode, extracts the correlations between it and the lane line encoding features and the agent encoding features through the cross-attention mechanism and the self-attention mechanism respectively, and generates the predicted trajectory.

[0098] It should be noted that a certain state information or input query password is obtained, such as the state of an agent at a certain moment, such as speed, position, etc.

[0099] Step S302: Fuse the agent encoding features and the lane line encoding features through the cross-attention mechanism, and extract the relevant features of the agent encoding features and the lane line encoding features.

[0100] It should be noted that the agent encoding features and the lane line encoding features are fused through the cross-attention mechanism. Q (Query), K (Key), and V (Value) are the relevant features obtained from the agent encoding features and the lane line encoding features.

[0101] Step S303: Use the relevant features and combine the self-attention mechanism to make a prediction to obtain the predicted trajectory.

[0102] Step S304: Decode the predicted trajectory through a multi-layer perceptron, and generate and output the future trajectory of the agent and the predicted distance probability distribution.

[0103] It can be understood that the final output is decoded through a multi-layer perceptron MLP to generate the specific future trajectory of the agent and the predicted distance probability distribution, that is, the possible movement trajectory or the next position of the agent in the future time.

[0104] It should be noted that as Figure 6 shown, two cross-attention mechanisms and one self-attention mechanism constitute one round of decoding process. During decoding, the decoding process is repeated N rounds to fully extract feature information and better perform trajectory prediction. After multiple rounds of decoding, the central mode query password already contains various types of feature information, and then a multi-layer perceptron is used for decoding to extract the predicted trajectory:

[0105]

[0106] Among them, is the central mode query password after multiple rounds of decoding, is the predicted position center of the agent at future time t, is its confidence level. Thus, the future trajectory of the agent and the predicted distance probability distribution are obtained.

[0107] In this embodiment, the trajectory prediction decoder for the central modal query password uses the trainable embedding encoder weights as the center and extracts trajectories respectively through the attention mechanism. Among them, the input dimension of the embedding encoder is the number of modalities, and different modalities can be set by adjusting the input dimension. The output dimension of the embedding encoder is hidden_dim, which is consistent with the encoder encoding feature dimension. After obtaining the central modal query password, it is used as the Query of the subsequent attention mechanism. After using the cross-attention mechanism to extract relevant features from the lane line encoding features and the agent encoding features respectively, prediction is performed through the self-attention mechanism to estimate the future trajectory.

[0108] Refer to Figure 7 , Figure 7 is a schematic structural diagram of an embodiment of a trajectory prediction system based on a hybrid Fourier encoder proposed by the present invention. Based on Figure 7 An embodiment of the trajectory prediction system based on the hybrid Fourier encoder of the present invention is proposed.

[0109] The trajectory prediction system based on the hybrid Fourier encoder includes the trajectory prediction method based on the hybrid Fourier encoder as described above. The trajectory prediction system based on the hybrid Fourier encoder includes: a trajectory prediction input module 10 for obtaining the historical trajectory of the agent and the lane center line; a trajectory prediction model module 20 for performing feature encoding on the historical trajectory of the agent and the lane center line through a mixture of Gaussian encoders and an attention mechanism to obtain agent encoding features and lane line encoding features; a trajectory prediction output module 30 for combining the agent encoding features and the lane line encoding features and performing prediction through the attention mechanism to obtain the predicted trajectory.

[0110] This application provides a trajectory prediction device based on a hybrid Fourier encoder. The trajectory prediction device based on the hybrid Fourier encoder includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the trajectory prediction method based on the hybrid Fourier encoder in the first embodiment above.

[0111] Next, refer to Figure 8, which shows a schematic structural diagram of a trajectory prediction device suitable for implementing the hybrid Fourier encoder-based trajectory prediction device according to the embodiments of the present application. The hybrid Fourier encoder-based trajectory prediction device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description: tablet computers), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The shown hybrid Fourier encoder-based trajectory prediction device is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0112] As Figure 8 shown, the hybrid Fourier encoder-based trajectory prediction device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the hybrid Fourier encoder-based trajectory prediction device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the hybrid Fourier encoder-based trajectory prediction device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a hybrid Fourier encoder-based trajectory prediction device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0113] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0114] The trajectory prediction device based on a hybrid Fourier encoder provided by the present application adopts the trajectory prediction method based on a hybrid Fourier encoder in the above embodiments, and can enhance the feature extraction ability of the trajectory prediction model for historical information and improve the trajectory prediction accuracy. Compared with the prior art, the beneficial effects of the trajectory prediction device based on a hybrid Fourier encoder provided by the present application are the same as those of the trajectory prediction method based on a hybrid Fourier encoder provided by the above embodiments, and other technical features in the trajectory prediction device based on a hybrid Fourier encoder are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated herein.

[0115] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0116] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0117] The present application provides a computer-readable storage medium, having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the trajectory prediction method based on a hybrid Fourier encoder in the above embodiments.

[0118] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0119] The above computer-readable storage medium can be included in a trajectory prediction device based on a hybrid Fourier encoder; or it can exist independently without being assembled into a trajectory prediction device based on a hybrid Fourier encoder.

[0120] The computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages - such as Java, Smalltalk, C++; and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as combinations of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0122] The modules described in the embodiments of the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0123] The readable storage medium provided by the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned trajectory prediction method based on a hybrid Fourier encoder, which can enhance the feature extraction ability of the trajectory prediction model for historical information and improve the trajectory prediction accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the trajectory prediction method based on a hybrid Fourier encoder provided by the above embodiments, and will not be elaborated here.

[0124] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made using the specification and drawings of the present application under the technical concept of the present application, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A trajectory prediction method based on a hybrid Fourier encoder, characterized in that: The trajectory prediction method based on the hybrid Fourier encoder comprises: Obtain the agent's historical trajectory and lane centerline; wherein the agent's historical trajectory refers to the path record that the vehicle, pedestrian or obstacle has traveled in the past period of time, including the position, speed and direction information of the vehicle or pedestrian, and the lane centerline refers to the characteristic line formed by connecting the center of the road surface width from the starting point to the end point of the road surface; Using a mixed Gaussian encoder to extract features of the agent's historical trajectory and the lane centerline, respectively, to obtain a feature matrix; Using a cross-attention mechanism and a self-attention mechanism respectively, the correlation between the historical trajectory of the agent and the lane centerline feature is extracted, and the agent encoding feature and the lane centerline encoding feature are obtained by combining the feature matrix; The agent encoding features and the lane centerline encoding features are combined and predicted through an attention mechanism to obtain a predicted trajectory.

2. The trajectory prediction method based on a hybrid Fourier encoder as claimed in claim 1, characterized in that: The step of obtaining the historical trajectory of the intelligent agent and the lane center line includes: Obtain the historical trajectory of the agent to be predicted and the historical trajectories of other agents around it; Get the centerline information of all lanes where the agent to be predicted is located.

3. The trajectory prediction method based on hybrid Fourier encoder according to claim 1, characterized in that: The step of using a mixed Gaussian encoder to respectively extract features of the agent's historical trajectory and the lane centerline to obtain a feature matrix includes: The trained embedding encoder weights are multiplied with the agent’s historical trajectory and lane centerline to obtain the dimension-enhanced features. Using Fourier coding method to extract features from the dimension-upgraded features to obtain extracted features; Using a multi-layer perception mechanism to establish a fully connected layer of the extracted features to obtain a representation adjacency matrix; The adjacency matrix representing the time dimension is extracted through a mathematical model to obtain the characteristic matrix of the mixed Fourier coding result.

4. The trajectory prediction method based on a hybrid Fourier encoder as claimed in claim 1, characterized in that: Before the step of combining the agent encoding feature and the lane centerline encoding feature to predict through an attention mechanism to obtain a predicted trajectory, the step includes: Obtain a central modality query password, and use the central modality query password as a query value of the attention mechanism.

5. The trajectory prediction method based on hybrid Fourier encoder according to claim 4, characterized in that: The step of combining the agent encoding feature and the lane centerline encoding feature to predict and obtain a predicted trajectory through an attention mechanism includes: The agent encoding feature and the lane centerline encoding feature are fused through a cross attention mechanism, and the related features of the agent encoding feature and the lane centerline encoding feature are extracted; The related features are used in combination with the self-attention mechanism to obtain the predicted trajectory.

6. The trajectory prediction method based on hybrid Fourier encoder according to claim 5, characterized in that: After the step of using the relevant features in combination with the self-attention mechanism to predict the predicted trajectory, the method further includes: The predicted trajectory is decoded through a multi-layer perceptron to generate and output the agent's future trajectory and predicted distance probability distribution.

7. An intelligent vehicle automatic parking path planning system, characterized in that: The intelligent vehicle automatic parking path planning system is used to execute the trajectory prediction method based on the hybrid Fourier encoder according to any one of claims 1 to 6, and the intelligent vehicle automatic parking path planning system includes: The trajectory prediction input module is used to obtain the agent's historical trajectory and lane centerline; wherein the agent's historical trajectory refers to the path record that the vehicle, pedestrian or obstacle has traveled in the past period of time, including the position, speed and direction information of the vehicle or pedestrian, and the lane centerline refers to the characteristic line formed by connecting the center of the road width from the starting point to the end point of the road surface; A Fourier encoding module, used for respectively extracting features of the agent's historical trajectory and the lane centerline using a mixed Gaussian encoder to obtain a feature matrix; an attention mechanism combining module, for respectively using a cross attention mechanism and a self-attention mechanism to extract the correlation between the agent historical trajectory and the lane centerline feature, and combining the feature matrix to obtain the agent encoding feature and the lane centerline encoding feature; The trajectory prediction output module is used to combine the agent encoding features and the lane centerline encoding features to obtain a predicted trajectory through an attention mechanism.

8. An intelligent vehicle automatic parking path planning device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the trajectory prediction method based on a hybrid Fourier encoder according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the trajectory prediction method based on a hybrid Fourier encoder as described in any one of claims 1 to 6 are implemented.

Images