Trajectory prediction method, system and device based on hybrid Fourier encoder and storage medium

Abstract

The invention relates to the technical field of trajectory prediction, and particularly discloses a trajectory prediction method, system and device based on a hybrid Fourier encoder, and a storage medium. The trajectory prediction method based on the hybrid Fourier encoder comprises the following steps: acquiring a historical trajectory of an intelligent agent and a lane center line; through a Gaussian mixture encoder and an attention mechanism, feature coding is carried out on the historical trajectory of the intelligent agent and the lane center line, and intelligent agent coding features and lane line coding features are obtained; and in combination with the intelligent agent coding features and the lane line coding features, performing prediction through an attention mechanism to obtain a predicted track. The invention designs a hybrid Fourier trajectory feature map encoder, introduces a Fourier encoder, firstly extracts encoding features of the historical trajectory of the intelligent agent and the center line of the lane, and then performs trajectory prediction through an attention mechanism, so that richer feature information can be extracted from discrete trajectory points, and the trajectory prediction precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of trajectory prediction, and in particular to a trajectory prediction method, system, device and storage medium based on a hybrid Fourier encoder. Background Art

[0002] Trajectory prediction is a subfield of motion prediction, which refers to the task of predicting the future position, speed, direction and other state information of a target given its past or current motion trajectory. In trajectory prediction tasks, encoders are usually used to embed historical trajectory and environmental information into contextual information. This information includes but is not limited to lane information, historical trajectory data, vehicle interaction information, etc. The encoder encodes this information through its internal mechanisms, such as neural network layers, attention mechanisms, etc., to generate feature representations with rich contextual information. These feature identifiers are then input into the decoder or other prediction models to generate future trajectory predictions. However, in the field of deep learning, graph encoders often use fully connected layers to construct adjacency matrices, which does not fully extract features. Summary of the invention

[0003] The main purpose of the present invention is to propose a trajectory prediction method, system, device and storage medium based on a hybrid Fourier encoder, aiming to enhance the feature extraction capability of the trajectory prediction model for historical information and improve the trajectory prediction accuracy.

[0004] To achieve the above object, the present invention proposes a trajectory prediction method based on a hybrid Fourier encoder, and the trajectory prediction method based on a hybrid Fourier encoder includes: Get the agent's historical trajectory and lane centerline; By using a mixed Gaussian encoder and an attention mechanism, feature encoding is performed on the historical trajectory of the agent and the center line of the lane to obtain agent encoding features and lane line encoding features; The agent encoding features and the lane line encoding features are combined and predicted through an attention mechanism to obtain a predicted trajectory.

[0005] In one embodiment, the step of obtaining the agent's historical trajectory and lane centerline includes: Obtain the historical trajectory of the agent to be predicted and the historical trajectories of other agents around it; Get the centerline information of all lanes where the agent to be predicted is located.

[0006] In one embodiment, the step of encoding the agent history trajectory and the lane centerline by a mixed Gaussian encoder and an attention mechanism to obtain the agent encoding feature and the lane line encoding feature includes: Using a mixed Gaussian encoder to extract features of the agent's historical trajectory and the lane centerline, respectively, to obtain a feature matrix; The attention mechanism is used to extract the relevant information between the historical trajectory of the agent and the center line of the lane, and the agent encoding features and the lane line encoding features are obtained by combining the feature matrix.

[0007] In one embodiment, the step of using a mixed Gaussian encoder to extract the features of the agent's historical trajectory and the lane centerline to obtain a feature matrix includes: The trained embedding encoder weights are multiplied with the agent’s historical trajectory and lane centerline to obtain the dimensionally upgraded features. Using Fourier coding method to extract features from the dimension-upgraded features to obtain extracted features; Using a multi-layer perception mechanism to establish a fully connected layer of the extracted features to obtain a representation adjacency matrix; The adjacency matrix representing the time dimension is extracted through a mathematical model to obtain the characteristic matrix of the mixed Fourier coding result.

[0008] In one embodiment, before the step of combining the agent encoding feature and the lane line encoding feature to predict through an attention mechanism to obtain a predicted trajectory, the step includes: Obtain a central modality query password, and use the central modality query password as a query value of the attention mechanism.

[0009] In one embodiment, the step of combining the agent encoding feature and the lane line encoding feature to predict the predicted trajectory through an attention mechanism includes: The agent encoding feature and the lane line encoding feature are fused through a cross attention mechanism, and related features of the agent encoding feature and the lane line encoding feature are extracted; The related features are used in combination with the self-attention mechanism to obtain the predicted trajectory.

[0010] In one embodiment, after the step of using the relevant features in combination with the self-attention mechanism to predict the predicted trajectory, the following steps are included: The predicted trajectory is decoded through a multi-layer perceptron to generate and output the agent's future trajectory and predicted distance probability distribution.

[0011] The present invention also proposes a trajectory prediction device based on a hybrid Fourier encoder, wherein the intelligent vehicle automatic parking path planning system is used to execute the above-mentioned trajectory prediction method based on a hybrid Fourier encoder, and the intelligent vehicle automatic parking path planning system comprises: Trajectory prediction input module, used to obtain the agent's historical trajectory and lane centerline; The trajectory prediction model module is used to encode the historical trajectory of the agent and the lane centerline through a mixed Gaussian encoder and an attention mechanism to obtain the agent encoding features and the lane line encoding features; The trajectory prediction output module is used to combine the agent encoding features and the lane line encoding features to obtain the predicted trajectory through an attention mechanism.

[0012] The present invention also proposes a trajectory prediction device based on a hybrid Fourier encoder, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the above-mentioned trajectory prediction method based on the hybrid Fourier encoder.

[0013] The present invention also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the trajectory prediction method based on the hybrid Fourier encoder as described above are implemented.

[0014] The present invention relates to the field of trajectory prediction technology, and specifically discloses a trajectory prediction method, system, device and storage medium based on a hybrid Fourier encoder. The trajectory prediction method based on the hybrid Fourier encoder includes: obtaining the historical trajectory of the intelligent agent and the center line of the lane; encoding the characteristics of the historical trajectory of the intelligent agent and the center line of the lane through a hybrid Gaussian encoder and an attention mechanism to obtain the encoding characteristics of the intelligent agent and the encoding characteristics of the lane line; combining the encoding characteristics of the intelligent agent and the encoding characteristics of the lane line, and predicting through an attention mechanism to obtain a predicted trajectory. The present invention designs a hybrid Fourier trajectory feature map encoder, introduces a Fourier encoder, first extracts the encoding characteristics of the historical trajectory of the intelligent agent and the center line of the lane, and then performs trajectory prediction through an attention mechanism, which can extract richer feature information from discrete trajectory points and improve trajectory prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0016] Figure 1 A schematic flow chart of a first embodiment of a trajectory prediction method based on a hybrid Fourier encoder provided by the present invention; Figure 2 A schematic diagram of a flow chart of a trajectory prediction method based on a hybrid Fourier encoder provided by the present invention; Figure 3 A schematic flow chart of a second embodiment of a trajectory prediction method based on a hybrid Fourier encoder provided by the present invention; Figure 4 An encoder diagram of a trajectory prediction method based on a hybrid Fourier encoder provided by the present invention; Figure 5 A schematic flow chart of a third embodiment of a trajectory prediction method based on a hybrid Fourier encoder provided by the present invention; Figure 6 A decoder diagram of the trajectory prediction method based on the hybrid Fourier encoder provided by the present invention; Figure 7 A schematic structural diagram of an embodiment of a trajectory prediction system based on a hybrid Fourier encoder provided by the present invention; Figure 8 The present invention is a schematic structural diagram of a trajectory prediction device based on a hybrid Fourier encoder of the hardware operating environment involved in the trajectory prediction method based on a hybrid Fourier encoder in an embodiment of the present application.

[0017] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of the features. In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0021] Currently in the field of deep learning, graph encoders often use fully connected layers to construct adjacency matrices, which does not fully extract features. The present invention designs a hybrid Fourier trajectory feature graph encoder, which introduces a Fourier encoder before the fully connected graph encoder. It first extracts high-frequency features of trajectory history information, and then performs fully connected graph encoding, which can extract richer feature information from discrete trajectory points and improve trajectory prediction accuracy. The present invention also designs a decoder architecture that extracts features and generates predicted trajectories with a trainable modal weight as the center, uses the embedding layer weight as the decoder center, extracts its correlation with the encoder features through the attention mechanism, and finally extracts the predicted trajectory architecture through a multi-layer perceptron.

[0022] The present invention relates to the field of trajectory prediction technology, and specifically discloses a trajectory prediction method, system, device and storage medium based on a hybrid Fourier encoder. The trajectory prediction method based on the hybrid Fourier encoder includes: obtaining the historical trajectory of the intelligent agent and the center line of the lane; encoding the characteristics of the historical trajectory of the intelligent agent and the center line of the lane through a hybrid Gaussian encoder and an attention mechanism to obtain the encoding characteristics of the intelligent agent and the encoding characteristics of the lane line; combining the encoding characteristics of the intelligent agent and the encoding characteristics of the lane line, and predicting through an attention mechanism to obtain a predicted trajectory. The present invention designs a hybrid Fourier trajectory feature map encoder, introduces a Fourier encoder, first extracts the encoding characteristics of the historical trajectory of the intelligent agent and the center line of the lane, and then performs trajectory prediction through an attention mechanism, which can extract richer feature information from discrete trajectory points and improve trajectory prediction accuracy.

[0023] It can be understood that the present application can predict the future trajectory of the intelligent agent more accurately. Since the Fourier encoder is introduced before the fully connected graph encoder, it can extract richer feature information from discrete trajectory points, and thus has a good prediction effect on the future trajectory of the intelligent agent, especially the trajectory of pedestrians with a relatively discrete trajectory distribution; the present application has strong interpretability and, compared with the virtual interaction force calculation method, has a smaller amount of computation and higher efficiency.

[0024] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a trajectory prediction device based on a hybrid Fourier encoder that can realize the above functions. The following takes the trajectory prediction model as an example to illustrate this embodiment and the following embodiments.

[0025] Based on this, the embodiment of the present application provides a trajectory prediction method based on a hybrid Fourier encoder, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the trajectory prediction method based on the hybrid Fourier encoder of the present application.

[0026] It is understandable that if Figure 2 As shown, trajectory prediction includes three parts: trajectory prediction input, trajectory prediction model and trajectory prediction output.

[0027] In this embodiment, the trajectory prediction method based on the hybrid Fourier encoder includes steps S10 to S:30.

[0028] Step S10: Obtain the agent's historical trajectory and lane centerline.

[0029] It can be understood that the agent history trajectory refers to the path record that the vehicle, pedestrian or obstacle has traveled in the past period of time, usually including the position, speed, direction and other information of the vehicle or pedestrian. The lane centerline refers to the characteristic line formed by connecting the center of the road width from the starting point to the end point of the road surface.

[0030] Step S20: Through a mixed Gaussian encoder and an attention mechanism, feature encoding is performed on the agent historical trajectory and the lane centerline to obtain agent encoding features and lane line encoding features.

[0031] It can be understood that the mixed Gaussian encoder is a kind of mixed Fourier encoder, in which the mixed Gaussian model is introduced into the decoder network of the variational autoencoder to simulate the ability to generate multimodal data. The decoder network no longer reconstructs the input data directly from the samples in the latent space, but generates a potential representation of each modality and reconstructs the modal data from the potential representation through an inverse transform. The present invention introduces a mixed Fourier model, which can enhance the decoder architecture of the trajectory prediction model for the trajectory of historical information and improve the generalization of the trajectory prediction model.

[0032] It should be noted that the encoder encodes the historical trajectory information through a mixed Gaussian encoder and an attention mechanism to obtain agent encoding features and lane line encoding features respectively.

[0033] Step S30: Combining the agent encoding features and the lane line encoding features, a predicted trajectory is obtained through prediction through an attention mechanism.

[0034] It can be understood that the lane line encoding features and the agent encoding features are respectively input into the cross-attention mechanism to extract relevant features, and then predicted through the self-attention mechanism to estimate the future trajectory.

[0035] In this embodiment, the historical trajectory of the agent and the center line of the lane are obtained as trajectory prediction inputs. The trajectory prediction model includes an encoder and a decoder. The encoder introduces a hybrid Fourier encoder, and combines the attention mechanism to perform feature encoding on the historical trajectory of the agent and the center line of the lane to obtain the agent encoding feature and the lane line encoding feature. The decoder includes an attention mechanism and a multi-layer perceptron. The agent encoding feature and the lane line encoding feature obtain the predicted trajectory through the attention mechanism. The introduction of the hybrid Fourier encoder can improve the generalization of the trajectory prediction model, and predict the distance probability distribution while predicting the future trajectory of the agent.

[0036] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 A flow chart of the second embodiment of the trajectory prediction method based on the hybrid Fourier encoder of the present application is provided. In the steps, the trajectory prediction method based on the hybrid Fourier encoder includes: Step S201: Obtain the historical trajectory of the agent to be predicted and the historical trajectories of other agents in its surroundings.

[0037] It should be noted that the agent history trajectory information includes the target-level history trajectory of the agent to be predicted (including the position and speed at the historical moment) and the target-level history trajectory of other agents around it:

[0038] in, It is the position and speed of the predicted intelligent agent in the central coordinate system with the current position of the predicted intelligent agent as the center and the direction of the predicted intelligent agent as the positive direction. t is the time information, and its range can be selected according to the actual situation.

[0039] Step S202: Obtain the centerline information of all lanes where the intelligent body to be predicted is located.

[0040] It should be noted that the lane centerline refers to the centerline information of all lanes near the intelligent body to be predicted, that is, the set of road center points discretely sampled at a certain distance:

[0041] Where x, y are the position coordinates of the lane centerline sampling point in the central coordinate system centered on the current position of the intelligent body to be predicted and the direction of the intelligent body to be predicted is the positive direction, and i is the lane centerline sampling point number.

[0042] Step S203: Use a mixed Gaussian encoder to extract the features of the agent's historical trajectory and the lane centerline respectively to obtain a feature matrix.

[0043] It is understandable that if Figure 4 The Q (Query), K (Key), and V (Value) shown are the three core feature vectors in the Attention Mechanism, which are used to calculate the correlation between elements in the sequence and perform weighted processing accordingly. The first feature vector is Query, which represents the trajectory information of the current time step. This Query is used to query the relevant information of other time steps.

[0044] The second feature vector, Key, is used as a benchmark for comparison with the Query. That is, in the encoded sequence given by the lane line mixed Gaussian encoder, the Query is compared with the Key of all elements to determine which elements are most relevant to the current Query.

[0045] The third feature vector, Value, is used in the final weighted calculation. Once the correlation between the Query and the Key of the elements in the sequence is determined, the Value is weighted and summed using these correlation weights to generate the final output.

[0046] Step S203 includes steps S231 to S234: Step S231: Use the trainable embedding encoder weights to multiply the agent's historical trajectory and lane centerline to obtain the dimensionally upgraded features.

[0047] Step S232: Use Fourier coding method to extract features from the dimension-upgraded features to obtain extracted features.

[0048] It can be understood that the mixed Gaussian encoder uses the trainable embedding encoder weights to multiply the input features to achieve dimensionality increase of the input features, and uses the Fourier encoding method to extract features from the dimensionality increased features:

[0049] Where V is the input agent’s historical trajectory or lane centerline, and the dimension is recorded as [num, input_dim]. Num represents the number of samples, that is, how many agent’s historical trajectory or lane centerline data there are. Input_dim is the feature dimension of each sample, which contains information used to describe the agent’s historical position or other related features. is the embedding encoder weight, whose input dimension is equal to input_dim and output dimension is hidden_dim. That is, each input feature is mapped to a new hidden space, the dimension of this space is hidden_dim.

[0050] Among them, The sine and cosine functions are applied to the result in order to introduce periodic characteristics and increase the nonlinear capability of the model. The trigonometric function operation here is performed on each element one by one.

[0051] Step S233: Use a multi-layer perception mechanism to establish a fully connected layer of the extracted features to obtain a representation adjacency matrix.

[0052] It can be understood that MLP is a multi-layer perceptron, which is the basic layer in a neural network and is used for nonlinear transformation. It is usually composed of multiple fully connected layers for further processing and decoding of the extracted features. and Each will produce an output of dimension [num, hidden_dim]. The MLP concatenates these two outputs with the original input V to form a new matrix of dimension [num, hidden_dim*2+1]. Each row of this new matrix is ​​a feature vector, which contains the original features, the features after sine transformation, and the features after cosine transformation. Therefore, the input dimension of MLP is hidden_dim*2+1, and the output dimension is hidden_dim.

[0053] After using Fourier coding, time coding is performed through a fully connected graph, that is, a multi-layer perceptron is used to construct a fully connected layer to represent the adjacency matrix:

[0054] in, Represents the node features after Fourier encoding. represents the i-th layer of the multilayer perceptron, which receives the Fourier-encoded features as input. represents the features after processing in the i-th fully connected layer, which will The output of Combined with other algorithms, the original frequency domain information is preserved and the deeper feature representation learned by MLP is preserved.

[0055] Step S234: extracting the representation adjacency matrix on the time dimension through a mathematical model to obtain a feature matrix of the mixed Fourier encoding result.

[0056] It can be understood that by extracting the important features in the time dimension through maximum pooling, the mixed Fourier coding result is obtained, that is, It is expressed as:

[0057] Step S204: Use the attention mechanism to extract the relevant information between the historical trajectory of the agent and the center line of the lane, and combine the feature matrix to obtain the agent encoding feature and the lane line encoding feature.

[0058] It should be noted that cross attention involves the interaction of attention between features from two different sources: lane line features and agent features are intertwined, and the model determines the correlation between lanes and agents through the attention mechanism. In this process, the center line of the lane may be used as the query (Q), and the agent's historical trajectory as the key (K) and value (V), or vice versa, depending on the needs of the scenario. Through cross attention, the model can capture the interactive information of the lane environment and the agent.

[0059] It can be understood that the self-attention mechanism is used to pay attention to each part of the same sequence of input. That is, each agent's features are paid attention to itself, which can be its own past historical trajectory, speed, acceleration and other features, to extract relevant spatiotemporal information. In this mechanism, Q, K, and V all come from the same agent feature set. In this way, the model can self-associate between the same features to generate new representations.

[0060] Specifically, a mixed Gaussian encoder is used to extract the features of the agent's historical trajectory and lane centerline, respectively, to obtain a feature matrix with dimension hidden_dim. Figure 4 As shown, the encoder uses the cross-attention mechanism and the self-attention mechanism to extract the correlation between the agent's historical trajectory and the lane line, and obtains the agent encoding features and the lane line encoding features.

[0061] In this embodiment, the encoder performs feature encoding on the historical trajectory information through a mixed Gaussian encoder and an attention mechanism to obtain agent encoding features and lane line encoding features respectively; wherein the mixed Gaussian encoder extracts the features of the agent's historical trajectory and the lane centerline respectively to obtain a feature matrix with a dimension of hidden_dim, and then the encoder uses a cross-attention mechanism and a self-attention mechanism to extract the correlation between the agent's historical trajectory and the lane line to obtain agent encoding features and lane line encoding features.

[0062] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can refer to the above introduction, and will not be repeated later. Figure 5 , Figure 5 A flow chart of the third embodiment of the trajectory prediction method based on the hybrid Fourier encoder of the present application is provided.

[0063] Step S301: Obtain a central modality query password, and use the central modality query password as a query value of the attention mechanism.

[0064] It is understandable that if Figure 6 As shown in the figure, the decoder uses the embedding encoder weights as the central modality query password, extracts its correlation with the lane line encoding features and the agent encoding features through the cross-attention mechanism and self-attention mechanism, and generates a predicted trajectory.

[0065] It should be noted that some status information is obtained or a query password is input, such as the status of a certain intelligent agent at the current moment, such as speed, position, etc.

[0066] Step S302: Fusing the agent coding features and the lane line coding features through a cross attention mechanism, and extracting related features of the agent coding features and the lane line coding features.

[0067] It should be noted that the agent encoding features and the lane line encoding features are fused through a cross attention mechanism. Q (Query), K (Key), and V (Value) are related features obtained from the agent encoding features and the lane line encoding features.

[0068] Step S303: Utilize the relevant features and combine with the self-attention mechanism to perform prediction to obtain a predicted trajectory.

[0069] Step S304: Decode the predicted trajectory through a multi-layer perceptron to generate and output the agent's future trajectory and predicted distance probability distribution.

[0070] It can be understood that the final output is decoded by the multi-layer perceptron MLP to generate a specific future trajectory of the agent and a predicted distance probability distribution, that is, the possible movement trajectory or next step position of the agent in the future time.

[0071] It should be noted that if Figure 6 As shown in the figure, two cross-attention mechanisms and one self-attention mechanism constitute one round of decoding process. During decoding, N rounds of decoding process are repeated to fully extract feature information and better predict trajectory. After multiple rounds of decoding, the central modality query password already contains various types of feature information, and then a multi-layer perceptron is used for decoding to extract the predicted trajectory:

[0072] in, is the central modal query password after multiple rounds of decoding, is the predicted position center of the agent at time t in the future, Then is its confidence. Thus, the probability distribution of the agent's future trajectory and predicted distance is obtained.

[0073] In this embodiment, the trajectory prediction decoder of the central modality query command uses the trainable embedding encoder weight as the center and extracts the trajectories separately through the attention mechanism. Among them, the input dimension of the embedding encoder is the number of modalities, and different modalities can be set by adjusting the input dimension. The output dimension of the embedding encoder is hidden_dim, which is consistent with the encoder encoding feature dimension. After obtaining the central modality query command, it is used as the query of the subsequent attention mechanism. After the lane line encoding features and the agent encoding features are respectively extracted with the cross attention mechanism, prediction is performed through the self-attention mechanism to estimate the future trajectory.

[0074] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of a trajectory prediction system based on a hybrid Fourier encoder proposed by the present invention. Figure 7 An embodiment of a trajectory prediction system based on a hybrid Fourier encoder is proposed in the present invention.

[0075] The trajectory prediction system based on the hybrid Fourier encoder includes the trajectory prediction method based on the hybrid Fourier encoder as described above, and the trajectory prediction system based on the hybrid Fourier encoder includes: a trajectory prediction input module 10, used to obtain the historical trajectory of the intelligent agent and the center line of the lane; a trajectory prediction model module 20, used to encode the feature of the historical trajectory of the intelligent agent and the center line of the lane through a hybrid Gaussian encoder and an attention mechanism to obtain the intelligent agent encoding feature and the lane line encoding feature; a trajectory prediction output module 30, used to combine the intelligent agent encoding feature and the lane line encoding feature, and predict through the attention mechanism to obtain the predicted trajectory.

[0076] The present application provides a trajectory prediction device based on a hybrid Fourier encoder, and the trajectory prediction device based on the hybrid Fourier encoder includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the trajectory prediction method based on the hybrid Fourier encoder in the above-mentioned embodiment 1.

[0077] Reference below Figure 8 , which shows a schematic diagram of the structure of a trajectory prediction device based on a hybrid Fourier encoder suitable for implementing an embodiment of the present application. The trajectory prediction device based on a hybrid Fourier encoder in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The trajectory prediction device based on the hybrid Fourier encoder shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0078] like Figure 8As shown, the trajectory prediction device based on the hybrid Fourier encoder may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the trajectory prediction device based on the hybrid Fourier encoder are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the trajectory prediction device based on the hybrid Fourier encoder to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a trajectory prediction device based on a hybrid Fourier encoder with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0079] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0080] The trajectory prediction device based on the hybrid Fourier encoder provided by the present application adopts the trajectory prediction method based on the hybrid Fourier encoder in the above embodiment, which can enhance the feature extraction capability of the trajectory prediction model for historical information and improve the trajectory prediction accuracy. Compared with the prior art, the beneficial effects of the trajectory prediction device based on the hybrid Fourier encoder provided by the present application are the same as the beneficial effects of the trajectory prediction method based on the hybrid Fourier encoder provided by the above embodiment, and the other technical features of the trajectory prediction device based on the hybrid Fourier encoder are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0081] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0082] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0083] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the trajectory prediction method based on the hybrid Fourier encoder in the above-mentioned embodiment.

[0084] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0085] The computer-readable storage medium may be included in the trajectory prediction device based on the hybrid Fourier encoder; or may exist independently without being assembled into the trajectory prediction device based on the hybrid Fourier encoder.

[0086] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0089] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned trajectory prediction method based on the hybrid Fourier encoder, which can enhance the feature extraction capability of the trajectory prediction model for historical information and improve the trajectory prediction accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the trajectory prediction method based on the hybrid Fourier encoder provided in the above-mentioned embodiment, and will not be described in detail here.

[0090] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A trajectory prediction method based on a hybrid Fourier encoder, characterized in that: The trajectory prediction method based on the hybrid Fourier encoder comprises: Get the agent's historical trajectory and lane centerline; By using a mixed Gaussian encoder and an attention mechanism, feature encoding is performed on the historical trajectory of the agent and the center line of the lane to obtain agent encoding features and lane line encoding features; The agent encoding features and the lane line encoding features are combined and predicted through an attention mechanism to obtain a predicted trajectory.

2. The trajectory prediction method based on a hybrid Fourier encoder as claimed in claim 1, characterized in that: The step of obtaining the historical trajectory of the intelligent agent and the lane center line includes: Obtain the historical trajectory of the agent to be predicted and the historical trajectories of other agents around it; Get the centerline information of all lanes where the agent to be predicted is located.

3. The trajectory prediction method based on hybrid Fourier encoder according to claim 1, characterized in that: The step of encoding the historical trajectory of the agent and the center line of the lane by mixing the Gaussian encoder and the attention mechanism to obtain the encoding features of the agent and the encoding features of the lane includes: Using a mixed Gaussian encoder to extract features of the agent's historical trajectory and the lane centerline, respectively, to obtain a feature matrix; The attention mechanism is used to extract the relevant information between the historical trajectory of the agent and the center line of the lane, and the agent encoding features and the lane line encoding features are obtained by combining the feature matrix.

4. The trajectory prediction method based on a hybrid Fourier encoder as claimed in claim 3, characterized in that: The step of using a mixed Gaussian encoder to respectively extract features of the agent's historical trajectory and the lane centerline to obtain a feature matrix includes: The trained embedding encoder weights are multiplied with the agent’s historical trajectory and lane centerline to obtain the dimension-enhanced features. Using Fourier coding method to extract features from the dimension-upgraded features to obtain extracted features; Using a multi-layer perception mechanism to establish a fully connected layer of the extracted features to obtain a representation adjacency matrix; The adjacency matrix representing the time dimension is extracted through a mathematical model to obtain the characteristic matrix of the mixed Fourier coding result.

5. The trajectory prediction method based on hybrid Fourier encoder according to claim 1, characterized in that: Before the step of combining the agent encoding feature and the lane line encoding feature to predict through an attention mechanism to obtain a predicted trajectory, the step includes: Obtain a central modality query password, and use the central modality query password as a query value of the attention mechanism.

6. The trajectory prediction method based on hybrid Fourier encoder according to claim 5, characterized in that: The step of combining the agent encoding feature and the lane line encoding feature to predict and obtain a predicted trajectory through an attention mechanism includes: The agent encoding feature and the lane line encoding feature are fused through a cross attention mechanism, and related features of the agent encoding feature and the lane line encoding feature are extracted; The related features are used in combination with the self-attention mechanism to obtain the predicted trajectory.

7. The trajectory prediction method based on hybrid Fourier encoder according to claim 6, characterized in that: After the step of using the relevant features in combination with the self-attention mechanism to predict the predicted trajectory, the method further includes: The predicted trajectory is decoded through a multi-layer perceptron to generate and output the agent's future trajectory and predicted distance probability distribution.

8. An intelligent vehicle automatic parking path planning system, characterized in that: The intelligent vehicle automatic parking path planning system is used to execute the trajectory prediction method based on the hybrid Fourier encoder according to any one of claims 1 to 7, and the intelligent vehicle automatic parking path planning system includes: Trajectory prediction input module, used to obtain the agent's historical trajectory and lane centerline; The trajectory prediction model module is used to encode the historical trajectory of the agent and the lane centerline through a mixed Gaussian encoder and an attention mechanism to obtain the agent encoding features and the lane line encoding features; The trajectory prediction output module is used to combine the agent encoding features and the lane line encoding features to obtain the predicted trajectory through an attention mechanism.

9. An intelligent vehicle automatic parking path planning device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the trajectory prediction method based on a hybrid Fourier encoder according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the trajectory prediction method based on a hybrid Fourier encoder as described in any one of claims 1 to 7 are implemented.

Images