A trajectory prediction method and device

Abstract

This application discloses a trajectory prediction method, apparatus, storage medium, and computer program to address the problems of low accuracy and low reliability in existing trajectory prediction methods. During the movement of a subject in the current environment, the trajectory prediction neural network acquires the input current trajectory data and the current map data of the current environment; it expresses the current trajectory data and current map data as a set of current trajectory points and a set of current map points in a high-dimensional space; it extracts global scene features based on the current trajectory point sets and current map point sets, wherein the global scene features include map features and trajectory features of the current environment; and it predicts and outputs multiple predicted trajectory point sets for the moving subject and the probability corresponding to each predicted trajectory point set based on the global scene features.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a trajectory prediction method and device, a storage medium, and a computer program. Background Art

[0002] Based on the research and development of artificial intelligence, many technical fields have been more widely applied and developed in combination with artificial intelligence, such as robotics and unmanned driving technology.

[0003] In the field of robotics and autonomous driving, trajectory prediction is an important link. Trajectory prediction largely determines the safety performance of autonomous driving systems. Taking autonomous driving as an example, during the driving process of a vehicle, trajectory prediction technology is needed to predict the driving trajectory of surrounding vehicles in the future in real time, so that the autonomous driving system can make safe path planning based on the prediction results to avoid collisions. Summary of the invention

[0004] The embodiments of the present application provide a trajectory prediction method and device, a storage medium, and a computer program to solve the problems of low accuracy and low reliability of prediction results in the trajectory prediction method in the prior art.

[0005] According to one aspect of an embodiment of the present application, a trajectory prediction method is provided, comprising: during a process in which a moving subject performs movement in a current environment, a trajectory prediction neural network obtains input current trajectory data and current map data of the current environment; wherein the current trajectory data includes a plurality of trajectory points of the moving subject and one or more other moving objects in the current environment within a predetermined time period in the past, each trajectory point includes a spatial position at a corresponding time point, and the current map data includes the spatial positions of a plurality of map points of road elements in the current environment within the predetermined time period;

[0006] The current trajectory data and the current map data are expressed as a current trajectory point set and a current map point set in a high-dimensional space, wherein the current trajectory point set includes a high-dimensional trajectory point corresponding to each trajectory point in the current trajectory data, and the current map point set includes a high-dimensional map point corresponding to each map point in the current map data;

[0007] A global scene feature is extracted based on the current trajectory point set and the current map point set, wherein the global scene feature has the map feature and trajectory feature of the current environment;

[0008] According to the global scene features, multiple predicted trajectory point sets of the moving subject and the probability corresponding to each predicted trajectory point set are predicted and output, and each predicted trajectory point set includes the spatial positions of the moving subject at multiple time points in a predetermined future time period.

[0009] According to another aspect of an embodiment of the present application, there is provided a trajectory prediction device, including an encoder and a decoder;

[0010] The encoder is used to obtain input current trajectory data and current map data of the current environment during the process of the moving subject performing movement in the current environment; wherein the current trajectory data includes multiple trajectory points of the moving subject and one or more other moving objects in the current environment within a predetermined time period in the past, each trajectory point includes a spatial position at a corresponding time point, and the current map data includes the spatial positions of multiple map points of road elements in the current environment within the predetermined time period; the current trajectory data and the current map data are expressed as a current trajectory point set and a current map point set in a high-dimensional space; global scene features are extracted based on the current trajectory point set and the current map point set, wherein the global scene features have map features and trajectory features of the current environment;

[0011] The decoder is used to predict and output multiple predicted trajectory point sets of the moving subject and the probability corresponding to each predicted trajectory point set based on the global scene features, and each predicted trajectory point set includes the spatial position of the moving subject at multiple time points in a predetermined future time period.

[0012] According to another aspect of an embodiment of the present application, a trajectory prediction device is provided, including at least one processor and at least one memory, wherein at least one memory stores at least one machine executable instruction, and after the at least one processor executes the at least one machine executable instruction, the trajectory prediction method as described above is executed.

[0013] According to another aspect of an embodiment of the present application, a computer program is provided. The computer program has a code segment configured to execute the following trajectory prediction process. The process includes the trajectory prediction method as described above.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program for trajectory prediction processing is stored, and the process includes the trajectory prediction method as described above.

[0015] According to the trajectory prediction method provided by the embodiment of the present application, the current trajectory data and current map data acquired in real time can be expressed as the current trajectory point set and the current map point set in the high-dimensional space during the real-time process of the moving subject performing the movement, and the global scene features can be obtained based on the current trajectory point set and the current map point set. The global scene features have the map features and trajectory features of the current environment, and the trajectory of the moving subject in the future predetermined time period is predicted based on the global scene features. Among them, the current trajectory data and the current map data are expressed as a data point set in a high-dimensional space, and a data point set with both time features and space features can be obtained; the global scene features integrate the map features and trajectory features of the current environment, which can significantly improve the accuracy of feature extraction. Further, based on efficient and high-precision feature extraction, the accuracy and precision of the trajectory prediction results can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present application and constitute a part of the specification. They are used to explain the present application together with the embodiments of the present application and do not constitute a limitation of the present application.

[0017] Figure 1 A processing flow chart of a trajectory prediction neural network training method provided in an embodiment of the present application;

[0018] Figure 2a A schematic diagram of the structure of a trajectory prediction neural network provided in an embodiment of the present application;

[0019] Figure 2b Another processing flow chart of the training method of the trajectory prediction neural network provided in the embodiment of the present application;

[0020] Figure 3 Another schematic diagram of the structure of the trajectory prediction neural network provided in the embodiment of the present application;

[0021] Figure 4 for Figure 2b Flow chart of the process of step 104a;

[0022] Figure 5a A schematic diagram of a dynamic object in a scene;

[0023] Figure 5b for Figure 5a A schematic diagram showing the trajectory of a dynamic object in the scene using a traditional method;

[0024] Figure 5c for Figure 5a A schematic diagram showing a scene in which the trajectory of a dynamic object is expressed using the method provided in an embodiment of the present application;

[0025] Figure 6 for Figure 2b Flow chart of the process of step 106a;

[0026] Figure 7a for Figure 6 Flow chart of the process of step 1062a;

[0027] Figure 7b for Figure 7a Flow chart of the process of step 621;

[0028] Figure 8 A schematic diagram of determining the adjacent high-dimensional map points of a high-dimensional trajectory point in an example;

[0029] Fig. 9 Schematic diagram of propagation for map relationship learning in an example;

[0030] Fig.10 for Figure 6 Flow chart of the process of step 1063a;

[0031] Fig.11 A processing flow chart of a trajectory prediction method in a real-time processing stage provided in an embodiment of the present application;

[0032] Fig.12 for Fig.11 Flow chart of the process of step 1106;

[0033] Fig.13 for Fig.12 Flow chart of the process of step 1062';

[0034] Fig.14 for Fig.12 Flow chart of the process of step 1063';

[0035] Fig.15 Another schematic diagram of the structure of the trajectory prediction neural network provided in the embodiment of the present application;

[0036] Fig.16a Another schematic diagram of the structure of the trajectory prediction neural network provided in the embodiment of the present application;

[0037] Fig.16b Another schematic diagram of the structure of the trajectory prediction neural network provided in the embodiment of the present application;

[0038] Fig.17 A flow chart of a global feature extraction operation provided in an embodiment of the present application;

[0039] Fig.18 A schematic diagram of the structure of the trajectory prediction device extracted in the embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0041] In robotics or unmanned driving technology, a very important technology is how to make safe and reliable path planning for robots or unmanned vehicles. Path planning depends on accurately and effectively predicting the motion trajectories of other dynamic objects in the environment of the robot or unmanned vehicle. Only by accurately and effectively predicting the motion trajectories of dynamic objects can safe and reliable path planning be made for robots or unmanned vehicles.

[0042] In the prior art, the motion trajectory of an object can be predicted by a variety of methods.

[0043] In a prediction method based on historical trajectory data, a neural network is built and trained using historical trajectory data to obtain a neural network that can perform trajectory prediction. In this method, the operations performed by the neural network generally include feature extraction operations and prediction operations. The feature extraction operation is used to obtain the time-space features (also known as spatiotemporal features) of the trajectory of a dynamic object (such as a vehicle, motorcycle, bicycle, or pedestrian) in the past period of time, and the prediction operation is used to predict the trajectory of the vehicle in the future based on the extracted spatiotemporal features. The feature extraction operation generally extracts multiple time points (or time features) first, then extracts the spatial features of each dynamic object at each time point, and merges the spatial features with the time features.

[0044] In the above-mentioned feature extraction operation, spatial feature extraction is a very time-consuming operation, and such a processing process cannot be applied in a real-time processing environment. In addition, the spatial connection or spatial influence between dynamic objects at different time points is abandoned, resulting in the inability of the feature extraction operation to simulate the complex connection and influence of spatial features between multiple time points. Such a prediction process can only bring unreliable prediction results. Furthermore, this method cannot handle noisy input data. Noisy input data usually refers to incomplete trajectory data of a dynamic object. This method usually has two ways to deal with incomplete trajectory data, one is to delete all trajectory data of the dynamic object, and the other is to fill in the incomplete trajectory data. Such limitations obviously affect the accuracy and precision of the prediction results, and make the above method unable to be applied in a real-time application environment.

[0045] In another method based on historical trajectory and road information, high-precision map information in the environment and the driving trajectories of different vehicles are usually represented by rasterized images from a bird's-eye view, and the pixel RGB values ​​in the rasterized images are input to the neural network. The neural network extracts features based on the input data and predicts trajectories based on the extracted features. Although this method can retain trajectory information and road structure information, the scene area that needs to be represented is large, and using images to represent it will result in the storage of excessive redundant information (for example, a large number of places where there are no vehicles will correspond to pixels in the image), which will lead to low computational efficiency of the subsequent neural network.

[0046] It can be seen that the trajectory prediction method in the existing technology has the problems of long prediction operation, low efficiency, and cannot be applied in real-time processing environment, as well as low accuracy, low reliability and low efficiency of the prediction results.

[0047] In view of this, an embodiment of the present application provides a trajectory prediction solution to solve part or all of the above problems.

[0048] This application takes unmanned driving technology as an example to illustrate the trajectory prediction solution provided by this application. It can be understood by those skilled in the art that the trajectory prediction solution provided by this application can also be applied to other technical fields, such as the field of robotics.

[0049] In this application, the term "vehicle" is broadly interpreted in this application to include any moving object, including, for example, aircraft, ships, spacecraft, cars, trucks, vans, semi-trailers, motorcycles, golf carts, off-road vehicles, warehouse transport vehicles or agricultural vehicles and transportation vehicles running on tracks, such as trams or trains and other rail vehicles. "Vehicle" in this application can generally include: power system, sensor system, control system, peripheral equipment and computer system. In other embodiments, the vehicle can include more, less or different systems.

[0050] Among them, the power system is the system that provides power movement for the vehicle, including: engine / motor, transmission and wheels / tires, and energy unit.

[0051] The control system may include a combination of devices that control the vehicle and its components, such as steering units, throttles, brake units, and other components.

[0052] Peripherals may be devices that allow the vehicle to interact with external sensors, other vehicles, external computing devices, and / or users, such as wireless communication systems, touch screens, microphones, and / or speakers.

[0053] Based on the vehicle described above, the unmanned vehicle is also equipped with a sensor system and an unmanned driving control device.

[0054] The sensor system may include multiple sensors for sensing information about the environment in which the vehicle is located, and one or more actuators for changing the position and / or orientation of the sensors. The sensor system may include any combination of sensors such as a global positioning system sensor, an inertial measurement unit, a radio detection and ranging (RADAR) unit, a camera, a laser rangefinder, a light detection and ranging (LIDAR) unit, and / or an acoustic sensor; the sensor system may also include sensors that monitor the vehicle's internal systems (e.g., an O2 monitor, a fuel gauge, an engine temperature gauge, etc.).

[0055] The unmanned driving control device may include at least one processor and at least one memory. The processor may include a central processing unit (CPU) and / or a graphics processing unit (GPU). At least one machine executable instruction is stored in the memory, and the program product including the at least one machine executable instruction may be called an unmanned driving control system. The unmanned driving control system may include a map engine, a positioning module, a perception module, a navigation or path module, and an automatic control module. Accordingly, the processor executes at least one machine executable instruction to realize a variety of corresponding functions, including map construction, positioning, perception, navigation or path planning, and vehicle control. The map engine and the positioning module are used to provide map information and positioning information. The perception module is used to perceive the environmental information of the vehicle's environment based on the information obtained by the sensor system and the map information provided by the map engine. The navigation or path module is used to plan a driving path for the vehicle based on the processing results of the map engine, the positioning module and the perception module. The automatic control module parses the decision information input from the navigation or path module and other modules and converts it into control command output for the vehicle control system, and sends the control command to the corresponding components in the vehicle control system through the vehicle network (for example, the vehicle's internal electronic network system implemented through CAN bus, local area interconnection network, multimedia directional system transmission, etc.) to achieve automatic control of the vehicle; the automatic control module can also obtain information about various components in the vehicle through the vehicle network.

[0056] In the process of planning a path for a vehicle, trajectory prediction technology is needed to predict in real time the driving trajectories of surrounding dynamic objects (such as vehicles, motorcycles, bicycles or pedestrians) in the future, so that the unmanned driving system can make safe and reliable path planning based on the prediction results.

[0057] An embodiment of the present application provides a learning-based trajectory prediction solution, which includes two stages: a training stage and a real-time processing stage.

[0058] In the training phase, a neural network is set up, which has a specific network structure and an initial parameter set. Using a supervised learning method, the neural network is trained to learn based on historical real data, and the parameter set of the neural network is obtained when the training results meet the convergence conditions. The neural network with a specific network structure and a parameter set obtained by training is the neural network for trajectory prediction, or called a trajectory prediction neural network.

[0059] In the real-time processing stage, when the trajectory of a dynamic subject is predicted, the current trajectory point set data of the dynamic subject and the surrounding dynamic objects and the current map point set data of the current environment are provided to the trajectory prediction neural network. The neural network processes the motion data and predicts multiple future trajectory point sets of the dynamic subject and the probability corresponding to each trajectory point set.

[0060] The technical solution of the present application is described below through multiple embodiments.

[0061] Train the trajectory prediction neural network

[0062] In the training phase, a neural network is constructed, which may include multiple network layers, and an initial parameter set is set for the neural network. The number of network layers in the neural network and the initial parameter set can be specifically set according to the needs of the application scenario.

[0063] In the embodiment of the present application, the constructed neural network is trained by using the historically authentic first trajectory data and the second trajectory data, as well as the historical map data.

[0064] In an embodiment of the present application, the first trajectory data and the second trajectory data include multiple trajectory points of the moving subject and one or more moving objects around, and each trajectory point includes a spatial position at a corresponding time point. The spatial position can be a two-dimensional position, a three-dimensional position, or a high-dimensional position. For example, when a vehicle is traveling on a relatively flat road, the amplitude of the road elevation change is very small, the elevation information can be ignored, and the trajectory data can only include the two-dimensional position of the moving subject and the moving object. In other fields, for example, for robots for special purposes, there may be a large amplitude elevation change during the movement process, and the trajectory data may include the three-dimensional position of the moving subject and the moving object. In other application scenarios, when there is or needs to be considered the position information of other dimensions, the trajectory data may also include high-dimensional position data. For the sake of simplicity and convenience, the two-dimensional position is used as an example for explanation and description in this application. It can be understood by those skilled in the art that when the spatial position is a three-dimensional position or a high-dimensional position, the three-dimensional position or the high-dimensional position can be subjected to corresponding position conversion, coordinate conversion, etc., or the corresponding position processing and coordinate processing can be performed according to the needs of the specific application scenario.

[0065] In an embodiment of the present application, the first trajectory data and the second trajectory data are data sets that are adjacent in time. For example, the first trajectory data is a data set for the i-th time period, and the second trajectory data is a data set for the i+1-th time period. The duration of the i-th time period is m, and the duration of the i+1-th time period is n. In some embodiments, m and n may be equal, for example, both include a duration of 3 seconds. In other embodiments, m and n may also be unequal, for example, the duration of the i-th time period is 3 seconds, and the duration of the i+1-th time period is 2 seconds. In the i-th time period, multiple time points are obtained according to the frequency u, and in the i+1-th time period, multiple time points are obtained according to the frequency v. u and v may be equal or unequal.

[0066] The historical map data includes the spatial positions of multiple map points of road elements in the environment where the moving subject is located within the time period corresponding to the first trajectory data (for example, the i-th time period). The historical map data can come from a high-precision map during the vehicle's driving process. Road elements can be lane lines, sidewalks, road edges, ramps, intersections, roundabouts and other elements. In this application, road elements such as lane lines are sampled at a predetermined distance to obtain a series of map points, and each map point records the spatial position of the point. Corresponding to the spatial position of the above-mentioned trajectory points, the spatial position of the map point can also be a two-dimensional position, a three-dimensional position or a high-dimensional position.

[0067] Figure 1 The processing flow of the training method of the trajectory prediction neural network provided in the embodiment of the present application is shown in FIG. 1 . The training process includes multiple iterative trainings, and one iterative training includes the following processing.

[0068] Step 102: Provide the historical first trajectory data and the historical map data as input to the neural network.

[0069] Step 104: The neural network expresses the first trajectory data and the historical map data as a first trajectory point set and a historical map point set in a high-dimensional space;

[0070] Step 106: extracting global scene features based on the first trajectory point set and the historical map point set; wherein the global scene features have map features and trajectory features of the current environment;

[0071] Step 108: predict and output multiple trajectory point sets of the moving subject and the probability corresponding to each trajectory point set according to the global scene features, each trajectory point set including the spatial position of the moving subject at multiple time points in a predetermined future time period;

[0072] Step 110: for each output trajectory point set, use the output trajectory point set and the second trajectory data as inputs of a loss function to obtain an output of a corresponding loss function;

[0073] Step 112: determine whether the output of all loss functions meets the preset convergence condition. If it is determined that the convergence condition is met, the process proceeds to step 116. If it is determined that the convergence condition is not met, the process returns to step 114.

[0074] Step 114, adjusting the parameters of the neural network, and returning to step 102;

[0075] Step 116: Determine that the training is completed, and obtain a parameter set of the trained neural network.

[0076] The following is an exemplary embodiment of Figure 1 The processing process shown is described below.

[0077] Assume that the number of moving subjects and moving objects is N, and the first trajectory data includes T time points. represents the two-dimensional position of the moving subject or object n at time point t, t∈{1,…,T}, R 2 is a two-dimensional space representation. The trajectory data of a moving subject or a moving object n can be expressed as X n ={H n , F n},in It represents the historical position of the object from the past time point 1 to the observation time point T. Indicates the future position of the object from the future time point T+1 to the time point T+s. n can be regarded as the first trajectory point set, F n It can be regarded as the second trajectory point set.

[0078] Assume that the historical map point set includes The time period corresponding to the first trajectory data is from time point 1 to observation time point T. The map data of the lane line is Indicates The lane lines include The map locations of the sampled map points, Indicates Sampling map points in lane lines 's map location.

[0079] The purpose of trajectory prediction is to and map data The predicted trajectory point set of multimodal distribution is predicted, which can be Create the best match.

[0080] In this exemplary embodiment, Figure 2a As shown, the structure of the neural network may include an encoder 1 and a decoder 2 according to the functions to be implemented. In this exemplary embodiment, the training process of the neural network includes the following steps: Figure 2b The process shown:

[0081] Step 102a: Provide the historical first trajectory data and the historical map data as input to the encoder 1.

[0082] Step 104a, the encoder 1 expresses the first trajectory data and the historical map data as a first trajectory point set and a historical map point set in a high-dimensional space;

[0083] Step 106a: The encoder 1 extracts global scene features based on the first trajectory point set and the historical map point set; wherein the global scene features have map features and trajectory features of the environment where the moving subject is located.

[0084] Step 108a: The decoder 2 predicts and outputs multiple trajectory point sets of the moving subject and the probability corresponding to each trajectory point set according to the global scene features, and each trajectory point set includes the spatial positions of the moving subject at multiple time points in a predetermined future time period.

[0085] Step 110: for each output trajectory point set, use the output trajectory point set and the second trajectory point set as inputs of a loss function to obtain an output of a corresponding loss function;

[0086] Step 112: determine whether the output of all loss functions meets the preset convergence condition. If it is determined that the convergence condition is met, the process proceeds to step 116. If it is determined that the convergence condition is not met, the process returns to step 114.

[0087] Step 114, adjusting the parameters of the neural network, and the process returns to step 102a;

[0088] Step 116: Determine that the training is completed, and obtain a parameter set of the trained neural network.

[0089] The processing of steps 104a to 108a is described below through the structural settings of the neural network in some embodiments.

[0090] In some embodiments, Figure 3 As shown, the input representation layer 11 can be set in the encoder 1 to implement step 104a. Figure 4 As shown, step 104a can be implemented as the following processing:

[0091] Step 1041a: the input representation layer 11 expresses the first trajectory data in a predetermined unordered trajectory data format to obtain an unordered first trajectory point set;

[0092] Step 1042a: Express the historical map data in a predetermined unordered map data format to obtain an unordered historical map point set.

[0093] In step 1041a, the preset trajectory data format can be specifically set according to the needs of the specific application scenario. In one example, for simplicity and scalability, the first trajectory data can be represented using the trajectory data format shown in the following formula (1).

[0094]

[0095] Where n∈{1,…,N} is the moving subject and one or more moving objects, t∈{1,…,T} is multiple time points, is the two-dimensional position of object n at time t (i.e., a trajectory point). is the spatial position of object n at time t Mapped to a corresponding point in the high-dimensional space, P is the first trajectory point set mapped in the high-dimensional space.

[0096] In step 1042a, the historical map point set is represented using the map data format shown in the following formula (2).

[0097]

[0098] in, For the Sampling map points in lane lines Map location of The lane lines include sampling map points, the historical map point set includes Lane line segment. For a sampling map point Mapped to a corresponding point in the high-dimensional space, M is a set of historical map points mapped in the high-dimensional space.

[0099] In an exemplary scenario, the input representation layer 11 uses formula (1) to map the data in the first trajectory data to a high-dimensional space (eg, a spatiotemporal coordinate system). Figure 5a As shown, the dynamic objects include two vehicles, a bicycle and a pedestrian. Figure 5bThe figure shows the situation of expressing the trajectory of the dynamic object in the scene by the traditional expression method, that is, expressing the two-dimensional position of the object in the two-dimensional space coordinate system. In the two-dimensional space coordinate system, each data point has only spatial features. Figure 5c The data expression method provided in the embodiment of the present application is used to express the first trajectory point set of the scene in a time-space coordinate system. In the coordinate system, the x-axis and the y-axis represent the two-dimensional space coordinate axes, and the z-axis is the time axis. Each data point in the coordinate system has both time characteristics and space characteristics.

[0100] By using a predetermined unordered and extensible data format to express the first trajectory data, the position data of the moving subject and the moving object that occurs in time sequence can be mapped to the space-time coordinate system, that is, the ordered data is mapped to a discrete unordered point set. The high-dimensional first trajectory point set expressed in the space-time coordinate system does not change with the change of the expression method or expression order of the first trajectory data, and the data of an object cannot be expressed due to the existence of missing data of the object. It can be seen that by mapping the first trajectory data to a high-dimensional space, the problem of requiring additional processing when there is noise in the input data in the prior art can be overcome, and the robustness of processing missing input data can be improved.

[0101] Moreover, all data in the first trajectory data can be uniformly expressed in the same high-dimensional space (such as a spatiotemporal coordinate system) through a predetermined data format, and the time features and the space features can be unified into the same expression, so that subsequent feature extraction processing can extract unified and fused time features and space features.

[0102] The input representation layer 11 can express the historical map data as a discrete and disordered point set in a high-dimensional space (such as a spatiotemporal coordinate system) by using a predetermined disordered data format, i.e., formula (2). The map point set expressed in the spatiotemporal coordinate system does not change with the change of the expression method or expression order of the road data, and the road element will not be unable to be expressed due to missing data in the road element.

[0103] On the other hand, by expressing the trajectory data of moving subjects and objects, and the map data of road elements as discrete, disordered high-dimensional data point sets in high-dimensional space, compared with the existing technology of representing high-precision map information and driving trajectories of different vehicles through rasterized images, and inputting the pixel RGB values ​​in the rasterized images into the neural network to extract features, the amount of data processed by the neural network can be significantly reduced, and the processing speed and efficiency of the neural network can be improved.

[0104] In some embodiments, Figure 3As shown, an embedding layer 12, a relationship learning layer 13, a global feature extraction layer 14 and a feature output layer 15 may be set in the encoder 1 to implement step 106a. Figure 6 As shown, step 106a can be implemented as the following processing:

[0105] Step 1061a, the embedding layer 12 embeds the first trajectory point set into a predetermined high-dimensional space to obtain high-dimensional trajectory point features; and embeds the historical map point set into a predetermined high-dimensional space to obtain high-dimensional map point features.

[0106] The purpose of the embedding operation is to transform the high-dimensional trajectory points in the high-dimensional space Mapping to hidden trajectory features The trajectory feature can unify the temporal and spatial features of the data. In some embodiments, the embedding layer 12 can be implemented by a multilayer perceptron (MLP). The MLP may include a fully connected multilayer network, and the activation function of the neurons in the network may be a ReLU activation function. Furthermore, a batch normalization layer (BN) may be set after each layer of the embedding layer.

[0107] For the convenience of explanation, the operation of embedding (or mapping) the first trajectory point set into the high-dimensional space can be expressed by the following formula (3). p A multi-layer perceptron.

[0108]

[0109] Similarly, the embedding principle of historical map point set is similar. The high-dimensional map point can be embedded into the historical map point set by the following formula (4): Mapping as map features Among them, MLP m A multi-layer perceptron.

[0110]

[0111] The embedding operation embeds the high-dimensional first trajectory point set and the high-dimensional historical map point set into a high-dimensional space composed of multiple dimensions, and the data in each dimension includes time features and spatial features.

[0112] Step 1062a, the relationship learning layer 13 obtains local scene features according to the first trajectory point set, the historical map point set, the high-dimensional trajectory point features and the high-dimensional map point features; the local scene features are high-dimensional trajectory point features with local map information;

[0113] In one example, if Figure 7a As shown, step 1062a can be implemented as the following processing:

[0114] Step 621, the relationship learning layer 13 determines, based on the first trajectory point set and the historical map point set, a plurality of map spatial relationships between each high-dimensional trajectory point and a plurality of predetermined high-dimensional map points around the high-dimensional trajectory point;

[0115] In one example, a network for performing computational processing may be provided in the relationship learning layer 13 to implement step 621, such as Figure 7b As shown, the implementation process of step 621 may include:

[0116] Step S1: determining a predetermined number of high-dimensional map points whose Euclidean distances to a high-dimensional trajectory point are within a preset distance threshold.

[0117] In one example, Figure 8 As shown, part (a) represents the trajectory points in the first trajectory data and the map points in the historical map data. After the first trajectory data and the historical map data are represented by the input representation layer 11, high-dimensional trajectory points and high-dimensional map points as shown in part (b) are obtained. Step S1 determines to obtain multiple high-dimensional map points adjacent to a high-dimensional trajectory point, as shown in part (c).

[0118] Fig. 9 This association is represented by an intuitive expression. There are four neighboring high-dimensional map points within a preset distance of: The arrow from the high-dimensional map point to the high-dimensional trajectory point represents the process of using relational learning to propagate local map features to trajectory features and obtain relative spatial relationships, which is the following step S2.

[0119] Step S2: respectively determine the relative spatial relationship between the high-dimensional trajectory point and each of the multiple high-dimensional map points, wherein a relative spatial relationship includes an absolute position and a relative position of the high-dimensional trajectory point and a high-dimensional map point, and a relative distance and a relative direction between the trajectory point and the map point.

[0120] in, is the absolute position of the high-dimensional trajectory point, is the absolute position of the high-dimensional map point, is the relative position between the high-dimensional trajectory point and the high-dimensional map point, is the relative distance between the high-dimensional trajectory point and the high-dimensional map point, is the relative direction between the high-dimensional trajectory point and the high-dimensional map point, It is the velocity of the moving subject or object n at time point t.

[0121] The relative spatial relationship between a high-dimensional trajectory point and a high-dimensional map point It can be expressed as the following formula (5):

[0122]

[0123] Step 623: embed each relative spatial relationship corresponding to a high-dimensional trajectory point into the high-dimensional space to obtain a high-dimensional map spatial relationship, perform dot multiplication on each high-dimensional map spatial relationship and the corresponding high-dimensional map point feature to obtain a weighted map feature; maximize multiple weighted map features of the high-dimensional trajectory point and select the maximum weighted map feature;

[0124] Among them, multiple relative spatial relationships corresponding to a high-dimensional trajectory point are all embedded into the high-dimensional space. This operation can refer to the embedding operation of the above formula (3) or (4). An embedding layer (such as a multi-layer perceptron MLP) can be set in the relationship learning layer 13 to perform the embedding operation. After the embedding operation, multiple high-dimensional map spatial relationships can be obtained. After performing dot multiplication of the multiple high-dimensional map spatial relationships with the high-dimensional trajectory point, multiple dot multiplication results, that is, multiple weighted map features, can be obtained in the relationship learning layer 13. A network layer that implements dot multiplication calculation can be set in the relationship learning layer 13 to perform the dot multiplication operation. After maximizing these multiple dot multiplication results, the maximum weighted map feature can be obtained. A pooling layer (such as a maximum pooling layer or an average pooling layer) can be set in the relationship learning layer 13 to achieve maximization processing.

[0125] Step 625: Add the maximum weighted map feature to the high-dimensional trajectory point feature corresponding to the high-dimensional trajectory point to obtain a local scene feature.

[0126] The processing of step 623 and step 625 can be expressed by formula (6).

[0127]

[0128] in, is a local scene feature, symbol A represents maximization processing, MLP e In order to embed the relative spatial relationship, is a high-dimensional map point feature, and k is a predetermined number of map points whose Euclidean distance to a trajectory point is within a preset distance threshold.

[0129] Through Figure 7aThe processing can propagate the map information of multiple high-dimensional map points around a high-dimensional trajectory point to the high-dimensional trajectory point, that is, propagate the local map information around a high-dimensional trajectory point to the high-dimensional trajectory point, and obtain the local scene features with local map information and the spatiotemporal information of the trajectory point. The local scene features provide the data basis for obtaining a more accurate predicted trajectory point set in the future.

[0130] Step 1063a: The global feature extraction layer 14 performs global feature extraction operations on the local scene features and the high-dimensional map point features respectively, and extracts global trajectory features and global map features.

[0131] In one embodiment, Fig.10 As shown, step 1063a can be implemented as the following processing:

[0132] Step 631: extract features from each feature in the local scene feature, maximize the extracted features, and select the maximum feature as the global trajectory feature;

[0133] Step 633: extract features from each feature in the high-dimensional map point feature, maximize the extracted features, and select the maximum feature as the global map feature.

[0134] After obtaining local scene features and high-dimensional map point features, it is necessary to maintain invariance for different data arrangements. The feature extraction layer extracts features from data of various dimensions in the high-dimensional space to obtain global map features that integrate temporal features and spatial features, as well as global trajectory features that integrate local map features; global features include multi-dimensional features, and each dimension corresponds to a part of the feature space, that is, a part of the expression of spatiotemporal features.

[0135] In some embodiments, the global feature extraction layer 14 may include a pooling layer, through which the global trajectory features and the global map features are extracted in sequence, or may include two parallel pooling layers, through which the global trajectory features and the global map features are extracted respectively.

[0136] Step 1064a, the feature output layer 15 determines the global scene feature according to the global trajectory feature and the global map feature.

[0137] The feature output layer 15 adds the global trajectory features and the global map features to obtain the global scene features. The global scene features include the global map features and the global trajectory features, which provide a data basis for obtaining a more accurate predicted trajectory point set in the future.

[0138] After decoder 1 determines to obtain the global scene features, decoder 2 may perform prediction processing according to the global scene features.

[0139] In some embodiments, as shown in FIG. 3 , a multi-head decoder 21 and a probability determination layer 22 may be provided in the decoder 2 .

[0140] The multi-head decoder 21 may include multiple sub-decoders, and the sub-decoders may be recurrent neural networks or multi-layer perceptrons. The recurrent neural network includes a long short-term memory network (Long-Short Term Memory, LSTM) or a gated recurrent unit GRU. In some other embodiments, a stochastic decoder may also be selected.

[0141] Each sub-decoder has a specific initial parameter set, and obtains a predicted trajectory point set according to the global scene feature prediction, and the multi-head decoder 21 outputs multiple predicted trajectory point sets.

[0142] The probability determination layer 22 determines a corresponding probability for each predicted trajectory point set. The probability determination layer 22 may be a classifier.

[0143] In the above step 110, for each output trajectory point set, the output trajectory point set and the second trajectory point set are used as inputs of the loss function to obtain a corresponding output of the loss function.

[0144] The loss function may be a negative log likelihood (NLL) function of a Gaussian distribution.

[0145] Through the above training, a neural network parameter set can be obtained. The neural network with the trained parameter set and a specific structure is a trajectory prediction neural network. Through the trajectory prediction neural network, the input current trajectory point set and current map point set can be processed in the real-time processing stage to predict multiple predicted trajectory point sets in the future time period and the probability corresponding to each predicted trajectory point set.

[0146] Through the above training process, by mapping the first historical trajectory data into the first trajectory point set in the high-dimensional space and mapping the historical map data into the historical map point set in the high-dimensional space, the ordered trajectory data is transformed into disordered data points, which can overcome the subsequent impact of data noise and improve the robustness of data processing; the feature extraction of trajectory data and map data is realized by end-to-end learning, which can significantly improve the speed and efficiency of feature extraction operations and better meet the real-time requirements of application scenarios; the extracted global scene features have global trajectory features and global map features, the global trajectory features integrate local map features and trajectory features, and the trajectory features have both time features and spatial features; the predicted trajectory point set obtained based on the global scene feature prediction can significantly improve the precision and accuracy of the trajectory prediction results.

[0147] Trajectory prediction in real-time processing

[0148] and Figure 1 The processing shown corresponds to Fig.11 The process of predicting the motion trajectory of an object through a trajectory prediction neural network in the real-time processing stage is shown.

[0149] Step 1102: during the process of the moving subject performing movement in the current environment, the trajectory prediction neural network obtains input current trajectory data and current map data of the current environment; wherein the current trajectory data includes multiple trajectory points of the moving subject and one or more other moving objects in the current environment within a predetermined time period in the past, each trajectory point includes a spatial position at a corresponding time point, and the current map data includes the spatial positions of multiple map points of road elements in the current environment within the predetermined time period;

[0150] Step 1104: expressing the current trajectory data and the current map data as a current trajectory point set and a current map point set in a high-dimensional space, wherein the current trajectory point set includes a high-dimensional trajectory point corresponding to each trajectory point in the current trajectory data, and the current map point set includes a high-dimensional map point corresponding to each map point in the current map data;

[0151] Step 1106: extracting global scene features based on the current trajectory point set and the current map point set, wherein the global scene features have map features and trajectory features of the current environment;

[0152] Step 1108: Predict and output multiple predicted trajectory point sets of the moving subject and the probability corresponding to each predicted trajectory point set based on the global scene features, wherein each predicted trajectory point set includes the spatial positions of the moving subject at multiple time points in a predetermined future time period.

[0153] The following is an exemplary embodiment of Fig.11 The processing shown is explained in detail. In this example, the neural network can be as follows Figure 3 The neural network shown in FIG. The neural network includes an encoder 1 and a decoder 2. The encoder 1 includes an input representation layer 11, an embedding layer 12, a relationship learning layer 13, a global feature extraction layer 14 and a feature output layer 15. The decoder 2 includes a multi-head decoder 21 and a probability determination layer 22.

[0154] In the above step 1102, the current trajectory data input to the trajectory prediction neural network may come from an upstream module, such as a perception module. The perception module selects the current trajectory data from the perception data and provides the current trajectory data to the encoder 1 of the trajectory prediction neural network. The current map data input to the trajectory prediction neural network may be high-precision map data from an upstream map module.

[0155] The settings of the current trajectory data may have the same settings as those of the first trajectory data described above, and the settings of the current map data may have the same settings as those of the current map data described above.

[0156] The above step 1104 can be performed by Figure 3 The input representation layer 11 is shown to perform step 1104 .

[0157] In this example, the input representation layer 11 expresses the current trajectory data in a predetermined unordered trajectory data format to obtain an unordered current trajectory point set mapped in a high-dimensional space. The trajectory data format may be the format shown in the above formula (1), and the current trajectory point set includes the high-dimensional trajectory point corresponding to each trajectory point in the current trajectory data; the current map data is expressed in a predetermined unordered map data format to obtain an unordered current map point set mapped in a high-dimensional space. The map data format may be the format shown in the above formula (2), and the current map point set includes the high-dimensional map point corresponding to each map point in the current map data.

[0158] The specific processing process of step 1104 can refer to the above step 104a, formula (1) and formula (2) and Figure 4 processing.

[0159] In this example, the above step 1106 can be performed as follows: Figure 3 The embedding layer 12, the relationship learning layer 13, the global feature extraction layer 14 and the feature output layer 15 shown in FIG. Figure 6 The processing shown is performed. Fig.12 As shown, step 1106 may include:

[0160] Step 1061', the embedding layer 12 embeds the current trajectory point set into a predetermined high-dimensional space to obtain high-dimensional trajectory point features, and embeds the current map point set into a predetermined high-dimensional space to obtain high-dimensional map point features. The specific operation of the embedding layer 12 can be implemented using the above formula (3) and formula (4) and referring to the above step 1061a.

[0161] Step 1062', the relationship learning layer 13 obtains the local scene feature according to the current trajectory point set, the current map point set, the high-dimensional trajectory point feature and the high-dimensional map point feature, and the local scene feature is the high-dimensional trajectory point feature with local map information. The specific operation of the relationship learning layer 13 can be implemented by referring to the processing of the above step 1062a.

[0162] Step 1063', the global feature extraction layer 14 performs global feature extraction operations on the local scene features and the high-dimensional map point features respectively, and extracts global trajectory features and global map features. The specific operation of the global feature extraction layer 14 can be implemented with reference to the above step 1063a.

[0163] Step 1064': the feature output layer 15 determines the global scene feature according to the global trajectory feature and the global map feature. The specific operation of the feature output layer 15 can be implemented with reference to the above step 1064a.

[0164] For example, reference Figure 7a and step 1062a above, such as Fig.13 As shown, the processing of the relational learning layer 13 in step 1062' may include:

[0165] Step 621', determine multiple relative spatial relationships between each high-dimensional trajectory point and multiple high-dimensional map points around the high-dimensional trajectory point according to the current trajectory point set and the current map point set; this process can refer to Figure 7b The processing of step 621 includes: determining a predetermined number of high-dimensional map points whose Euclidean distances to a high-dimensional trajectory point are within a preset distance threshold; respectively determining a relative spatial relationship between the high-dimensional trajectory point and each of the high-dimensional map points, wherein a relative spatial relationship includes an absolute position and a relative position of the high-dimensional trajectory point and a high-dimensional map point, and a relative distance and a relative direction between the trajectory point and the map point. The expression of the relative spatial relationship can refer to the above formula (5).

[0166] Step 623', embed each map space relationship corresponding to a high-dimensional trajectory point into the high-dimensional space to obtain a high-dimensional map space relationship, perform dot multiplication on each high-dimensional map space relationship and the corresponding high-dimensional map point feature to obtain a weighted map feature; perform maximization processing on multiple weighted map features of the high-dimensional trajectory point, and select the maximum weighted map feature;

[0167] Step 625', add the maximum weighted map feature and the high-dimensional trajectory point feature corresponding to the high-dimensional trajectory point to obtain a local scene feature.

[0168] The operations of step 623 ′ and step 625 ′ may be implemented with reference to the above-mentioned step 623 and step 625 .

[0169] The local scene feature corresponding to a high-dimensional trajectory point can be expressed using the above formula (6). Multiple high-dimensional trajectory points in the current trajectory point set can obtain corresponding multiple local scene features.

[0170] For example, reference Fig.10 and the above step 1063a, such as Fig.14 As shown, the processing of the global feature extraction layer 14 in step 1063' includes:

[0171] Step 631', extracting features from each feature in the local scene features, maximizing the extracted features, and selecting the maximum feature as the global trajectory feature;

[0172] Step 633': extract features from each feature in the high-dimensional map point feature, maximize the extracted features, and select the maximum feature as the global map feature.

[0173] Exemplarily, referring to the above step 1064a, the processing of the feature output layer 15 may include: adding the global trajectory feature and the global map feature to obtain the global scene feature.

[0174] Through the above processing, the current trajectory data is mapped to the current trajectory point set in the high-dimensional space, and the current map data is mapped to the current map point set in the high-dimensional space, and the ordered trajectory data is converted into disordered data points, which can overcome the subsequent impact of data noise and improve the robustness of data processing; the feature extraction of trajectory data and map data is realized through end-to-end processing, which can significantly improve the speed and efficiency of feature extraction operations and better meet the real-time requirements of application scenarios; the extracted global scene features have global trajectory features and global map features, the global trajectory features integrate local map features and trajectory features, the trajectory features have both time features and space features, and the global scene features can provide an accurate and effective data basis for trajectory prediction.

[0175] In this example, in the process of step 1108 , the multi-head decoder 21 in the decoder 21 outputs a plurality of predicted trajectory point sets, and the probability determination layer 22 determines a corresponding probability for each predicted trajectory point set.

[0176] Decoder 2 can predict a trajectory point set with higher precision and accuracy based on more accurate and effective global scene features.

[0177] In some other embodiments, based on any of the above embodiments, the data items in the first trajectory data, the second trajectory data, and the current trajectory data may be divided and expanded, and the data items may be divided into trajectory data and attribute data. Accordingly, the trajectory data format used by the input representation layer 11 may be correspondingly expanded.

[0178] In one example, the trajectory data may include the spatial position and velocity of the object at multiple time points in a predetermined time period. In this case, based on the scalability of the predetermined data format, the trajectory data format may be defined using formula (7).

[0179]

[0180] in, is the velocity of the object at time t.

[0181] Experimental and test results show that adding speed data to trajectory data can further improve the accuracy of trajectory prediction results.

[0182] In another exemplary embodiment, the trajectory point set may include trajectory data and attribute data. The attribute data may include the category data of the object, such as whether the object is a pedestrian, a bicycle, or a vehicle. In a specific application scenario, the category data of the object may also be other attributes, such as the attributes of a vehicle including a van, a tractor, a semi-trailer, etc.; the attribute items of the headlights may also be added, such as the attribute data of the headlights or the attribute data of the taillights, and the attribute data of the headlights may be used to identify whether the headlights are turned on; and more may be added. In this case, the trajectory data format may be defined using formula (8).

[0183]

[0184] in, Indicates the category of the object. The value of the category can be preset, for example, 1 for vehicles, 2 for pedestrians, and 3 for bicycles. Formula (8) exemplarily expands one attribute data. According to the requirements of specific application scenarios, multiple attribute data can also be expanded.

[0185] The attribute data may also include identification data of the object, such as whether the object is a moving subject or a surrounding moving object. In this case, the trajectory data format can be defined using formula (9).

[0186]

[0187] Among them, id identifies the object as a moving subject or a surrounding moving object. For example, when the value of this data is 1, it means that object n is a moving subject, and when the value is 0, it means that object n is a surrounding moving object.

[0188] Since the predetermined extensible data format can include multiple data according to the needs of the application scenario, the length of the data format is variable. In the prior art, if it is necessary to increase or decrease data items in the input data, the R&D personnel are required to manually arrange and fine-tune the data, which is a huge workload. The embodiment of the present application provides a data format with a variable length, and the data can be subsequently processed by simply adding or decreasing data items in the data format, thus avoiding the extra work caused by manual arrangement.

[0189] Furthermore, adding attribute data to the trajectory data can train the trajectory prediction network to learn and understand the attributes of the object, thereby improving the precision and accuracy of trajectory prediction.

[0190] In some other embodiments, based on any of the above embodiments, the data items in the historical map data and the current map data may be divided and expanded. Accordingly, the map data format used by the input presentation layer 11 is correspondingly expanded.

[0191] In one example, the map data may also include the position difference between each map point and other adjacent map points on the same road element where each map point is located. In this case, based on the scalability of the predetermined data format, the map data format may be defined using formula (10).

[0192]

[0193] For map points The adjacent previous map point The position difference between them, such as the position difference between two-dimensional coordinates or the position difference between three-dimensional coordinates.

[0194] By setting the position difference between two adjacent map points, the directionality and connectivity between multiple map points of road elements can be obtained in the subsequent processing of the trajectory prediction neural network.

[0195] In this example, after the embedding layer 12 processes the high-dimensional map point features, that is, after the above step 1061a and the above step 1061', the high-dimensional map point features are enhanced in terms of directionality and connectivity. Fig.15 As shown, it can be Figure 3 On the basis of the neural network shown, an enhancement layer 12' is provided after the embedding layer 12. The enhancement layer 12' may be a self-attention neural network.

[0196] In another example, the map data may also include attribute data of each map point, where the attribute data includes the category of the road element to which the map point belongs and the sequence number of the map point among multiple map points included in the road element to which the map point belongs within the predetermined time period.

[0197] In some other embodiments, based on any of the above embodiments, in order to extract more accurate and effective global trajectory features and global map features, multiple global feature extraction operations may be performed, that is, multiple global feature extraction operations may be performed in the above step 1063a and the above step 1063'. Fig.16a or Fig.16b As shown, in Figure 3 Or Fig.15On the basis of the neural network shown, a plurality of stacked sub-global feature extraction layers 141 are set in the global feature extraction layer 14.

[0198] The global feature extraction layer 14 performs multiple global feature extraction operations, each of which includes:

[0199] After the previous adjacent sub-global feature extraction layer 141 splices the global trajectory feature extracted from the local scene feature to the high-dimensional trajectory point feature, the spliced ​​high-dimensional trajectory point feature is output to the next adjacent sub-global feature extraction layer 141;

[0200] The adjacent previous sub-global feature extraction layer 141 splices the global map features extracted from the high-dimensional map point features into the high-dimensional map point features, and then outputs the spliced ​​high-dimensional map point features to the adjacent next sub-global feature extraction layer 141 .

[0201] Fig.17 A schematic diagram of a global feature extraction operation is shown. A global feature extraction is performed on a point-level feature (i.e., a local scene feature or a high-dimensional map point feature) to obtain an aggregated feature (i.e., a global trajectory feature or a global map feature obtained by a single extraction). The global feature extraction is spliced ​​to the point-level feature input to the global feature extraction operation of the word to obtain a spliced ​​local scene feature and a spliced ​​high-dimensional map point feature. The spliced ​​local scene feature and the spliced ​​high-dimensional map point feature are input to the next sub-global feature extraction layer 141 to perform the next global feature extraction operation.

[0202] When the encoder 1 includes a global feature extraction layer 14, the global feature extraction layer 14 can extract the first sequence information of the environment where the moving subject is located, that is, the global spatiotemporal features, and the second sequence information of the environment, such as the mutual influence between different objects, can be learned and extracted through multiple global feature extraction layers 14. The stacked global feature extraction layers 14 can more deeply understand the state and relationship between individual objects and the global spatiotemporal environment, so that the stacked feature layers can capture the interaction and influence between multiple objects.

[0203] For example, when the front car slows down, it will affect the following car to slow down accordingly. This effect and influence can be captured through the stacked feature layers.

[0204] By setting a stacked global feature extraction layer 14 in the encoder 1, the interactions and influences between multiple objects can be extracted, improving the learning and understanding ability of the neural network to the environment, thereby further improving the accuracy and precision of trajectory prediction.

[0205] The present application also provides a trajectory prediction device, which can be the above-mentioned Figure 2a , Figure 3 , Fig.15 , Fig.16a or Fig.16b The structure shown, the corresponding network structure refers to the above description, and the above trajectory prediction processing is completed accordingly.

[0206] Fig.18 FIG. 8 shows a structure of an exemplary trajectory prediction device. For example, the device includes a processor 81 and a memory 82. In the real-time processing process, the processor 81 executes at least one machine executable instruction stored in the memory 82, such as Figures 11 to 14 During the training process, the processor 81 executes at least one machine executable instruction stored in the storage 82, and performs the following steps: Figure 1 , Figure 2b , Figure 4 , Figure 6 , Figure 7a , Figure 7b , Fig.10 Processing shown.

[0207] The embodiment of the present application also provides a computer program having a code segment configured to perform trajectory prediction processing, the processing including: Figures 11 to 14 The processing shown may further include: Figure 1 , Figure 2b , Figure 4 , Figure 6 , Figure 7a , Figure 7b , Fig.10 Processing shown.

[0208] The present application also provides a storage medium in which a computer program for trajectory prediction processing is stored. The processing includes: Figures 11 to 14 The processing shown may further include Figure 1 , Figure 2b , Figure 4 , Figure 6 , Figure 7a , Figure 7b , Fig.10 Processing shown.

[0209] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A trajectory prediction method, characterized in that: include: During the process of the moving subject performing movement in the current environment, the trajectory prediction neural network obtains input current trajectory data and current map data of the current environment; wherein the current trajectory data includes multiple trajectory points of the moving subject and one or more other moving objects in the current environment within a predetermined time period in the past, each trajectory point includes a spatial position at a corresponding time point, and the current map data includes the spatial positions of multiple map points of road elements in the current environment within the predetermined time period; The current trajectory data and the current map data are expressed as a discrete, unordered current trajectory point set and a current map point set in a high-dimensional space, wherein the current trajectory point set includes a high-dimensional trajectory point corresponding to each trajectory point in the current trajectory data, and the current map point set includes a high-dimensional map point corresponding to each map point in the current map data; A global scene feature is extracted based on the current trajectory point set and the current map point set, wherein the global scene feature has the map feature and trajectory feature of the current environment; According to the global scene features, multiple predicted trajectory point sets of the moving subject and the probability corresponding to each predicted trajectory point set are predicted and output, and each predicted trajectory point set includes the spatial positions of the moving subject at multiple time points in a predetermined future time period.

2. The method according to claim 1, characterized in that The trajectory prediction neural network expresses the current trajectory data and the current map data as discrete, unordered current trajectory point sets and current map point sets in high-dimensional space, including: The current trajectory point set is expressed in a predetermined unordered trajectory data format to obtain an unordered current trajectory point set; The current map point set is expressed in a predetermined unordered map data format to obtain an unordered current map point set.

3. The method according to claim 1, characterized in that The trajectory prediction neural network extracts global scene features based on the current trajectory point set and the current map point set, including: Embed the current trajectory point set into a predetermined high-dimensional space to obtain high-dimensional trajectory point features, and embed the current map point set into a predetermined high-dimensional space to obtain high-dimensional map point features; A local scene feature is obtained according to the current trajectory point set, the current map point set, the high-dimensional trajectory point feature and the high-dimensional map point feature, where the local scene feature is a high-dimensional trajectory point feature with local map information; Perform global feature extraction operations on local scene features and high-dimensional map point features respectively to extract global trajectory features and global map features; The global scene features are determined based on the global trajectory features and the global map features.

4. The method according to claim 3, characterized in that The trajectory prediction neural network also enhances the directionality and connectivity of high-dimensional trajectory point features and high-dimensional map point features.

5. The method according to claim 3, characterized in that: The local scene features are obtained based on the current trajectory point set, the current map point set, the high-dimensional trajectory point features and the high-dimensional map point features, including: Determine, according to the current trajectory point set and the current map point set, a plurality of relative spatial relationships between each high-dimensional trajectory point and a plurality of predetermined high-dimensional map points around the high-dimensional trajectory point; Each map spatial relationship corresponding to a high-dimensional trajectory point is respectively embedded into the high-dimensional space to obtain a high-dimensional map spatial relationship, and each high-dimensional map spatial relationship is multiplied by the corresponding high-dimensional map point feature to obtain a weighted map feature; multiple weighted map features of the high-dimensional trajectory point are maximized and selected to obtain the maximum weighted map feature; A local scene feature is obtained by adding the maximum weighted map feature to the high-dimensional trajectory point feature corresponding to the high-dimensional trajectory point.

6. The method according to claim 5, characterized in that Determining multiple relative spatial relationships between each high-dimensional trajectory point and multiple high-dimensional map points predetermined around the high-dimensional trajectory point according to the current trajectory point set and the current map point set includes: determining a predetermined number of multiple high-dimensional map points having a Euclidean distance to a high-dimensional trajectory point within a preset distance threshold; A relative spatial relationship between the high-dimensional trajectory point and each of the multiple high-dimensional map points is determined respectively, wherein a relative spatial relationship includes an absolute position and a relative position of the high-dimensional trajectory point and a high-dimensional map point, and a relative distance and a relative direction between the trajectory point and the map point.

7. The method according to claim 3, characterized in that The global trajectory features and global map features are extracted from the local scene features and high-dimensional map point features respectively, including: Extract features from each feature in the local scene feature, maximize the extracted features, and select the maximum feature as the global trajectory feature; Extract features from each feature of the high-dimensional map point features, maximize the extracted features, and select the maximum feature as the global map feature.

8. The method according to claim 3 or 7, characterized in that: The trajectory prediction neural network extracts global trajectory features and global map features from local scene features and high-dimensional map point features, including: The trajectory prediction neural network performs multiple global feature extraction operations. Each global feature extraction operation includes: After the global trajectory features extracted from the local scene features are spliced ​​into the high-dimensional trajectory point features, the spliced ​​high-dimensional trajectory point features are output to the next global feature extraction operation; After the global map features extracted from the high-dimensional map point features are spliced ​​to the high-dimensional map point features, the spliced ​​high-dimensional map point features are output to the next global feature extraction operation.

9. The method according to claim 3, characterized in that: The global scene features are determined based on the global trajectory features and the global map features, including: The global trajectory features and the global map features are added together to obtain the global scene features.

10. The method according to claim 1, characterized in that The current trajectory data also includes the speed of the moving subject and other one or more moving objects at each trajectory point.

11. The method according to claim 1, characterized in that: The current trajectory data also includes: Attribute data of the moving subject and each moving object, the attribute data including object category data of the moving subject and each moving object.

12. The method according to claim 11, characterized in that When the moving subject or object is a vehicle, the attribute data further includes one or more of the following: headlight attribute data, taillight attribute data.

13. The method according to claim 1, characterized in that The current map data also includes: the position difference between each map point and other adjacent map points on the same road element where it is located.

14. The method according to claim 13, characterized in that The current map data also includes: attribute data of each map point, the attribute data including the category of the road element to which the map point belongs, and the sequence number of the map point among the multiple map points included in the road element to which the map point belongs within the predetermined time period.

15. The method according to claim 1, characterized in that The trajectory prediction neural network obtains the input current trajectory data and current map data, including: Get the input current trajectory data and current map data from the upstream module.

16. The method according to claim 1, characterized in that The processing of training the trajectory prediction neural network includes: In the training phase, the preset neural network is iteratively trained multiple times according to the historical first trajectory data, the second trajectory data and the historical map data; wherein the first trajectory data and the second trajectory data are data sets that are adjacent to each other in time, and both include multiple trajectory points of the moving subject and one or more surrounding moving objects within a predetermined time period, and each trajectory point includes a spatial position at a corresponding time point; the historical map data includes the spatial positions of multiple map points of road elements in the environment where the moving subject is located within the time period corresponding to the first trajectory data; the preset neural network has an initial parameter set; One iteration of training includes: The first trajectory data and the historical map data are provided as inputs to the neural network; the neural network expresses the first trajectory data and the historical map data as a first trajectory point set and a historical map point set in a high-dimensional space; a global scene feature is extracted based on the first trajectory point set and the historical map point set; wherein the global scene feature has a map feature and a trajectory feature of the environment where the moving subject is located; and a plurality of trajectory point sets of the moving subject and a probability corresponding to each trajectory point set are predicted and output based on the global scene feature, wherein each trajectory point set includes the spatial position of the moving subject at a plurality of time points in a predetermined future time period; Each output trajectory point set and the corresponding probability and the second trajectory data are respectively provided as input to a corresponding loss function to obtain the output of the loss function; whether the outputs of all loss functions meet the preset convergence conditions; if it is judged that the convergence conditions are met, the training is determined to be completed and the parameter set of the trained neural network is obtained; if it is judged that the convergence conditions are not met, the parameters of the neural network are adjusted and the next iterative training is performed.

17. A trajectory prediction device, characterized in that: Includes encoder and decoder; The encoder is used to obtain input current trajectory data and current map data of the current environment during the process of the moving subject performing movement in the current environment; wherein the current trajectory data includes multiple trajectory points of the moving subject and one or more other moving objects in the current environment in the past predetermined time period, each trajectory point includes a spatial position at a corresponding time point, and the current map data includes the spatial positions of multiple map points of road elements in the current environment in the predetermined time period; the current trajectory data and the current map data are expressed as a discrete, unordered current trajectory point set and a current map point set in a high-dimensional space; global scene features are extracted based on the current trajectory point set and the current map point set, wherein the global scene features have map features and trajectory features of the current environment; The decoder is used to predict and output multiple predicted trajectory point sets of the moving subject and the probability corresponding to each predicted trajectory point set based on the global scene features, and each predicted trajectory point set includes the spatial position of the moving subject at multiple time points in a predetermined future time period.

18. The device according to claim 17, characterized in that The encoder includes an input representation layer; The encoder expresses the current trajectory data and the current map data as discrete, unordered current trajectory point sets and current map point sets in high-dimensional space, including: The input representation layer expresses the current trajectory point set in a predetermined unordered trajectory data format to obtain an unordered high-dimensional current trajectory point set; The current map point set is expressed in a predetermined unordered map data format to obtain an unordered current map point set.

19. The device according to claim 17, characterized in that The encoder includes an embedding layer, a map relationship learning layer, a global feature extraction layer, and a feature output layer; The encoder extracts global scene features based on the current trajectory point set and the current map point set, including: The embedding layer is used to embed the current trajectory point set into a predetermined high-dimensional space to obtain high-dimensional trajectory point features, and embed the current map point set into a predetermined high-dimensional space to obtain high-dimensional map point features; The map relationship learning layer is used to obtain local scene features based on the current trajectory point set, the current map point set, the high-dimensional trajectory point features and the high-dimensional map point features. The local scene features are high-dimensional trajectory point features with local map information. The global feature extraction layer is used to perform global feature extraction operations on local scene features and high-dimensional map point features respectively, and extract global trajectory features and global map features; The feature output layer is used to determine the global scene features based on the global trajectory features and the global map features.

20. The device according to claim 19, characterized in that The encoder also includes an enhancement layer; The enhancement layer is used to enhance the directionality and connectivity of high-dimensional trajectory point features and high-dimensional map point features; the enhancement layer includes a self-attention neural network.

21. The device according to claim 19, characterized in that The map relationship learning layer obtains local scene features based on the current trajectory point set, the current map point set, the high-dimensional trajectory point features, and the high-dimensional map point features, including: Determine, according to the current trajectory point set and the current map point set, a plurality of relative spatial relationships between each high-dimensional trajectory point and a plurality of predetermined high-dimensional map points around the high-dimensional trajectory point; Each map spatial relationship corresponding to a high-dimensional trajectory point is respectively embedded into the high-dimensional space to obtain a high-dimensional map spatial relationship, and each high-dimensional map spatial relationship is multiplied by the corresponding high-dimensional map point feature to obtain a weighted map feature; multiple weighted map features of the high-dimensional trajectory point are maximized and selected to obtain the maximum weighted map feature; A local scene feature is obtained by adding the maximum weighted map feature to the high-dimensional trajectory point feature corresponding to the high-dimensional trajectory point.

22. The device according to claim 21, characterized in that The map relationship learning layer determines multiple relative spatial relationships between each high-dimensional trajectory point and multiple predetermined high-dimensional map points around the trajectory point based on the current trajectory point set and the current map point set, including: determining a predetermined number of multiple high-dimensional map points having a Euclidean distance to a high-dimensional trajectory point within a preset distance threshold; A relative spatial relationship between the high-dimensional trajectory point and each of the multiple high-dimensional map points is determined respectively, wherein a relative spatial relationship includes an absolute position and a relative position of the high-dimensional trajectory point and a high-dimensional map point, and a relative distance and a relative direction between the trajectory point and the map point.

23. The device according to claim 19, characterized in that The global feature extraction layer extracts global trajectory features and global map features from local scene features and high-dimensional map point features, including: Extract features from each feature in the local scene feature, maximize the extracted features, and select the maximum feature as the global trajectory feature; Extract features from each feature of the high-dimensional map point features, maximize the extracted features, and select the maximum feature as the global map feature.

24. The device according to claim 19 or 23, characterized in that The global feature extraction layer includes a plurality of stacked sub-global feature extraction layers; After the global trajectory features extracted by the previous adjacent sub-global feature extraction layer from the local scene features are spliced ​​into the high-dimensional trajectory point features, the spliced ​​high-dimensional trajectory point features are output to the next adjacent sub-global feature extraction layer; The adjacent previous sub-global feature extraction layer splices the global map features extracted from the high-dimensional map point features to the high-dimensional map point features, and then outputs the spliced ​​high-dimensional map point features to the adjacent next sub-global feature extraction layer.

25. The device according to claim 19, characterized in that The feature output layer determines the global scene features based on the global trajectory features and global map features, including: The feature output layer adds the global trajectory features and the global map features to obtain the global scene features.

26. The device according to claim 17, characterized in that The decoder includes a multi-head decoder and a probability determination layer; The multi-head decoder is used to predict and output multiple predicted trajectory point sets of the moving subject according to the global scene characteristics; the multi-head decoder includes multiple recurrent neural networks or multi-layer perceptrons, and the recurrent neural network includes a long short-term memory network or a gated recurrent unit; The probability determination layer is used to determine the probability corresponding to each predicted trajectory point set.

27. A trajectory prediction device, characterized in that: The method comprises a processor and at least one memory, wherein at least one machine executable instruction is stored in the at least one memory, and the processor executes the at least one machine executable instruction to implement the method according to any one of claims 1 to 16.

28. A computer program, characterized in that The computer program has a code segment configured to execute the following trajectory prediction process, which includes the method according to any one of claims 1 to 16.

29. A non-volatile storage medium, characterized in that: At least one machine executable instruction is stored, and after the at least one machine executable instruction is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images