An open scenario-oriented agent motion pattern prediction method and device

Abstract

The application discloses an open-scene-oriented agent motion mode prediction method and device, and aims to solve the technical problem that the existing agent motion mode prediction technology results in poor feature fusion effect. The method comprises the following steps: based on a preset multi-scale history aggregation encoder and a plurality of preset efficient autoregressive decoders, processing multi-modal features, i.e. an acquired agent history motion mode sequence, an agent future motion planning sequence, a monitoring video dataset and a city road map, and outputting an initial prediction result; and combining a preset binary Gaussian mixture model with the initial prediction result to generate a target agent motion mode prediction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an open-scene-oriented agent motion pattern prediction method and device. BACKGROUND

[0002] The motion pattern recognition technology for crowds, motor vehicles and other agents is widely used in the fields of intelligent monitoring, robot navigation, automatic driving and the like.

[0003] Open-scene-oriented motion pattern prediction refers to, in an open scene such as a city road or a public place, accurately recognizing the motion behavior patterns of all agents in the scene through real-time data collection and intelligent analysis to predict possible future motion trends, thereby providing a dynamic collision risk warning for system decision-making of a mobile robot or an autonomous vehicle and further making a safer path planning.

[0004] Existing agent motion pattern prediction technologies mostly adopt an autoregressive generation method, that is, a future motion pattern is gradually predicted through a deep learning method. However, when processing multi-modal features, this method usually simply connects features of different modalities together, and this simple feature connection method cannot fully capture the dynamic interaction relationship between multi-modal features, resulting in poor feature fusion effect. SUMMARY

[0005] The present application provides an open-scene-oriented agent motion pattern prediction method and device, which are used to solve the technical problem of poor feature fusion effect caused by existing agent motion pattern prediction technologies.

[0006] The first aspect of the present application provides an open-scene-oriented agent motion pattern prediction method, comprising:

[0007] obtaining an agent historical motion pattern sequence, an agent future motion planning sequence, a monitoring video dataset and a city road map, and performing map encoding on the city road map to generate a map feature vector;

[0008] generating a plurality of two-dimensional spatial position coordinates according to an original annotation file of the monitoring video dataset, and performing homogeneous coordinate transformation and normalization on the plurality of two-dimensional spatial position coordinates to generate a plurality of motion state information;

[0009] generating a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates by using a preset spatial edge rule;

[0010] respectively, the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence are embedded to generate a historical motion time step embedding corresponding to the agent historical motion pattern sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence;

[0011] The historical motion time step embedding, the future motion time step embedding and the edge time step embedding are encoded by using a preset multi-scale historical aggregation encoder to generate a multi-scale historical motion feature corresponding to the historical motion time step embedding, a multi-scale future motion feature corresponding to the future motion time step embedding and a multi-scale edge feature corresponding to the edge time step embedding;

[0012] An initial prediction result is output by a plurality of preset efficient autoregressive decoders according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion pattern sequence;

[0013] A target agent motion pattern prediction result is generated by using a preset binary Gaussian mixture model according to the initial prediction result.

[0014] Optionally, the city road map is encoded to generate a map feature vector, comprising:

[0015] The city road map is divided to output a plurality of map subblocks;

[0016] Each of the map subblocks is transformed to generate a subblock vector corresponding to each of the map subblocks;

[0017] Each of the subblock vectors is linearly transformed to output a linear embedding corresponding to each of the subblock vectors;

[0018] A preset position encoding matrix is added to each of the linear embeddings to generate an intermediate embedding corresponding to each of the linear embeddings;

[0019] Based on a multi-head self-attention mechanism, attention weight calculation is performed on each of the map subblocks to determine an attention weight corresponding to each of the map subblocks;

[0020] Each of the attention weights is added to the intermediate embedding corresponding to the attention weight to generate a target embedding corresponding to each of the intermediate embeddings;

[0021] Each of the target embeddings is layer normalized to determine a normalized embedding corresponding to each of the target embeddings;

[0022] A map feature vector is generated by using a multi-layer perceptron according to a plurality of the normalized embeddings.

[0023] Optionally, the generating the time-dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates based on the preset spatial edge rule comprises:

[0024] performing smoothing processing on the plurality of motion state information and the plurality of two-dimensional spatial position coordinates to generate a plurality of smoothed motion state information and a plurality of smoothed two-dimensional spatial position coordinates;

[0025] constructing a node set based on the plurality of smoothed motion state information and the plurality of smoothed two-dimensional spatial position coordinates;

[0026] constructing an edge set based on a plurality of nodes in the node set according to the preset spatial edge rule;

[0027] determining the time-dimension edge sequence based on the edge set.

[0028] Optionally, the embedding processing on the agent historical motion pattern sequence, the agent future motion planning sequence and the time-dimension edge sequence respectively comprises:

[0029] performing one-dimensional convolution on the agent historical motion pattern sequence, the agent future motion planning sequence and the time-dimension edge sequence respectively to output historical motion convolution features corresponding to the agent historical motion pattern sequence, future motion convolution features corresponding to the agent future motion planning sequence and edge convolution features corresponding to the time-dimension edge sequence;

[0030] performing nonlinear mapping on the historical motion convolution features, the future motion convolution features and the edge convolution features respectively to generate historical motion label embeddings corresponding to the historical motion convolution features, future motion label embeddings corresponding to the future motion convolution features and edge label embeddings corresponding to the edge convolution features;

[0031] performing position embedding processing on the agent historical motion pattern sequence, the agent future motion planning sequence and the time-dimension edge sequence respectively to output historical motion position embeddings corresponding to the agent historical motion pattern sequence, future motion position embeddings corresponding to the agent future motion planning sequence and edge position embeddings corresponding to the time-dimension edge sequence;

[0032] performing weighted summation on the historical motion label embeddings and the historical motion position embeddings to determine the historical motion time step embedding;

[0033] weighting sum of the future motion marker embedding and the future motion position embedding, to determine a future motion time step embedding;

[0034] weighting sum of the edge marker embedding and the edge position embedding, to determine an edge time step embedding.

[0035] Optionally, the preset multi-scale history aggregation encoder comprises a self-attention submodule, a first multi-scale branch module, a second multi-scale branch module, and a third multi-scale branch module; the data processing process of the preset multi-scale history aggregation encoder is specifically as follows:

[0036] slicing the input time step embedding input to the preset multi-scale history aggregation encoder, to generate a first sub-feature, a second sub-feature, and a third sub-feature;

[0037] inputting the input time step embedding as an input of the self-attention submodule, to output a first self-attention feature;

[0038] performing multi-scale feature extraction on the input time step embedding by using the first multi-scale branch module, to generate a first multi-scale feature;

[0039] performing multi-scale feature extraction on the first sub-feature by using the second multi-scale branch module, to generate a second multi-scale feature;

[0040] inputting the second sub-feature into the third multi-scale branch module for multi-scale feature extraction, to generate a third multi-scale feature;

[0041] inputting the third sub-feature as an input of the self-attention submodule, to output a second self-attention feature;

[0042] performing layer normalization on the first self-attention feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the second self-attention feature, respectively, to output a first self-attention normalized feature, a first multi-scale normalized feature, a second multi-scale normalized feature, a third multi-scale normalized feature, and a second self-attention normalized feature;

[0043] concatenating the first self-attention normalized feature, the first multi-scale normalized feature, the second multi-scale normalized feature, the third multi-scale normalized feature, and the second self-attention normalized feature, to generate an output multi-scale feature.

[0044] Optionally, the inputting the input time step embedding as an input of the self-attention submodule, to output a first self-attention feature, comprises:

[0045] based on a multi-head self-attention mechanism, performing attention calculation on the input time step embedding, to determine a multi-head self-attention corresponding to the input time step embedding;

[0046] performing element-wise addition on the multi-head self-attention and the input time step embedding, to output a multi-head self-attention time step feature;

[0047] performing layer normalization on the multi-head self-attention time step feature, to generate a multi-head self-attention normalized feature;

[0048] performing one-dimensional convolution on the multi-head self-attention normalized feature, to output a multi-head self-attention one-dimensional convolution feature;

[0049] performing nonlinear mapping on the multi-head self-attention one-dimensional convolution feature, to generate a multi-head self-attention nonlinear feature;

[0050] performing one-dimensional convolution on the multi-head self-attention nonlinear feature, to output a first self-attention feature.

[0051] Optionally, the third multi-scale branch module comprises a self-attention sub-module and a self-attention distillation down-sampling sub-module; the inputting of the second sub-feature into the third multi-scale branch module for multi-scale feature extraction to generate a third multi-scale feature comprises:

[0052] inputting the second sub-feature as an input of the self-attention sub-module, to output a self-attention feature corresponding to the second sub-feature;

[0053] adopting the self-attention distillation down-sampling sub-module to perform distillation down-sampling on the self-attention feature corresponding to the second sub-feature, to generate a second sub-feature maximum pooling feature;

[0054] inputting the second sub-feature maximum pooling feature as an input of the self-attention sub-module, to output the third multi-scale feature.

[0055] Optionally, the adopting of the self-attention distillation down-sampling sub-module to perform distillation down-sampling on the self-attention feature corresponding to the second sub-feature, to generate a second sub-feature maximum pooling feature, comprises:

[0056] performing one-dimensional convolution on the self-attention feature corresponding to the second sub-feature, to output a second sub-feature convolution feature;

[0057] performing batch normalization on the second sub-feature convolution feature, to generate a second sub-feature normalized feature;

[0058] performing down-sampling on the second sub-feature normalized feature, to output a second sub-feature down-sampling feature;

[0059] performing maximum pooling on the second sub-feature down-sampling feature, to generate a second sub-feature maximum pooling feature.

[0060] Optionally, the data processing process of the preset high-efficiency autoregressive decoder comprises the following steps:

[0061] The input history motion pattern sequence input into the preset high-efficiency autoregressive decoder is mapped by a multi-layer perception to generate a high-dimensional feature of history motion;

[0062] The high-dimensional feature of history motion and the input edge feature, the input map feature and the input future motion feature input into the preset high-efficiency autoregressive decoder are spliced to generate a spliced feature;

[0063] A global adaptive one-dimensional average pooling operation is performed on the spliced feature to generate a global feature representation;

[0064] The global feature representation is time-activated to output a time-activated feature;

[0065] The time-activated feature and the spliced feature are multiplied element by element to generate a time sequence feature representation;

[0066] The time sequence feature representation is mapped by a multi-layer perception to output a high-dimensional feature of time sequence;

[0067] The high-dimensional feature of time sequence is subjected to layer normalization and nonlinear mapping to output a nonlinear feature of time sequence;

[0068] The nonlinear feature of time sequence and the prediction result of history time are added element by element to generate a prediction result of current time;

[0069] The prediction result of current time and a target sub-sequence in the input history motion pattern sequence are spliced to generate a new input history motion pattern sequence;

[0070] The new input history motion pattern sequence is subjected to time step embedding and position encoding to output an encoded feature sequence;

[0071] Based on a single-layer mask multi-head probability sparse self-attention mechanism, the encoded feature sequence is subjected to attention operation to output a probability sparse self-attention;

[0072] The probability sparse self-attention and the encoded feature sequence are added element by element to output a probability sparse self-attention feature sequence;

[0073] The probability sparse self-attention feature sequence is subjected to layer normalization to determine a query matrix;

[0074] The input history motion feature input into the preset high-efficiency autoregressive decoder is linearly projected to generate a key matrix and a value matrix;

[0075] According to the query matrix, the key matrix and the value matrix, a high-efficiency decoding feature sequence is generated;

[0076] The high-efficiency decoding feature sequence is layer normalized to output a high-efficiency decoding normalized feature sequence;

[0077] The high-efficiency decoding normalized feature sequence is subjected to feature mapping by using a multi-layer perception to generate an output prediction result.

[0078] The second aspect of the application provides an open scene-oriented agent motion mode prediction device, comprising:

[0079] An acquisition module is configured to acquire an agent historical motion mode sequence, an agent future motion planning sequence, a monitoring video dataset and a city road map, and perform map encoding on the city road map to generate a map feature vector;

[0080] An according module is configured to generate a plurality of two-dimensional spatial position coordinates according to an original annotation file of the monitoring video dataset, and perform homogeneous coordinate transformation and normalization on the plurality of two-dimensional spatial position coordinates to generate a plurality of motion state information;

[0081] An adopting module is configured to generate a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates by using a preset spatial edge rule;

[0082] An embedding module is configured to respectively perform embedding processing on the agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence to generate a historical motion time step embedding corresponding to the agent historical motion mode sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence;

[0083] An encoding module is configured to respectively encode the historical motion time step embedding, the future motion time step embedding and the edge time step embedding by using a preset multi-scale historical aggregation encoder to generate a multi-scale historical motion feature corresponding to the historical motion time step embedding, a multi-scale future motion feature corresponding to the future motion time step embedding and a multi-scale edge feature corresponding to the edge time step embedding;

[0084] A decoding module is configured to output an initial prediction result according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion mode sequence by using a plurality of preset high-efficiency autoregressive decoders;

[0085] An output result module is configured to generate a target agent motion mode prediction result according to the initial prediction result by using a preset binary Gaussian mixture model.

[0086] The third aspect of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the open scenario-oriented agent motion pattern prediction method according to any one of the above.

[0087] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the steps of the open scenario-oriented agent motion pattern prediction method according to any one of the above.

[0088] The fifth aspect of the present application provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, wherein when the program instructions are executed by a computer, the computer executes the steps of the open scenario-oriented agent motion pattern prediction method according to any one of the above.

[0089] From the above technical solutions, the present application has the following advantages:

[0090] The technical scheme of the present application provides an open-scene-oriented agent motion mode prediction method, which first acquires an agent historical motion mode sequence, an agent future motion planning sequence, a monitoring video dataset and a city road map, and encodes the city road map to generate a map feature vector; then, according to the original annotation file of the monitoring video dataset, a plurality of two-dimensional spatial position coordinates are generated, and the plurality of two-dimensional spatial position coordinates are subjected to homogeneous coordinate transformation and normalization to generate a plurality of motion state information; a preset spatial edge rule is used to generate a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates; the agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence are subjected to embedding processing respectively to generate a historical motion time step embedding corresponding to the agent historical motion mode sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence; a preset multi-scale historical aggregation encoder is used to encode the historical motion time step embedding, the future motion time step embedding and the edge time step embedding respectively to generate a multi-scale historical motion feature corresponding to the historical motion time step embedding, a multi-scale future motion feature corresponding to the future motion time step embedding and a multi-scale edge feature corresponding to the edge time step embedding; an initial prediction result is outputted by a plurality of preset efficient autoregressive decoders according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion mode sequence; finally, a preset bivariate Gaussian mixture model is used to generate a target agent motion mode prediction result according to the initial prediction result; based on the above scheme, the multi-modal features, i.e. the agent historical motion mode sequence, the agent future motion planning sequence, the monitoring video dataset and the city road map acquired, are processed by the preset multi-scale historical aggregation encoder and the plurality of preset efficient autoregressive decoders to output the initial prediction result, and then the preset bivariate Gaussian mixture model is combined to generate the target agent motion mode prediction result according to the initial prediction result, so that the feature fusion in the time dimension can be optimized in depth, thereby improving the multi-modal feature fusion effect. BRIEF DESCRIPTION OF DRAWINGS

[0091] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the drawings needed in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0092] Figure 1 A step flow chart of an open-scene-oriented agent motion mode prediction method provided for the first embodiment of the present application;

[0093] Figure 2 The structural schematic diagram of the preset multi-scale history aggregation encoder provided for the embodiment one of the present application is shown in the figure.

[0094] Figure 3 The overall framework diagram of the agent motion mode prediction method facing the open scene provided for the embodiment one of the present application is shown in the figure.

[0095] Figure 4 The schematic diagram of the time compression and excitation (TSE module) provided for the embodiment one of the present application is shown in the figure.

[0096] Figure 5 The training flowchart of the preset binary Gaussian mixture model provided for the embodiment two of the present application is shown in the figure.

[0097] Figure 6 The flowchart of the motion mode prediction using the preset binary Gaussian mixture model provided for the embodiment two of the present application is shown in the figure.

[0098] Figure 7 The structural block diagram of the agent motion mode prediction device facing the open scene provided for the embodiment three of the present application is shown in the figure. DETAILED DESCRIPTION

[0099] The embodiment of the present application provides an agent motion mode prediction method and device facing an open scene, which is used for solving the technical problem that the existing agent motion mode prediction technology causes poor feature fusion effect.

[0100] In order to make the technical scheme of the present application more apparent and easy to understand, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the embodiments described below are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0101] Please refer to Figure 1 , Figure 1 The step flowchart of the agent motion mode prediction method facing the open scene provided for the embodiment one of the present application is shown in the figure.

[0102] The agent motion mode prediction method facing the open scene provided by the present application comprises:

[0103] In step 101, the historical motion mode sequence of the agent, the future motion planning sequence of the agent, the monitoring video data set and the city road map are acquired, and the city road map is encoded to generate a map feature vector.

[0104] It should be noted that the present application simultaneously supports intelligent monitoring, automatic driving, mobile robots and other multi-type use scenarios. Users can select one of the following two types of data sets according to the needs for model training or model prediction: a roof bird's-eye view monitoring video data set, and a large-scale data set providing complete sensor suite data for an autonomous vehicle (such as camera / laser radar / millimeter wave radar / inertial measurement unit / GPS data).

[0105] Further, according to the factors that have a key influence on the motion pattern in the multi-agent scene of the data set, the input data source required for modeling and feature extraction of model training is determined. When using a video data set with a third-person perspective, the input source required for modeling and feature extraction in model training includes historical motion patterns and social interactions between agents in the video scene. When using a data set of a mobile agent with complete sensor suite data, the input source required for modeling and feature extraction in model training includes: historical motion patterns, past interactions between self and other agents; city road map (such as grid map); future motion planning of self.

[0106] Further, the kinematic properties required for motion pattern modeling are calculated according to the original annotations of the data set. Since the original annotation data of a large-scale autonomous driving data set is very large and complex, the present application takes a monitoring video type data set as an example for illustration.

[0107] Further, for the physical modeling of the motion pattern, the present application defines a time step, and defines the motion state at different times in the motion for different types of agents. It is assumed that there are agents in the scene, forming a set . The current time is denoted as .

[0108] The historical time step is represented by a set of discrete time steps, and is defined as: . Wherein, represents a specific time in the historical data, ranging from -H (the start of the observation window) to 0 (the most recent observation time).

[0109] The future time step is also represented by a set of discrete time steps, and is defined as: . Wherein, represents a future time, ranging from 1 (the first time step after the most recent observation time) to T (the end of the prediction window).

[0110] Historical state definition: for the agent in the scene ( ), the actual state at the historical time is defined as: . Wherein, The dimension of the agent state varies according to the specific application scenario. For example, the state dimension of a pedestrian is 3 (position, velocity, and acceleration); the state dimension of a vehicle is 4 (position, velocity, acceleration, and heading angle).

[0111] Future state definition: at a future time , the predicted state of the th agent only contains position information, defined as: The actual state of the ego agent at a future time includes position, velocity, acceleration, and heading angle, defined as: .

[0112] Based on the above, the physical representation of different types of motion pattern data is generated: the historical motion pattern (sequence of agent historical motion patterns) is , representing the state of the th agent at the historical time step; the predicted future motion pattern is , representing the predicted state of the th agent at the future time step; the future motion planning of the ego agent (sequence of agent future motion plans) is

[0113] Further, the process of map encoding of the urban road map to generate the map feature vector can be implemented by performing the following sub-steps S11 to S18:

[0114] Step S11, divide the urban road map to output a plurality of map sub-blocks;

[0115] Step S12, transform each map sub-block to generate a sub-block vector corresponding to each map sub-block;

[0116] Step S13, linearly transform each sub-block vector to output a linear embedding corresponding to each sub-block vector;

[0117] Step S14, add a preset position encoding matrix to each linear embedding to generate an intermediate embedding corresponding to each linear embedding;

[0118] Step S15, based on the multi-head self-attention mechanism, calculate the attention weight of each map sub-block to determine the attention weight corresponding to each map sub-block;

[0119] Step S16, add each attention weight to the intermediate embedding corresponding to the attention weight to generate a target embedding corresponding to each intermediate embedding;

[0120] Step S17, normalize each target embedding to determine a normalized embedding corresponding to each target embedding;

[0121] Step S18, generating a map feature vector according to the plurality of normalized embeddings by using a multilayer perceptron.

[0122] It should be noted that first, the image of the input map (urban road map) is divided into 25x25 pixel fixed size sub-blocks, and all image sub-blocks are converted into corresponding sub-block vectors. Each sub-block vector is linearly embedded to match the specified dimension, i.e. a plurality of linear embeddings are obtained, a preset position encoding matrix is added to the linear embedding, a plurality of intermediate embeddings are obtained, and a multi-head self-attention mechanism is applied to calculate the attention weight of each patch (map sub-block) relative to other patches. The output of the self-attention of each map sub-block is added to its corresponding intermediate embedding, then layer normalization is performed, and finally, a multilayer perceptron (MLP, Multilayer Perceptron) is used to process the embedding of each sub-block and map to a specified output size, denoted as a map feature vector .

[0123] Step 102, generating a plurality of two-dimensional spatial position coordinates according to the original annotation file of the monitoring video dataset, and performing homogeneous coordinate transformation and normalization on the plurality of two-dimensional spatial position coordinates to generate a plurality of motion state information.

[0124] The motion state information includes a velocity component and an acceleration.

[0125] It should be noted that the two-dimensional spatial position coordinates of the plurality of agents in each frame are extracted from the original annotation file of the monitoring video dataset, denoted as and represent the horizontal coordinate and the vertical coordinate, respectively.

[0126] Further, if the coordinate units are inconsistent, unit conversion and calibration are required to ensure the uniformity of the model input. In some cases, the original coordinates may be pixel coordinates (such as image plane units) rather than world coordinates (such as actual geographic units). Since the present application is based on the world coordinate system, homogeneous coordinate transformation is required at this time. The homogeneous coordinate conversion formula is: wherein, represents the homogeneous form of the pixel coordinates, is the transformation matrix from pixel coordinates to world coordinates.

[0127] Further, the converted world coordinates need to be normalized, i.e. wherein, is the scaling factor in the homogeneous coordinates; represents the normalized world coordinates.

[0128] Further, the normalized world coordinates are used to calculate the motion state information. It is assumed that the frame rate is fixed at The calculation formula of the velocity component (including the transverse velocity component and the longitudinal velocity component) is:

[0129] , ;

[0130] wherein, and represent the transverse and longitudinal coordinates of the next frame; and represent the transverse and longitudinal coordinates of the current frame; represents the time interval between frames (fixed according to the data frame rate).

[0131] Further, on the basis of the known velocity calculation formula, the calculation formula of the acceleration (including the transverse acceleration and the longitudinal acceleration) is as follows:

[0132] , ;

[0133] wherein, and represent the transverse and longitudinal velocities of the next frame; and represent the transverse and longitudinal velocities of the current frame.

[0134] Step 103, generating a time-dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates by using a preset spatial edge rule.

[0135] Specifically, step 103 can include the following sub-steps S31-S34:

[0136] Step S31, performing smoothing processing on the plurality of motion state information and the plurality of two-dimensional spatial position coordinates to generate a plurality of smoothed motion state information and a plurality of smoothed two-dimensional spatial position coordinates;

[0137] Step S32, constructing a node set based on the plurality of smoothed motion state information and the plurality of smoothed two-dimensional spatial position coordinates;

[0138] Step S33, constructing an edge set by using the preset spatial edge rule according to a plurality of nodes in the node set;

[0139] Step S34, determining the time-dimension edge sequence based on the edge set.

[0140] It should be noted that, for the noise (for example, artificial labeling error) in the position or velocity data, smoothing processing needs to be performed, such as using a low-pass filter or a moving average method. After completing the smoothing processing, at each time step , the scene is represented as a spatial graph . For the node set: , each node Represents intelligent agents exist Location and motion status information at any given time. Each node is classified as either a vehicle or a pedestrian based on its type. For the edge set: , represents a directed edge in a spatial graph. Each edge Representative at Time Node For nodes The impact.

[0141] Furthermore, to capture asymmetric interactions between agents and enhance the generalization ability of spatial graphs, the edge definition rules are as follows, i.e., the predefined spatial edge rules are: the condition for the existence of a directed edge: when a node... and The two-dimensional spatial position distance is less than When considering the perceptual scope of a semantic type, there exists a directed edge. The perception range is acquired through experience and adjusted according to the actual scenario. Edge weights: If there are directed edges between nodes, then... ;otherwise .

[0142] Furthermore, construct the set of adjacent edges: for each node The set of edges between a node and its neighboring nodes is defined as follows: Adjacent edges with the same semantic relationship (such as pedestrian → pedestrian, pedestrian → vehicle) are combined into a shared weight vector and processed uniformly.

[0143] Based on the above, a time-dimensional edge sequence is generated. Specifically, all processed spatial edges constitute a time sequence in the time dimension, defined as: Time series All nodes representing historical states will be used as input and fed into the encoder for feature encoding.

[0144] Step 104: Perform embedding processing on the agent's historical motion pattern sequence, agent's future motion planning sequence, and time dimension edge sequence respectively to generate historical motion time step embeddings corresponding to the agent's historical motion pattern sequence, future motion time step embeddings corresponding to the agent's future motion planning sequence, and edge time step embeddings corresponding to the time dimension edge sequence.

[0145] It should be noted that the high-resolution map is used as a static context. The time series of dynamic motion patterns and interactions in the scene are vectorized through time-step embedding and used as input to the encoder. A combination of marker embedding and location embedding is employed for time-step embedding.

[0146] Specifically, step 104 can include the following sub-steps S41-S46:

[0147] Step S41, respectively, one-dimensional convolution is performed on the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence, and the historical motion convolution feature corresponding to the agent historical motion pattern sequence, the future motion convolution feature corresponding to the agent future motion planning sequence and the edge convolution feature corresponding to the time dimension edge sequence are outputted;

[0148] Step S42, non-linear mapping is performed on the historical motion convolution feature, the future motion convolution feature and the edge convolution feature respectively, and the historical motion label embedding corresponding to the historical motion convolution feature, the future motion label embedding corresponding to the future motion convolution feature and the edge label embedding corresponding to the edge convolution feature are generated;

[0149] Step S43, position embedding processing is performed on the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence respectively, and the historical motion position embedding corresponding to the agent historical motion pattern sequence, the future motion position embedding corresponding to the agent future motion planning sequence and the edge position embedding corresponding to the time dimension edge sequence are outputted;

[0150] Step S44, weighted summation is performed on the historical motion label embedding and the historical motion position embedding, and the historical motion time step embedding is determined;

[0151] Step S45, weighted summation is performed on the future motion label embedding and the future motion position embedding, and the future motion time step embedding is determined;

[0152] Step S46, weighted summation is performed on the edge label embedding and the edge position embedding, and the edge time step embedding is determined.

[0153] It should be noted that the label embedding is a process of mapping the time series of multiple input sources to a high-dimensional continuous vector space, which is beneficial to capture the semantic similarity between labels. The one-dimensional convolution filter (kernel width 3, step 1) and the non-linear activation function Leaky ReLU are used for label embedding. The label embedding is represented as a fixed dimension size Specifically, the time series of multiple input sources are the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence, one-dimensional convolution is performed on the three inputs respectively using the one-dimensional convolution filter, the convolution features corresponding to the three inputs are obtained, and then the non-linear activation function Leaky ReLU is used to perform non-linear mapping on the convolution features corresponding to the three inputs, and the label embeddings corresponding to the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence are obtained.

[0154] ​Further, to preserve the timing information in the motion pattern, a position embedding based on sine and cosine functions is introduced to generate the time step embedding, i.e., the position embedding based on sine and cosine functions performs position embedding processing on the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence, to obtain the position embedding corresponding to the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence. A position code is added to each token to represent the relative position of the token in the motion pattern sequence, thereby helping the model to distinguish the motion state at different time points or different steps. The calculation method of the position embedding is:

[0155] ;

[0156] ;

[0157] wherein, , is the d-th feature of the C-dimensional vector space, is the position embedding vector (position embedding). Finally, the result of the time step embedding is represented as (i.e., the weighted sum of the position embedding and the token embedding):

[0158] ;

[0159] wherein, is a scaling factor for keeping the numerical magnitude of the token embedding and the position embedding consistent. If the input sequence has been normalized, it is recommended to set .

[0160] Step 105, the pre-set multi-scale historical aggregation encoder is used to encode the historical motion time step embedding, the future motion time step embedding and the edge time step embedding respectively, to generate the multi-scale historical motion feature corresponding to the historical motion time step embedding, the multi-scale future motion feature corresponding to the future motion time step embedding and the multi-scale edge feature corresponding to the edge time step embedding.

[0161] It should be noted that the present application designs a multi-scale historical aggregation encoder (MHAE), i.e., the pre-set multi-scale historical aggregation encoder encodes the features of the time step embedding. The time series of several data sources of the time step embedding are input into the multi-scale historical aggregation encoder for feature encoding. The MHAE can generate a three-dimensional feature tensor under multiple time scales, and finally output a representation that aggregates features of different time scales.

[0162] Further, please refer to Figure 2, the preset multi-scale history aggregation encoder comprises two self-attention submodules, a first multi-scale branch module, a second multi-scale branch module and a third multi-scale branch module, wherein the size of the input data of the first self-attention submodule is , the size of the input data of the first self-attention submodule is ; the first multi-scale branch module is composed of four self-attention submodules and three self-attention distillation down-sampling submodules, and the size of the input data thereof is ; the second multi-scale branch module is composed of three self-attention submodules and two self-attention distillation down-sampling submodules, and the size of the input data thereof is ; the third multi-scale branch module is composed of two self-attention submodules and one self-attention distillation down-sampling submodule, and the size of the input data thereof is .

[0163] Optionally, the data processing process of the preset multi-scale history aggregation encoder can comprise the following sub-steps S51-S58:

[0164] Step S51, slicing the input time step embedding input to the preset multi-scale history aggregation encoder to generate a first sub-feature, a second sub-feature and a third sub-feature;

[0165] Step S52, taking the input time step embedding as the input of the self-attention submodule to output a first self-attention feature;

[0166] Specifically, step S52 can comprise the following sub-steps S521-S526:

[0167] Step S521, based on the multi-head self-attention mechanism, attention calculation is performed on the input time step embedding to determine the multi-head self-attention corresponding to the input time step embedding;

[0168] Step S522, element-wise addition is performed on the multi-head self-attention and the input time step embedding to output a multi-head self-attention time step feature;

[0169] Step S523, layer normalization is performed on the multi-head self-attention time step feature to generate a multi-head self-attention normalized feature;

[0170] Step S524, one-dimensional convolution is performed on the multi-head self-attention normalized feature to output a multi-head self-attention one-dimensional convolution feature;

[0171] Step S525, nonlinear mapping is performed on the multi-head self-attention one-dimensional convolution feature to generate a multi-head self-attention nonlinear feature;

[0172] Step S526, one-dimensional convolution is performed on the multi-head self-attention nonlinear feature to output the first self-attention feature.

[0173] It should be noted that a self-attention submodule is constructed in the feature branch at each scale. MHAE feature extraction relies on a multi-head self-attention mechanism, where the self-attention formula is:

[0174] ;

[0175] The formula for multi-head self-attention is:

[0176] ;

[0177] ;

[0178] Where Q, K, and V are the query, key, and value matrices, , , and These are the weight matrices corresponding to the i-th head. yes and Dimensions This is the scaling factor. The scaling factor is 5, and the number of attention heads is... It's 8.

[0179] Furthermore, a self-attention block is constructed as a unit module for MHAE feature extraction, and its output is denoted as... The data processing principle of the self-attention submodule can be expressed as follows:

[0180] ;

[0181] Among them, symbols Represents input data, It is an element-by-element addition. It is an activation function. and Both use one-dimensional convolutional filters with a kernel size of 1, but The number of output channels is four times the number of input channels, and vice versa. The number of output channels is 1 / 4 of the number of input channels; LayerNorm is for layer normalization.

[0182] It is worth mentioning that the data processing principle of the self-attention submodule in the first multi-scale branch module, the second multi-scale branch module, and the third multi-scale branch module is the same as that described here, and will not be elaborated further in this invention.

[0183] Step S53: Use the first multi-scale branch module to extract multi-scale features from the input time step embedding to generate the first multi-scale features;

[0184] Step S54, performing multi-scale feature extraction on the first sub-feature through a second multi-scale branch module to generate a second multi-scale feature;

[0185] Step S55, inputting the second sub-feature into a third multi-scale branch module for multi-scale feature extraction to generate a third multi-scale feature;

[0186] The third multi-scale branch module includes a self-attention sub-module and a self-attention distillation down-sampling sub-module.

[0187] Specifically, step S55 can include the following sub-steps S551-S553:

[0188] Step S551, taking the second sub-feature as the input of the self-attention sub-module, and outputting the self-attention feature corresponding to the second sub-feature;

[0189] Step S552, performing distillation down-sampling on the self-attention feature corresponding to the second sub-feature by using the self-attention distillation down-sampling sub-module to generate a second sub-feature maximum pooling feature;

[0190] Specifically, step S552 can include the following sub-steps S5521-S5524:

[0191] Step S5521, performing one-dimensional convolution on the self-attention feature corresponding to the second sub-feature to output a second sub-feature convolution feature;

[0192] Step S5522, performing batch normalization on the second sub-feature convolution feature to generate a second sub-feature normalization feature;

[0193] Step S5523, performing down-sampling on the second sub-feature normalization feature to output a second sub-feature down-sampling feature;

[0194] Step S5524, performing maximum pooling on the second sub-feature down-sampling feature to generate a second sub-feature maximum pooling feature.

[0195] Step S553, taking the second sub-feature maximum pooling feature as the input of the self-attention sub-module to output the third multi-scale feature.

[0196] It should be noted that information distillation and time dimension down-sampling are performed after each self-attention block. On the one hand, distillation can reduce the parameter quantity and calculation quantity in the model by reducing the feature dimension, thereby reducing the calculation cost of the model. On the other hand, information distillation can provide a regularization mechanism. It improves the generalization ability of the model by introducing a lower-dimensional feature representation, thereby preventing the model from overfitting the training data. Assuming that the feature output by any layer of self-attention distillation (except the last layer) is , then from the th layer to the The data processing principle of the self-attention distillation down-sampling sub-module can be expressed as a forward function of a distillation operation, that is,

[0197] ;

[0198] wherein, is the input of the self-attention distillation down-sampling sub-module; is the output of the self-attention distillation down-sampling sub-module; is a one-dimensional convolution filter with a kernel size of 3 and the same number of input and output channels, has a kernel size of 3 and a step size of 2, and BN is batch normalization of the sample. Through the (max pooling) operation, each distillation layer can reduce the size of the time dimension of the feature tensor by half, thereby reducing the redundant information contained in the sequence; ELU is down-sampling.

[0199] It is worth mentioning that the data processing principle of the self-attention distillation down-sampling sub-module in the first multi-scale branch module, the second multi-scale branch module, and the third multi-scale branch module is consistent with the data processing principle of the self-attention distillation down-sampling sub-module described herein, and the present application will not be described in more detail.

[0200] Step S56, taking the third sub-feature as the input of the self-attention sub-module, and outputting a second self-attention feature;

[0201] Step S57, performing layer normalization on the first self-attention feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the second self-attention feature, respectively, and outputting a first self-attention normalized feature, a first multi-scale normalized feature, a second multi-scale normalized feature, a third multi-scale normalized feature, and a second self-attention normalized feature;

[0202] Step S58, concatenating the first self-attention normalized feature, the first multi-scale normalized feature, the second multi-scale normalized feature, the third multi-scale normalized feature, and the second self-attention normalized feature to generate an output multi-scale feature.

[0203] The input time step embedding is feature data input to the preset multi-scale historical aggregation encoder, and it can be understood that the input time step embedding can correspond to any feature data input to the preset multi-scale historical aggregation encoder for data processing in the process of model training or model application;

[0204] The first sub-feature, the second sub-feature, and the third sub-feature are intermediate feature data generated in the preset multi-scale historical aggregation encoder;

[0205] The output multi-scale feature is the feature data output by the preset multi-scale history aggregation encoder. It can be understood that it can correspond to any output feature data after the preset multi-scale history aggregation encoder processes the data in the model training or model application process.

[0206] It should be noted that the splicing and aggregation of the feature of each scale branch are completed by stacking different numbers of self-attention modules and self-attention distillation layers. Different stacking layers have different numbers of self-attention distillation layers, indicating that the time dimension distillation multiples of different stacking layers in the multi-scale branch are different. The layer with a lower distillation ratio is used for a smaller time dimension slice, and the layer with a higher distillation ratio is used for a larger time dimension slice. Specifically, the ratio of information distillation matches the ratio of the time dimension of the input slice, and the output dimension of the multi-scale branch after the stacking layer is . Finally, the feature map of the multi-scale branch and the feature map of the constraint branch are spliced in the time dimension to obtain the expression of the multi-scale history feature map, that is, the processing process of the multi-scale feature can be represented as:

[0207] ;

[0208] wherein, represents a splicing operation along the time dimension, , represent the feature maps of the input sequence complete copy, 1 / 2 slice, 1 / 4 slice and 1 / 8 slice, respectively, that is, the first multi-scale normalized feature, the second multi-scale normalized feature, the third multi-scale normalized feature and the second self-attention normalized feature. represents the feature map of the original input sequence (that is, the first self-attention normalized feature). The multi-scale feature (multi-scale feature) is used as a length of simulated motion pattern feature, which is added before the earliest observation time of the original input feature, and is spliced by slice features of different history lengths. Inside these slice sequences, the motion pattern data conforms to the change rule of the real motion pattern over time. This The addition of the multi-scale motion pattern feature of the length makes the MHAE provide and cover important feature information of the motion pattern data at different time scales.

[0209] Further, the processing procedure that the historical motion time step embedding, the future motion time step embedding and the edge time step embedding are input to the preset multi-scale historical aggregation encoder for encoding can refer to the processing procedure of the preset multi-scale historical aggregation encoder for processing the input time step embedding, so that the multi-scale historical motion feature corresponding to the historical motion time step embedding, the multi-scale future motion feature corresponding to the future motion time step embedding and the multi-scale edge feature corresponding to the edge time step embedding are obtained, and the present application will not be described in more details.

[0210] It is worth mentioning that the present application takes the original input tensor (input time step embedding) as a separate constraint branch of the MHAE, and constructs four copies of the input to extract features at different time scales. In the branch used as a constraint term, only one self-attention block is applied to encode the original input feature without changing its time dimension. In the multi-scale branch, the four copies of the original input are processed separately. One keeps the full time dimension, and the other three extract subsequences of 1 / 2, 1 / 4 and 1 / 8 time dimensions closest to the current time step by slicing the input feature respectively. In the Figure 2 the sizes of the three slices in the middle are , , where denotes the batch size. The multi-time scale feature aggregation is realized by encoding these historical slices of different lengths.

[0211] Further, motion pattern data of different scales may have different time variation patterns, such as periodic variation or rapid variation. By extracting features at different scales, the MHAE can simultaneously focus on the global and local structures of the motion pattern data, and better adapt to data with different time variation patterns. Features with larger time dimensions can capture long-term dependencies and overall trends of the motion pattern sequence, while features with smaller time dimensions can capture local details and short-term changes. In addition, the collected data may be affected by noise interference. Features with smaller scales can help the model to be more sensitive to local changes in a short time, so that the model has better adaptability and robustness to noise.

[0212] Step 106, outputting an initial prediction result according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion pattern sequence through a plurality of preset efficient autoregressive decoders.

[0213] It should be noted that please refer to Figure 3, the present application designs an efficient autoregressive decoder (EAD) for decoding. The EAD is an efficient decoder, and its contributions mainly include two parts. On the one hand, the real-time calculation time of the model is reduced by using a probabilistic sparse self-attention mechanism; on the other hand, the prediction accuracy is improved by combining the autoregressive decoding of historical labels, which is different from the traditional autoregressive decoder. The specific structure of EAD is shown in the following figure: Figure 3 The autoregressive decoding part on the right side extracts part of the output generated by it as the input of the multi-layer perception, and combines the preset binary Gaussian mixture model trained based on the reparameterization to generate the target agent motion pattern prediction result.

[0214] It is worth mentioning that the present application decodes the multi-scale historical motion features, multi-scale future motion features, multi-scale edge features, map feature vectors and agent historical motion pattern sequences through seven preset efficient autoregressive decoders to output the initial prediction result; wherein the input of the first preset efficient autoregressive decoder is the multi-scale historical motion features, multi-scale future motion features, multi-scale edge features, map feature vectors and agent historical motion pattern sequence, and the input of the remaining preset efficient autoregressive decoder is the multi-scale historical motion features, multi-scale future motion features, multi-scale edge features, map feature vectors and new agent historical motion pattern sequence, and the output prediction result generated by the previous preset efficient autoregressive decoder; the output prediction result generated by the last preset efficient autoregressive decoder is the initial prediction result.

[0215] Optionally, the data processing process of the preset efficient autoregressive decoder can include the following substep S61:

[0216] Step S61, the input historical motion pattern sequence input into the preset efficient autoregressive decoder is mapped to a high-dimensional feature of historical motion by using a multi-layer perception;

[0217] Step S62, the historical motion high-dimensional feature and the input edge feature, input map feature and input future motion feature input into the preset efficient autoregressive decoder are spliced to generate a spliced feature;

[0218] Step S63, a global adaptive one-dimensional average pooling operation is performed on the spliced feature to generate a global feature representation;

[0219] Step S64, the global feature representation is time-activated to output a time-activated feature;

[0220] It should be noted that please refer to Figure 4 , a time compression and excitation (TSE module) mechanism is designed for feature fusion and time dimension expansion. The decoder further combines social interaction features (i.e. multi-scale edge features) , grid map features (i.e. map feature vectors) Vehicle motion planning characteristics (i.e., multi-scale future motion characteristics) By concatenating features along the time dimension and optimizing feature representation using TSE (Time Sequencing), a high-dimensional nonlinear prediction result is formed. The input to the TSE module consists of multiple input sources at any given time. The feature splicing is denoted as Time compression and time excitation are two key steps in the TSE module. They work together to change the representation and extraction of time series features by learning and adjusting the channel weights. A schematic diagram of the TSE module is shown below. Figure 4 As shown.

[0221] Furthermore, regarding time compression, in the time compression step, the TSE module performs a global adaptive one-dimensional average pooling (GAP) operation on the concatenated multimodal sequence features. The purpose is to compress the temporal dimension of the features while retaining important information. The global adaptive one-dimensional average pooling operation can be expressed as:

[0222] ;

[0223] in, This represents the global features. This is a globally adaptive one-dimensional average pooling.

[0224] Furthermore, for each time channel, the TSE module calculates the average of all modes in that channel. In this way, the TSE module helps the model better understand the overall time series characteristics and reduces its dependence on any particular mode, thereby enhancing the model's robustness.

[0225] Furthermore, for temporal stimulus, the goal is to learn weights for each channel and adjust the input features according to their importance. The calculation process (i.e., the temporal stimulus processing) is as follows:

[0226] ;

[0227] in, This represents the global feature representation obtained through the time compression step. It is a multilayer perceptron that maps pooled global features to a very low dimension (1 / 16 of the original dimension) to focus on important features. The introduction of the ReLU function enhances the non-linearity of the module, enabling the TSE module to learn more complex weight distributions. Multilayer perceptron The features are mapped back to the original high dimension for more fine-grained adjustment and weighting. The activation output is normalized to the range of [0, 1] by a Sigmoid function, thus obtaining the weight of each channel. These weights represent the importance of each channel, where the weight value close to 1 indicates that the channel has a greater contribution to the feature representation, and the weight value close to 0 indicates a smaller contribution.

[0228] Step S65, element-wise multiplication is performed on the time incentive features and the spliced features to generate time series feature representation;

[0229] Based on the above basis, the TSE module reweights the original input features using these channel weights. For each time channel, the original features are multiplied by the corresponding channel weight to obtain the reweighted features. In this way, the features from important time channels are amplified, while the features from unimportant time channels are suppressed. The overall forward function expression of the TSE module, i.e. the processing process of the time series feature representation, can be expressed as:

[0230]

[0231] wherein, represents an element-wise multiplication operation. This step completes the reweighting of the features and generates the final time series feature representation .

[0232] Step S66, the time series feature representation is mapped by a multi-layer perception to output a time series high-dimensional feature;

[0233] Step S67, layer normalization and nonlinear mapping are performed on the time series high-dimensional feature to output a time series nonlinear feature;

[0234] Step S68, element-wise addition is performed on the time series nonlinear feature and the historical time point prediction result to generate a current time point prediction result;

[0235] Step S69, the current time point prediction result and the target subsequence in the input historical motion pattern sequence are spliced to generate a new input historical motion pattern sequence;

[0236] The target subsequence in the input historical motion pattern sequence is the feature data corresponding to the historical motion state pattern label (-H to -1).

[0237] ​It should be noted that after the time step embedding operation (TE) is performed on the input of the decoder, the multi-layer perception, layer normalization and LeakyReLU activation function are sequentially processed, and then the current time prediction result is obtained by element-wise addition with the historical time prediction result (initialized as the input of the first preset efficient autoregressive decoder agent historical motion pattern sequence). The process can be represented as:

[0238] wherein, is the current time prediction result; is the historical time prediction result; is the input historical motion pattern sequence.

[0239] Further, the newly generated prediction result is spliced into the historical input to form a new input sequence, that is, the current time prediction result is spliced with the target sub-sequence in the input historical motion pattern sequence to generate a new input historical motion pattern sequence , and the process can be represented as:

[0240]

[0241] It is worth mentioning that the newly generated input historical motion pattern sequence is used as the input of the next preset efficient autoregressive decoder.

[0242] Step S610, time step embedding and position encoding are performed on the new input historical motion pattern sequence, and an encoded feature sequence is output;

[0243] It should be noted that the processing principle of time step embedding can refer to the embedding process of the above-mentioned agent historical motion pattern sequence, agent future motion planning sequence and time dimension edge sequence. The present application will not be described in more detail.

[0244] Further, the position encoding is to add a preset position encoding matrix to the new input historical motion pattern sequence after time step embedding processing.

[0245] Step S611, based on the single-layer mask multi-head probability sparse self-attention mechanism, attention operation is performed on the encoded feature sequence, and probability sparse self-attention is output;

[0246] Step S612, the probability sparse self-attention and the encoded feature sequence are element-wise added, and a probability sparse self-attention feature sequence is output;

[0247] Step S613, layer normalization is performed on the probability sparse self-attention feature sequence, and a query matrix is determined;

[0248] Step S614, linearly projecting the input history motion feature input to the preset efficient autoregressive decoder to generate a key matrix and a value matrix;

[0249] Step S615, generating an efficient decoding feature sequence according to the query matrix, the key matrix and the value matrix;

[0250] Step S616, layer normalizing the efficient decoding feature sequence to output an efficient decoding normalized feature sequence;

[0251] Step S617, using a multi-layer perception to perform feature mapping on the efficient decoding normalized feature sequence to generate an output prediction result.

[0252] The input history motion pattern sequence, the input edge feature, the input map feature, the input future motion feature and the input history motion feature are feature data input to the preset efficient autoregressive decoder, and it can be understood that the input history motion pattern sequence, the input edge feature, the input map feature, the input future motion feature and the input history motion feature can correspond to any feature data input to the preset efficient autoregressive decoder for data processing in the process of model training or model application;

[0253] The history motion high-dimensional feature and the time excitation feature are intermediate feature data generated in the preset efficient autoregressive decoder;

[0254] The output prediction result is feature data output by the preset efficient autoregressive decoder, and it can be understood that it can correspond to any output feature data after the preset efficient autoregressive decoder performs data processing in the process of model training or model application.

[0255] It should be noted that the present application constructs an efficient decoding module (EDB) to optimize real-time computing performance. Real-time performance is the key to the successful deployment of motion pattern prediction models in autonomous driving systems. Unlike the encoding process, the decoder usually only needs to focus on the relevant part of the generated sequence, without the need to capture the semantic and spatial information of the complete input sequence. The traditional self-attention model calculates the weight of all elements, which brings high computational cost and low efficiency. The probability sparse self-attention mechanism solves this problem by limiting the attention mechanism to only act on key query interactions that meet the sparsity criterion. The probability sparse self-attention mechanism is represented as follows:

[0256] ;

[0257] wherein, is a sparse matrix associated only with the main query that meets the sparsity criterion. The sparsity criterion is defined as:

[0258] ;

[0259] where, is the query q i is the maximum mean, ; is the square root of the dimension of the input data feature; is the transpose of the jth key; is the ith query vector; L is the number of keys, usually denoted as L K or L Q ; is a part of the attention weight, representing the similarity between a given query and a particular key.

[0260] Further, by focusing on the most critical data elements, the probabilistic sparse self-attention reduces unnecessary calculations, thereby accelerating processing time. Based on single-layer masked multi-head probabilistic sparse self-attention (MMPSA), the present invention constructs an efficient decoding module (EDB), whose forward function is used to calculate the query matrix and apply the mask operation in the time dimension to improve computational efficiency, and its structure is as follows (i.e. the processing process of the query matrix):

[0261] ;

[0262] Further, the processing process of the efficient decoding feature sequence can be represented as:

[0263] where, is the query matrix; is the input data; K, V are the input historical motion features corresponding key matrix and value matrix.

[0264] It is worth mentioning that the present invention introduces a history label to improve the autoregressive decoding structure. Another core feature of EAD is the uniqueness of its autoregressive decoding structure. Unlike conventional methods, the input data of EAD includes all agent position data from the observation starting time to the current time , defined as:

[0265] ;

[0266] where, for , the data is the real motion pattern label; and for , the data is the prediction result generated by the model at the previous time. By introducing the joint input of real labels and historical prediction results, this decoder can achieve stronger model robustness.

[0267] Further, the output prediction results generated by the first six preset high-efficiency autoregressive decoders are used as inputs instead of the input historical motion pattern sequence, and after being processed by the multi-layer perception, the input edge features, input map features, and input future motion features are spliced to perform subsequent processing, and the output prediction result generated by the last preset high-efficiency autoregressive decoder is used as the initial prediction result.

[0268] In step 107, a preset binary Gaussian mixture model is used to generate a target agent motion pattern prediction result according to the initial prediction result.

[0269] It should be noted that the prediction part is extracted from the initial prediction result The prediction vectors are mapped into intermediate feature vectors by the multi-layer perception (MLP), and then input into the preset binary Gaussian mixture model to output a plurality of possible motion pattern distributions. The target agent motion pattern prediction result is generated by maximum probability sampling or random sampling on the plurality of possible motion pattern distributions.

[0270] As a comparison of technical effects, the motion pattern recognition technology is widely used in intelligent monitoring, robot navigation, automatic driving, and other fields. The open scene dynamic warning accurately identifies the motion pattern of the agent and predicts the future motion trend through real-time data collection and intelligent analysis, provides dynamic collision risk warning, and supports path planning. The traditional method has limited prediction ability in complex scenes, and the neural network method has made some progress, but there are still problems such as insufficient time domain feature mining, high decoding complexity, and insufficient feature fusion. Therefore, the present application proposes an open scene-oriented agent motion pattern prediction method, which aims to solve the problems of insufficient time domain feature exploration, decoder calculation complexity, and multi-modal feature fusion in the prior art. By introducing a multi-scale history aggregation encoder and a high-efficiency autoregressive decoder, combined with a binary Gaussian mixture model, the prediction accuracy is improved, and the motion pattern of multiple agents (such as crowds, motor vehicles, etc.) can be more accurately predicted, providing more reliable dynamic warning and path planning support for intelligent monitoring, robot navigation, and automatic driving, etc.

[0271] In the embodiment of the present application, the present application provides an open scene-oriented agent motion mode prediction method. First, the agent historical motion mode sequence, the agent future motion planning sequence, the monitoring video dataset and the city road map are obtained, and the city road map is encoded to generate a map feature vector. Then, according to the original annotation file of the monitoring video dataset, a plurality of two-dimensional spatial position coordinates are generated, and the plurality of two-dimensional spatial position coordinates are subjected to homogeneous coordinate transformation and normalization to generate a plurality of motion state information. A preset spatial edge rule is used to generate a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates. The agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence are embedded respectively to generate a historical motion time step embedding corresponding to the agent historical motion mode sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence. A preset multi-scale historical aggregation encoder is used to encode the historical motion time step embedding, the future motion time step embedding and the edge time step embedding respectively to generate a multi-scale historical motion feature corresponding to the historical motion time step embedding, a multi-scale future motion feature corresponding to the future motion time step embedding and a multi-scale edge feature corresponding to the edge time step embedding. An initial prediction result is outputted by a plurality of preset efficient autoregressive decoders according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion mode sequence. Finally, a target agent motion mode prediction result is generated by a preset bivariate Gaussian mixture model according to the initial prediction result. Based on the above scheme, the multi-modal features, i.e. the agent historical motion mode sequence, the agent future motion planning sequence, the monitoring video dataset and the city road map obtained, are processed by the preset multi-scale historical aggregation encoder and the plurality of preset efficient autoregressive decoders to output the initial prediction result, and then the target agent motion mode prediction result is generated by combining the preset bivariate Gaussian mixture model according to the initial prediction result. The present application can optimize the feature fusion in the time dimension, thereby improving the effect of multi-modal feature fusion.

[0272] For better illustration, refer to Figure 5 , which shows Figure 5 The training flowchart of the preset bivariate Gaussian mixture model provided in the second embodiment of the present application, which can include the following steps:

[0273] 1. Obtain a dataset for model training; 2. Calculate the kinematic properties required for modeling according to the original annotation of the dataset; 3. Physical modeling of motion patterns; 4. Spatial graph representation, temporal dimension sequence representation and feature preprocessing of agent social interaction; 4. Timestamp embedding of multi-source time series, city map encoding; 5. Multi-scale historical aggregation encoder (MHAE) for feature encoding of timestamp embedding; 6. Efficient autoregressive decoder (EAD) for decoding; 7. Time compression and excitation (TSE) mechanism for feature fusion and time dimension expansion; 7. Model training.

[0274] It should be noted that for the training of the preset binary Gaussian mixture model, the initial output of the last step decoding in the autoregressive decoder is ; then, the prediction part is extracted from the initial output as the final output of the decoder: , wherein represents the predicted spatial coordinate position of the agent at the time step, i.e. the position of the agent at each predicted time step can be represented as a two-dimensional position vector, is a two-dimensional position vector of agent i at any time step t within the time range of t = -H to t = T.

[0275] Further, these prediction vectors are mapped to intermediate feature vectors through multi-layer perceptron (MLP). Then four relatively simple predefined sub-modules are designed to convert the intermediate features into Gaussian mixture model (GMM) parameters. Each sub-module only contains linear layers for mapping the hidden dimensions of the intermediate features to the GMM parameter space with different dimension sizes.

[0276] Specifically, the outputs of the four sub-modules include four types of parameters of the binary Gaussian mixture model : mean , log standard deviation , correlation coefficient , and log value of mixing coefficient . For components, there are the following constraints: . Wherein , and are initialized by MLP, and is initialized by Tanh activation function and MLP. These parameters can further calculate the covariance matrix and probability density function of the GMM.

[0277] represents the covariance matrix of the position at the time step , and the calculation formula is:

[0278] ;

[0279] where, is the probability density function of the bivariate Gaussian distribution, which is calculated using the following equation:

[0280] where, is the log standard deviation of the lateral direction; is the log standard deviation of the longitudinal direction.

[0281] Further, since sampling directly from the GMM would cause the gradients to not propagate, the present invention uses a reparameterization technique. First, a standardized two-dimensional random variable is sampled from a standard normal distribution . By an affine transformation, the standard normal distribution sample is converted to a sample of the target distribution. For a bivariate Gaussian distribution, the affine transformation is as follows:

[0282] ;

[0283] To minimize the gap between the predicted output and the true motion pattern, the present invention supervises the final output of the model by using a regression loss:

[0284] ;

[0285] where, represents the total number of agents in a batch of samples, and represent the predicted motion pattern of any agent in the training sample at time step and the ground truth.

[0286] In the validation phase of training, the present invention supervises the learning of all trainable parameters of the model by minimizing the negative log-likelihood loss:

[0287] ;

[0288] Based on the above basis, the present invention can obtain a motion pattern prediction model with better prediction effect and smaller bias, i.e., a preset bivariate Gaussian mixture model.

[0289] Exemplarily, please refer to Figure 6, real-time acquisition of input data required by the model according to the use scene; model inference and future motion pattern generation are performed by using the trained preset binary Gaussian mixture model. Based on the GMM parameters of the trained preset binary Gaussian mixture model, possible future motion patterns are generated by sampling from the GMM. Two optional sampling methods include: 1) selecting the maximum probability mode of the GMM to generate the most likely motion pattern; 2) randomly sampling several motion patterns from the GMM to capture the diversity and uncertainty of the future motion pattern. By switching the sampling strategy, the most likely future motion pattern and the motion pattern with greater randomness can be simulated.

[0290] Further, the predicted motion pattern generated by the model is based on the world coordinate system, and if there is a practical need for visualization, it needs to be converted to other coordinate systems (such as pixel coordinate system). The homogeneous coordinate conversion formula from the world coordinate system to the pixel coordinate system is: ;

[0291] wherein, represents the homogeneous form of the world coordinate; is the transformation matrix from the world coordinate to the pixel coordinate. After conversion, the coordinates need to be normalized, that is: , is the scaling factor in the homogeneous coordinate.

[0292] In the embodiment of the present application, since direct sampling from the GMM will cause the gradient to be unable to be transmitted, the reparameterization technique is adopted. The present application first randomly samples a two-dimensional random variable from the standard normal distribution, and converts it into a sample of the target distribution through affine transformation to generate training samples of the Gaussian distribution. The samples are used to calculate the regression loss of the model to minimize the difference between the predicted motion pattern and the real motion pattern. In the verification stage of the training, the learning of all trainable parameters of the model is supervised by minimizing the negative log-likelihood loss.

[0293] Please refer to Figure 7 , Figure 7 for the structure block diagram of the agent motion pattern prediction device for an open scene provided in Embodiment Three of the present application.

[0294] The agent motion pattern prediction device for an open scene provided by the present application comprises:

[0295] The acquisition module 701 is configured to acquire the historical motion pattern sequence of the agent, the future motion planning sequence of the agent, the monitoring video dataset and the city road map, and perform map encoding on the city road map to generate a map feature vector.

[0296] According to the module 702, a plurality of two-dimensional spatial position coordinates are generated according to the original annotation file of the monitoring video data set, and the plurality of two-dimensional spatial position coordinates are subjected to homogeneous coordinate transformation and normalization to generate a plurality of motion state information;

[0297] The module 703 is adopted to generate a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates according to the preset spatial edge rule;

[0298] The embedding module 704 is adopted to respectively embed the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence to generate a historical motion time step embedding corresponding to the agent historical motion pattern sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence;

[0299] The encoding module 705 is adopted to encode the historical motion time step embedding, the future motion time step embedding and the edge time step embedding respectively by using a preset multi-scale historical aggregation encoder to generate a multi-scale historical motion feature corresponding to the historical motion time step embedding, a multi-scale future motion feature corresponding to the future motion time step embedding and a multi-scale edge feature corresponding to the edge time step embedding;

[0300] The decoding module 706 is adopted to output an initial prediction result according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion pattern sequence by using a plurality of preset efficient autoregressive decoders;

[0301] The output result module 707 is adopted to generate a target agent motion pattern prediction result according to the initial prediction result by using a preset binary Gaussian mixture model.

[0302] Further, the acquisition module 701 is specifically configured to:

[0303] The city road map is divided to output a plurality of map subblocks;

[0304] Each map subblock is transformed to generate a subblock vector corresponding to each map subblock;

[0305] Each subblock vector is subjected to linear transformation to output a linear embedding corresponding to each subblock vector;

[0306] A preset position encoding matrix is added to each linear embedding to generate an intermediate embedding corresponding to each linear embedding;

[0307] Based on a multi-head self-attention mechanism, attention weight calculation is performed on each map subblock to determine an attention weight corresponding to each map subblock;

[0308] add each attention weight to a corresponding intermediate embedding corresponding to the attention weight to generate a target embedding corresponding to the intermediate embedding;

[0309] perform layer normalization on each target embedding to determine a normalized embedding corresponding to the target embedding;

[0310] generate a map feature vector according to the plurality of normalized embeddings using a multi-layer perception.

[0311] Further, the module 703 is specifically configured to:

[0312] perform smoothing processing on the plurality of motion state information and the plurality of two-dimensional spatial position coordinates to generate a plurality of smoothed motion state information and a plurality of smoothed two-dimensional spatial position coordinates;

[0313] construct a node set based on the plurality of smoothed motion state information and the plurality of smoothed two-dimensional spatial position coordinates;

[0314] construct an edge set according to a plurality of nodes in the node set using a pre-set spatial edge rule;

[0315] determine a time-dimension edge sequence based on the edge set.

[0316] Further, the embedding module 704 is specifically configured to:

[0317] perform one-dimensional convolution on the agent historical motion pattern sequence, the agent future motion planning sequence, and the time-dimension edge sequence respectively to output a historical motion convolution feature corresponding to the agent historical motion pattern sequence, a future motion convolution feature corresponding to the agent future motion planning sequence, and an edge convolution feature corresponding to the time-dimension edge sequence;

[0318] perform nonlinear mapping on the historical motion convolution feature, the future motion convolution feature, and the edge convolution feature respectively to generate a historical motion token embedding corresponding to the historical motion convolution feature, a future motion token embedding corresponding to the future motion convolution feature, and an edge token embedding corresponding to the edge convolution feature;

[0319] perform position embedding processing on the agent historical motion pattern sequence, the agent future motion planning sequence, and the time-dimension edge sequence respectively to output a historical motion position embedding corresponding to the agent historical motion pattern sequence, a future motion position embedding corresponding to the agent future motion planning sequence, and an edge position embedding corresponding to the time-dimension edge sequence;

[0320] perform weighted summation on the historical motion token embedding and the historical motion position embedding to determine a historical motion time step embedding;

[0321] perform weighted summation on the future motion token embedding and the future motion position embedding to determine a future motion time step embedding;

[0322] The edge time step embedding is determined by weighted sum of the edge label embedding and the edge position embedding.

[0323] Further, the preset multi-scale history aggregation encoder comprises a self-attention submodule, a first multi-scale branch module, a second multi-scale branch module, and a third multi-scale branch module.

[0324] The input time step embedding input into the preset multi-scale history aggregation encoder is sliced to generate a first sub-feature, a second sub-feature, and a third sub-feature.

[0325] The input time step embedding is taken as the input of the self-attention submodule to output a first self-attention feature.

[0326] The first multi-scale branch module is used to perform multi-scale feature extraction on the input time step embedding to generate a first multi-scale feature.

[0327] The second multi-scale branch module is used to perform multi-scale feature extraction on the first sub-feature to generate a second multi-scale feature.

[0328] The second sub-feature is input into the third multi-scale branch module for multi-scale feature extraction to generate a third multi-scale feature.

[0329] The third sub-feature is taken as the input of the self-attention submodule to output a second self-attention feature.

[0330] The first self-attention feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the second self-attention feature are respectively subjected to layer normalization to output a first self-attention normalized feature, a first multi-scale normalized feature, a second multi-scale normalized feature, a third multi-scale normalized feature, and a second self-attention normalized feature.

[0331] The first self-attention normalized feature, the first multi-scale normalized feature, the second multi-scale normalized feature, the third multi-scale normalized feature, and the second self-attention normalized feature are spliced to generate an output multi-scale feature.

[0332] Further, the input time step embedding is taken as the input of the self-attention submodule to output a first self-attention feature, which comprises:

[0333] Based on a multi-head self-attention mechanism, attention calculation is performed on the input time step embedding to determine a multi-head self-attention corresponding to the input time step embedding.

[0334] The multi-head self-attention and the input time step embedding are subjected to element-wise addition to output a multi-head self-attention time step feature.

[0335] The multi-head self-attention time step features are layer normalized to generate multi-head self-attention normalized features;

[0336] The multi-head self-attention normalized features are one-dimensional convolution to output multi-head self-attention one-dimensional convolution features;

[0337] The multi-head self-attention one-dimensional convolution features are nonlinearly mapped to generate multi-head self-attention nonlinear features;

[0338] The multi-head self-attention nonlinear features are one-dimensional convolution to output first self-attention features.

[0339] Further, the third multi-scale branch module includes a self-attention sub-module and a self-attention distillation down-sampling sub-module; the second sub-feature is input into the third multi-scale branch module for multi-scale feature extraction to generate third multi-scale features, including:

[0340] The second sub-feature is taken as the input of the self-attention sub-module, and the self-attention features corresponding to the second sub-feature are output;

[0341] The self-attention distillation down-sampling sub-module is used to distill and down-sample the self-attention features corresponding to the second sub-feature to generate second sub-feature maximum pooling features;

[0342] The second sub-feature maximum pooling features are taken as the input of the self-attention sub-module, and the third multi-scale features are output.

[0343] Further, the self-attention distillation down-sampling sub-module is used to distill and down-sample the self-attention features corresponding to the second sub-feature to generate second sub-feature maximum pooling features, including:

[0344] The self-attention features corresponding to the second sub-feature are one-dimensional convolution to output second sub-feature convolution features;

[0345] The second sub-feature convolution features are batch normalized to generate second sub-feature normalized features;

[0346] The second sub-feature normalized features are down-sampled to output second sub-feature down-sampled features;

[0347] The second sub-feature down-sampled features are maximum pooled to generate second sub-feature maximum pooled features.

[0348] Further, the data processing process of the preset high-efficiency autoregressive decoder is as follows:

[0349] The multi-layer perception is used to map the input historical motion pattern sequence input into the preset high-efficiency autoregressive decoder to generate high-dimensional historical motion features;

[0350] The input edge feature, the input map feature, and the input future motion feature input to the preset efficient autoregressive decoder are spliced with the high-dimensional feature of the historical motion to generate a spliced feature;

[0351] A global adaptive one-dimensional average pooling operation is performed on the spliced feature to generate a global feature representation;

[0352] The global feature representation is time-activated to output a time-activated feature;

[0353] The time-activated feature and the spliced feature are element-wise multiplied to generate a time series feature representation;

[0354] The time series feature representation is mapped by a multi-layer perceptron to output a time series high-dimensional feature;

[0355] The time series high-dimensional feature is layer-normalized and nonlinearly mapped to output a time series nonlinear feature;

[0356] The time series nonlinear feature and the prediction result at the historical time are element-wise added to generate a prediction result at the current time;

[0357] The prediction result at the current time and the target sub-sequence in the input historical motion pattern sequence are spliced to generate a new input historical motion pattern sequence;

[0358] The new input historical motion pattern sequence is time-step embedded and position encoded to output an encoded feature sequence;

[0359] Based on a single-layer mask multi-head probability sparse self-attention mechanism, the encoded feature sequence is subjected to attention operation to output a probability sparse self-attention;

[0360] The probability sparse self-attention and the encoded feature sequence are element-wise added to output a probability sparse self-attention feature sequence;

[0361] The probability sparse self-attention feature sequence is layer-normalized to determine a query matrix;

[0362] The input historical motion feature input to the preset efficient autoregressive decoder is linearly projected to generate a key matrix and a value matrix;

[0363] According to the query matrix, the key matrix, and the value matrix, an efficient decoding feature sequence is generated;

[0364] The efficient decoding feature sequence is layer-normalized to output an efficient decoding normalized feature sequence;

[0365] The efficient decoding normalized feature sequence is mapped by a multi-layer perceptron to generate an output prediction result.

[0366] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0367] The embodiment of the present application further provides a computer device, including a memory and a processor, and the memory stores a computer program; the computer program is executed by the processor, so that the processor executes the steps of the open scene-oriented agent motion mode prediction method in the foregoing embodiment one.

[0368] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the open scene-oriented agent motion mode prediction method in the foregoing embodiment one.

[0369] The embodiment of the present application further provides a computer program product, including a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the open scene-oriented agent motion mode prediction method in the foregoing embodiment one.

[0370] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. For example, the above-described device embodiments are merely schematic, and the division of units is merely a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0371] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0372] The above-described embodiments are merely used to describe the technical solutions of the present application, rather than limiting them; although the foregoing embodiments of the present application have been described in detail, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An open-scene-oriented agent motion pattern prediction method, characterized in that, The method comprises the following steps: obtain the agent historical motion pattern sequence, the agent future motion planning sequence, the monitoring video dataset and the urban road map, and perform map encoding on the urban road map to generate a map feature vector; generate a plurality of two-dimensional spatial position coordinates according to the original annotation file of the monitoring video dataset, and perform homogeneous coordinate transformation and normalization on the plurality of two-dimensional spatial position coordinates to generate a plurality of motion state information; generate a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates by using a preset spatial edge rule; perform embedding processing on the agent historical motion pattern sequence, the agent future motion planning sequence and the time dimension edge sequence respectively to generate a historical motion time step embedding corresponding to the agent historical motion pattern sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence; encode the historical motion time step embedding, the future motion time step embedding and the edge time step embedding respectively by using a preset multi-scale historical aggregation encoder to generate a multi-scale historical motion feature corresponding to the historical motion time step embedding, a multi-scale future motion feature corresponding to the future motion time step embedding and a multi-scale edge feature corresponding to the edge time step embedding; output an initial prediction result according to the multi-scale historical motion feature, the multi-scale future motion feature, the multi-scale edge feature, the map feature vector and the agent historical motion pattern sequence by using a plurality of preset efficient autoregressive decoders; generate a target agent motion pattern prediction result according to the initial prediction result by using a preset binary Gaussian mixture model.

2. The open scenario oriented agent motion pattern prediction method according to claim 1, characterized in that, The method comprises the following steps: divide the urban road map to output a plurality of map subblocks; transform each of the map subblocks to generate a subblock vector corresponding to each of the map subblocks; perform linear transformation on each of the subblock vectors to output a linear embedding corresponding to each of the subblock vectors; add a preset position encoding matrix to each of the linear embeddings to generate an intermediate embedding corresponding to each of the linear embeddings; calculate attention weights of each of the map subblocks based on a multi-head self-attention mechanism to determine an attention weight corresponding to each of the map subblocks; add each of the attention weights to the intermediate embedding corresponding to the attention weight to generate a target embedding corresponding to each of the intermediate embeddings; perform layer normalization on each of the target embeddings to determine a normalized embedding corresponding to each of the target embeddings; generate a map feature vector according to a plurality of the normalized embeddings by using a multi-layer perceptron. 3.The open scenario oriented agent motion pattern prediction method according to claim 1, characterized in that, The method comprises the following steps: smooth the plurality of motion state information and the plurality of two-dimensional spatial position coordinates to generate a plurality of smoothed motion state information and a plurality of smoothed two-dimensional spatial position coordinates; Based on the plurality of smooth motion state information and the plurality of smooth two-dimensional spatial position coordinates, a node set is constructed; An edge set is constructed according to a plurality of nodes in the node set by adopting a preset spatial edge rule; Based on the edge set, a time dimension edge sequence is determined.

4. The open scenario oriented agent motion pattern prediction method according to claim 1, characterized in that, The embedding processing of the agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence respectively generates the historical motion time step embedding corresponding to the agent historical motion mode sequence, the future motion time step embedding corresponding to the agent future motion planning sequence and the edge time step embedding corresponding to the time dimension edge sequence, which includes: The one-dimensional convolution of the agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence respectively outputs the historical motion convolution feature corresponding to the agent historical motion mode sequence, the future motion convolution feature corresponding to the agent future motion planning sequence and the edge convolution feature corresponding to the time dimension edge sequence; The nonlinear mapping of the historical motion convolution feature, the future motion convolution feature and the edge convolution feature respectively generates the historical motion label embedding corresponding to the historical motion convolution feature, the future motion label embedding corresponding to the future motion convolution feature and the edge label embedding corresponding to the edge convolution feature; The position embedding processing of the agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence respectively outputs the historical motion position embedding corresponding to the agent historical motion mode sequence, the future motion position embedding corresponding to the agent future motion planning sequence and the edge position embedding corresponding to the time dimension edge sequence; The weighted sum of the historical motion label embedding and the historical motion position embedding determines the historical motion time step embedding; The weighted sum of the future motion label embedding and the future motion position embedding determines the future motion time step embedding; The weighted sum of the edge label embedding and the edge position embedding determines the edge time step embedding.

5. The open scenario oriented agent motion pattern prediction method according to claim 1, wherein, The preset multi-scale historical aggregation encoder includes a self-attention submodule, a first multi-scale branch module, a second multi-scale branch module and a third multi-scale branch module; the data processing process of the preset multi-scale historical aggregation encoder is specifically: The input time step embedding input into the preset multi-scale historical aggregation encoder is sliced to generate a first sub-feature, a second sub-feature and a third sub-feature; The input time step embedding is taken as the input of the self-attention submodule to output a first self-attention feature; The input time step embedding is subjected to multi-scale feature extraction by the first multi-scale branch module to generate a first multi-scale feature; The first sub-feature is subjected to multi-scale feature extraction by the second multi-scale branch module to generate a second multi-scale feature; The second sub-feature is input into the third multi-scale branch module for multi-scale feature extraction to generate a third multi-scale feature; The third sub-feature is taken as the input of the self-attention submodule to output a second self-attention feature; perform layer normalization on the first self-attention feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the second self-attention feature respectively, to output a first self-attention normalized feature, a first multi-scale normalized feature, a second multi-scale normalized feature, a third multi-scale normalized feature, and a second self-attention normalized feature; concatenate the first self-attention normalized feature, the first multi-scale normalized feature, the second multi-scale normalized feature, the third multi-scale normalized feature, and the second self-attention normalized feature to generate an output multi-scale feature.

6. The open scenario oriented agent motion pattern prediction method according to claim 5, characterized in that, The input time step embedding is input into the self-attention submodule to output a first self-attention feature, which includes: based on the multi-head self-attention mechanism, attention calculation is performed on the input time step embedding to determine the corresponding multi-head self-attention of the input time step embedding; perform element-wise addition on the multi-head self-attention and the input time step embedding to output a multi-head self-attention time step feature; perform layer normalization on the multi-head self-attention time step feature to generate a multi-head self-attention normalized feature; perform one-dimensional convolution on the multi-head self-attention normalized feature to output a multi-head self-attention one-dimensional convolution feature; perform nonlinear mapping on the multi-head self-attention one-dimensional convolution feature to generate a multi-head self-attention nonlinear feature; perform one-dimensional convolution on the multi-head self-attention nonlinear feature to output a first self-attention feature.

7. The open scenario oriented agent motion pattern prediction method according to claim 5, wherein, The third multi-scale branch module includes a self-attention submodule and a self-attention distillation downsampling submodule; the second sub-feature is input into the third multi-scale branch module for multi-scale feature extraction to generate a third multi-scale feature, which includes: the second sub-feature is input into the self-attention submodule to output a self-attention feature corresponding to the second sub-feature; the self-attention distillation downsampling submodule is used to distill and downsample the self-attention feature corresponding to the second sub-feature to generate a second sub-feature maximum pooling feature; the second sub-feature maximum pooling feature is input into the self-attention submodule to output a third multi-scale feature.

8. The open scenario oriented agent motion pattern prediction method according to claim 7, characterized in that, The self-attention distillation downsampling submodule is used to distill and downsample the self-attention feature corresponding to the second sub-feature to generate a second sub-feature maximum pooling feature, which includes: perform one-dimensional convolution on the self-attention feature corresponding to the second sub-feature to output a second sub-feature convolution feature; perform batch normalization on the second sub-feature convolution feature to generate a second sub-feature normalized feature; downsample the second sub-feature normalized feature to output a second sub-feature downsampled feature; perform maximum pooling on the second sub-feature downsampled feature to generate a second sub-feature maximum pooling feature.

9. The open scenario oriented agent motion pattern prediction method according to claim 1, wherein, The data processing process of the preset high-efficiency autoregressive decoder is specifically: a multi-layer perception is used to map the features of the input historical motion pattern sequence input into the preset high-efficiency autoregressive decoder to generate a high-dimensional historical motion feature; concatenate the historical motion high-dimensional feature and an input edge feature, an input map feature and an input future motion feature input to the preset efficient autoregressive decoder to generate a concatenated feature; perform a global adaptive one-dimensional average pooling operation on the concatenated feature to generate a global feature representation; perform time excitation on the global feature representation to output a time excitation feature; perform element-wise multiplication on the time excitation feature and the concatenated feature to generate a time sequence feature representation; perform feature mapping on the time sequence feature representation by using a multilayer perceptron to output a time sequence high-dimensional feature; perform layer normalization and nonlinear mapping on the time sequence high-dimensional feature to output a time sequence nonlinear feature; perform element-wise addition on the time sequence nonlinear feature and a historical time point prediction result to generate a current time point prediction result; concatenate the current time point prediction result and a target subsequence in the input historical motion mode sequence to generate a new input historical motion mode sequence; perform time step embedding and position encoding on the new input historical motion mode sequence to output an encoded feature sequence; perform attention operation on the encoded feature sequence based on a single-layer mask multi-head probability sparse self-attention mechanism to output a probability sparse self-attention; perform element-wise addition on the probability sparse self-attention and the encoded feature sequence to output a probability sparse self-attention feature sequence; perform layer normalization on the probability sparse self-attention feature sequence to determine a query matrix; perform linear projection on an input historical motion feature input to the preset efficient autoregressive decoder to generate a key matrix and a value matrix; generate an efficient decoding feature sequence according to the query matrix, the key matrix and the value matrix; perform layer normalization on the efficient decoding feature sequence to output an efficient decoding normalized feature sequence; perform feature mapping on the efficient decoding normalized feature sequence by using a multilayer perceptron to generate an output prediction result.

10. An open scenario oriented agent motion pattern prediction apparatus, characterized by, comprise: an acquisition module configured to acquire an agent historical motion mode sequence, an agent future motion planning sequence, a monitoring video dataset and a city road map, and perform map encoding on the city road map to generate a map feature vector; an according module configured to generate a plurality of two-dimensional spatial position coordinates according to an original annotation file of the monitoring video dataset, and perform homogeneous coordinate transformation and normalization on the plurality of two-dimensional spatial position coordinates to generate a plurality of motion state information; a using module configured to generate a time dimension edge sequence according to the plurality of motion state information and the plurality of two-dimensional spatial position coordinates by using a preset spatial edge rule; an embedding module configured to respectively perform embedding processing on the agent historical motion mode sequence, the agent future motion planning sequence and the time dimension edge sequence to generate a historical motion time step embedding corresponding to the agent historical motion mode sequence, a future motion time step embedding corresponding to the agent future motion planning sequence and an edge time step embedding corresponding to the time dimension edge sequence; The encoding module is configured to encode the history motion time step embedding, the future motion time step embedding and the edge time step embedding respectively by using a preset multi-scale history aggregation encoder, to generate multi-scale history motion features corresponding to the history motion time step embedding, multi-scale future motion features corresponding to the future motion time step embedding, and multi-scale edge features corresponding to the edge time step embedding; The decoding module is configured to output an initial prediction result according to the multi-scale history motion features, the multi-scale future motion features, the multi-scale edge features, the map feature vector and the agent history motion pattern sequence by using a plurality of preset efficient autoregressive decoders; The output result module is configured to generate a target agent motion pattern prediction result according to the initial prediction result by using a preset binary Gaussian mixture model.

Images