A pedestrian trajectory prediction method and system based on dynamic envelope

Abstract

This invention discloses a pedestrian trajectory prediction method and system based on a dynamic envelope, relating to the field of trajectory prediction. The method includes: acquiring images and performing 3D target detection using a binocular camera, projecting the data into a 2D bird's-eye view to extract crowd motion characteristics, clustering pedestrians using feature vectors such as velocity variance, density gradient, and historical trajectory similarity, and constructing a crowd envelope geometric model using an α-shape algorithm. The Transformer model is then used to predict the future state of the envelope, and the results are optimized using a residual correction module to achieve accurate prediction of pedestrian trajectories. By combining 3D target detection, 2D bird's-eye view modeling, crowd feature extraction, and the Transformer prediction model, the invention achieves high-precision trajectory prediction for pedestrians in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of trajectory prediction, and in particular to a pedestrian trajectory prediction method and system based on a dynamic envelope. Background Art

[0002] In intelligent transportation systems, autonomous navigation of service robots, and public security monitoring, motion trajectory prediction and abnormal behavior identification in high-density pedestrian scenarios are key technologies for dynamic obstacle avoidance and real-time decision-making. Existing technologies lack a macroscopic analysis of the entire crowd, resulting in high computational complexity, reaching O(N²) when the number of pedestrians is large. Furthermore, mainstream anomaly detection technologies rely on isolated parameters such as speed thresholds and trajectory offsets to determine anomalies, ignoring the propagation effect of individual behavioral disturbances in the crowd's potential energy field and failing to predict the critical phase transition point of the crowd's motion trend.

[0003] Traditional methods often use graph attention networks to extract spatial interactions between individuals, combined with temporal attention networks for trajectory prediction (e.g., the patent "Trajectory Prediction Method and Device Based on Dual Attention Mechanism" (CN114117259B)). However, focusing solely on interactions between individuals, they lack dynamic modeling of the overall geometric shape of the group. This makes it difficult to effectively represent complex morphological changes such as group splitting and merging in dense crowds. Furthermore, when the number of pedestrians N exceeds 30, the computational complexity of traditional methods reaches O(N²) (e.g., CN114117259B requires constructing an interaction matrix for each pedestrian).

[0004] In terms of dynamic decoupling of anomaly detection, mainstream technologies (such as the patent "Pedestrian Identification Method and System Based on Image and Motion Behavior Prediction" (CN119810759A)) rely on isolated parameters such as speed thresholds and trajectory offsets to determine anomalies, but ignore the propagation effect of individual behavioral disturbances within the group's potential energy field. This detection mechanism, based on local observations, cannot predict critical phase transitions in group motion trends (such as the energy accumulation phase before panicked flight).

[0005] Existing prediction architectures based on sequence modeling suffer from closed-loop instability. While LSTM-based models (such as those described in the patent "A Trajectory Prediction Method and Apparatus Based on a Dual Attention Mechanism" (CN114117259B)) can capture short-term motion patterns, their open-loop prediction characteristics lead to the continuous accumulation of errors due to environmental disturbances. In densely populated intersections, these models, lacking real-time feedback and correction of crowd dynamics parameters, often result in irreversible trajectory deviations, with nonlinear divergence as the prediction duration increases.

[0006] A core, long-standing contradiction in these fields lies in the fact that traditional methods treat groups as a linear superposition of individual trajectories, lacking a dynamic model of the group as a whole. This not only results in poor performance in representing complex morphological changes such as group splits and merges, but also significantly increases computational complexity with the number of pedestrians, making it difficult to meet real-time requirements. Furthermore, mainstream anomaly detection technologies, ignoring the propagation effect of individual behavioral disturbances in the group's potential energy field, are unable to predict the critical points of phase transitions in group motion trends. Summary of the Invention

[0007] In order to solve the above problems, the present invention proposes a pedestrian trajectory prediction method based on dynamic envelope. Through binocular camera 3D target detection, 2D bird's-eye view dimensionality reduction projection, crowd motion feature extraction and clustering grouping modeling, combined with the Transformer prediction model and residual correction module, accurate prediction of pedestrian trajectories is achieved.

[0008] The specific plan is as follows:

[0009] A pedestrian trajectory prediction method based on a dynamic envelope, comprising:

[0010] S1 performs 3D object detection on the original images captured by the binocular camera to obtain 3D object detection information. Based on the spatial coordinate information and depth information in the 3D object detection information, pedestrian bounding box and category label detection are performed. The overlapping bounding boxes in the detection results caused by crowd occlusion are removed, and the 3D object detection information is projected into a 2D bird's-eye view image to obtain 2D data with a unified scale representation of pedestrians at different distances. The dynamic motion characteristics of the crowd are captured in real time based on the 2D data.

[0011] S2, extracts the speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector from the crowd dynamic motion characteristics, classifies the pedestrian motion pattern based on the speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector, and classifies pedestrians with the same motion pattern and a distance less than the set threshold into the same cluster group. The algorithm dynamically models the geometric figures of the envelope of different cluster groups in a 2D bird's-eye view to obtain a dynamic model of the crowd envelope;

[0012] S3, based on the transformer model, predicts the dynamic model of the crowd envelope to obtain the future geometric features of the crowd envelope;

[0013] S4, updating the future geometric features of the crowd envelope through the residual correction module to obtain the updated future geometric features of the crowd envelope, and predicting the pedestrian trajectory based on the updated future geometric features of the crowd envelope.

[0014] Furthermore, in S1, the 3D target detection information is projected into the 2D bird's-eye view image using the following calculation formula:

[0015] ;

[0016] in, and Respectively represent the horizontal and vertical coordinates of the 2D bird's-eye view plane coordinates; 、 and They represent the horizontal coordinate, vertical coordinate, and depth coordinate in the 3D coordinate system in the world coordinate system, respectively.

[0017] Furthermore, in S2, the velocity variance feature vector is extracted from the dynamic motion characteristics of the crowd, and the calculation formula is as follows:

[0018] ;

[0019] in, The speed variance eigenvector reflects the pedestrian the degree of velocity fluctuation over the historical time period; Indicates the number of historical frames used to calculate velocity variance; represents the time step index, The value range is 1 to n, indicating the kth frame traced back from the current moment; Indicates the current moment; The unique identification index of the pedestrian to distinguish different pedestrians; Indicates pedestrians Velocity vector at time tk; Indicates pedestrians The average velocity vector over the past n frames;

[0020] The density gradient feature vector is extracted from the dynamic motion characteristics of the crowd. The calculation formula is as follows:

[0021] ;

[0022] in, Indicates pedestrians The neighborhood set of ; The number of pedestrians in the neighborhood set; Represents the density gradient feature vector, reflecting the pedestrian The rate of change of density between the location and the neighborhood; Represents the unique identification index of pedestrians in the neighborhood set; Indicates pedestrians The density of people at the location; Indicates pedestrians Coordinates in the 2D bird's-eye view; represents the coordinates of pedestrian j in the 2D bird's-eye view; Indicates pedestrians With pedestrians The Euclidean distance between represents a very small constant;

[0023] The formula for extracting the historical trajectory similarity feature vector from the dynamic motion characteristics of the crowd is as follows:

[0024] ;

[0025] in, and Indicates pedestrians and pedestrians In the near The trajectory vector formed by the position sequence within the frame; represents the cosine similarity between the two trajectories, which is used to measure the consistency of historical motion directions; 𝑚 represents the number of historical frames used to calculate trajectory similarity; Indicates pedestrians 2D coordinates at time tm; Represents the trajectory vector The L2 norm of Trajectory Vector Furthermore, the parameters of the crowd envelope dynamic model include: the coordinates of the envelope's centroid, used to describe the average displacement trend of the entire crowd; the major and minor axes of the envelope, used to describe the density of the crowd distribution; the area of ​​the envelope, used to indicate the size of the area covered by the envelope; the envelope deformation rate, used to reflect the degree of change in the crowd boundary; and the envelope density, used to indicate the number of pedestrians per unit area of ​​the envelope.

[0026] Furthermore, the calculation formula of the center of mass coordinates of the envelope is as follows:

[0027] ;

[0028] in, Represents the crowd envelope dynamic model The center of mass of the envelope at the moment; is the total number of vertices on the envelope boundary; The unique identification index of the envelope vertex; and Represents a vertex At the moment The two-dimensional coordinates of and Represents the coordinates of the center of mass in the x and y directions;

[0029] The calculation formulas for the major and minor axes of the envelope are as follows:

[0030] ;

[0031] ;

[0032] in, represents the major axis length of the envelope, The minor axis length of the envelope; The unique identification index of the envelope vertex; Indicates the The velocity vector of each pedestrian at time t is used to measure the degree of motion fluctuation of an individual or group; Indicates the boundary of the envelope The projection value of each vertex in the long axis direction is the first principal direction projection determined by principal component analysis; Represents the transpose of a unit vector in a certain direction, used for the second principal direction in projection operations.

[0033] Furthermore, the area calculation formula of the envelope shape is as follows:

[0034] ;

[0035] in, represents the area of ​​the envelope; represents the coordinates of the adjacent vertices of the envelope; s represents the total number of vertices on the boundary of the envelope; and Indicates the envelope The coordinates of the vertices at time t.

[0036] Furthermore, the calculation formula of the envelope deformation rate is as follows:

[0037] ;

[0038] in, represents the envelope shape change rate at time t; represents the time interval; s represents the total number of vertices on the boundary of the envelope; Indicates the envelope The velocity vector of each vertex at time t.

[0039] Furthermore, the calculation formula of the envelope density is as follows:

[0040] ;

[0041] in, Indicates the density of the envelope, which is used to indicate the number of pedestrians per unit area of ​​the envelope; The number of pedestrians within the envelope; Represents the area of ​​the envelope.

[0042] Furthermore, in S3, the crowd envelope dynamic model is predicted based on the transformer model, and the calculation formula is as follows:

[0043] ;

[0044] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and E represents the input feature matrix, which is used to input the geometric features of the crowd envelope dynamic model at the time step; It means that by associating the feature Q at each position with the feature K at all positions, it can adaptively capture the long-distance dependencies in the envelope time step features. Represents the scaling factor, which is used to prevent the softmax gradient from disappearing due to excessive dot product; represents the normalized attention weight.

[0045] On the other hand, a pedestrian trajectory prediction system based on a dynamic envelope includes:

[0046] The crowd dynamic motion feature capture module performs 3D target detection on the original images captured by the binocular camera to obtain 3D target detection information. Based on the spatial coordinate information and depth information in the 3D target detection information, it detects pedestrian bounding boxes and category labels, removes overlapping bounding boxes in the detection results caused by crowd occlusion, and reduces the dimension of the 3D target detection information and projects it into a 2D bird's-eye view image to obtain 2D data with a unified scale representation of pedestrians at different distances. Based on this 2D data, the dynamic motion features of the crowd are captured in real time.

[0047] The crowd envelope dynamic model acquisition module extracts the speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector from the crowd dynamic motion characteristics, and classifies the pedestrian motion patterns based on the speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector. Pedestrians with the same motion pattern and a distance less than the set threshold are grouped into the same cluster. The algorithm dynamically models the geometric figures of the envelope of different cluster groups in a 2D bird's-eye view to obtain a dynamic model of the crowd envelope;

[0048] The envelope geometric feature prediction module predicts the crowd envelope dynamic model based on the transformer model to obtain the future geometric features of the crowd envelope;

[0049] The pedestrian trajectory prediction module updates the future geometric features of the crowd envelope through the residual correction module, obtains the updated future geometric features of the crowd envelope, and predicts the pedestrian trajectory based on the updated future geometric features of the crowd envelope.

[0050] The present invention adopts the above technical solution and has the following beneficial effects:

[0051] (1) This paper performs 3D target detection on the original image of the binocular camera, removes overlapping bounding boxes based on depth information, and then projects the data into a 2D bird's-eye view image to unify the scale representation, effectively improving the accuracy of pedestrian detection and dynamic capture in complex crowd scenes;

[0052] (2) The present invention extracts characteristic vectors such as velocity variance, density gradient, and trajectory similarity, and combines them with the α-shape algorithm to perform geometric modeling on different cluster groups. This method can dynamically construct a crowd envelope model with semantic information to reflect the characteristics of group behavior.

[0053] (3) The present invention predicts the geometric features of the crowd envelope dynamic model based on the Transformer model and optimizes the prediction results through the residual correction module, thereby improving the robustness and long-term prediction ability of pedestrian trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a method for predicting pedestrian trajectories based on a dynamic envelope according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of a group trajectory prediction method based on the crowd envelope concept according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the overall architecture of a trajectory prediction model according to an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of a pedestrian trajectory prediction system based on dynamic envelope according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0059] like Figure 1 As shown, the pedestrian trajectory prediction method based on the dynamic envelope of the present invention includes:

[0060] S1 performs 3D target detection on the original images captured by the binocular camera to obtain 3D target detection information. Based on the spatial coordinate information and depth information in the 3D target detection information, pedestrian bounding box and category label detection are performed. The overlapping bounding boxes in the detection results caused by crowd occlusion are removed, and the 3D target detection information is projected into a 2D bird's-eye view image to obtain 2D data with a unified scale representation of pedestrians at different distances. Based on the 2D data, the dynamic motion characteristics of the crowd are captured in real time.

[0061] Specifically, in S1, the 3D target detection information is projected into the 2D bird's-eye view image using the following calculation formula:

[0062] ;

[0063] in and Represents the horizontal and vertical coordinates of the 2D bird's-eye view plane coordinates , and Represents the horizontal coordinate, vertical coordinate, and depth coordinate in the 3D coordinate system in the world coordinate system.

[0064] S2, extracts speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector from the crowd dynamic motion characteristics, classifies pedestrian motion patterns based on speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector, and groups pedestrians with the same motion pattern and a distance less than a set threshold into the same cluster group. The algorithm dynamically models the geometric figures of the envelope of different cluster groups to obtain the dynamic model of the crowd envelope.

[0065] Specifically, such as Figure 2 As shown, it can be seen that the concept of crowd envelope is visualized, and its principle is as follows;

[0066] Pedestrians and directions: In the figure, "O" represents a detected pedestrian object, and the arrow indicates the direction of movement of the individual;

[0067] Crowd envelope and envelope: The dotted line "detection envelope" represents the crowd contour (i.e., envelope) generated by the α-shape algorithm at the current moment, and the "prediction envelope" represents the predicted result of the crowd contour at the future moment;

[0068] Smart Car and Forward Direction: The "smart car" at the bottom moves in the forward direction and predicts the group's movement trends and plans its path by sensing the dynamics of the surrounding "crowd envelope" (including the current detected outline and the future predicted outline). Specifically, in S2, the velocity variance feature vector is extracted from the dynamic motion characteristics of the crowd. The calculation formula is as follows:

[0069] ;

[0070] in, The speed variance eigenvector reflects the pedestrian the degree of velocity fluctuation over the historical time period; Indicates the number of historical frames used to calculate velocity variance; represents the time step index, The value range is 1 to n, indicating the kth frame traced back from the current moment; Indicates the current moment; The unique identification index of the pedestrian to distinguish different pedestrians; Indicates pedestrians Velocity vector at time tk; Indicates pedestrians The average velocity vector over the past n frames.

[0071] The density gradient feature vector is extracted from the dynamic motion characteristics of the crowd. The calculation formula is as follows:

[0072] ;

[0073] in, Indicates pedestrians The neighborhood set of ; The number of pedestrians in the neighborhood set; Represents the density gradient feature vector, reflecting the pedestrian The rate of change of density between the location and the neighborhood; Represents the unique identification index of pedestrians in the neighborhood set; Indicates pedestrians The density of people at the location; Indicates pedestrians Coordinates in the 2D bird's-eye view; Indicates pedestrians Coordinates in the 2D bird's-eye view; Pedestrian j and pedestrian The Euclidean distance between represents a very small constant;

[0074] The formula for extracting the historical trajectory similarity feature vector from the dynamic motion characteristics of the crowd is as follows

[0075] ;

[0076] in, and Indicates pedestrians and pedestrians In the near The trajectory vector formed by the position sequence within the frame; represents the cosine similarity between the two trajectories, which is used to measure the consistency of historical motion directions; m represents the number of historical frames used to calculate trajectory similarity; Indicates pedestrians 2D coordinates at time tm; Represents the trajectory vector The L2 norm of Trajectory Vector The L2 norm of .

[0077] Specifically, the parameters of the crowd envelope dynamic model include: coordinates of the envelope's center of mass, the major and minor axes of the envelope, the area of ​​the envelope's shape, the envelope's shape change rate, and the envelope's density;

[0078] The centroid coordinates of the envelope are used to describe the average displacement trend of the entire crowd;

[0079] The major and minor axes of the envelope are used to describe the compactness of the population distribution;

[0080] The area of ​​the envelope shape is used to indicate the size of the area covered by the envelope;

[0081] The envelope deformation rate is used to reflect the degree of change of crowd boundaries;

[0082] Envelope density is used to indicate the number of pedestrians per unit area of ​​the envelope.

[0083] Specifically, the calculation formula of the envelope body's center of mass coordinates is as follows:

[0084] ;

[0085] in, It represents the mass center of the crowd envelope at time t of the dynamic model of the crowd envelope; is the total number of vertices on the envelope boundary; i represents the unique identification index of the envelope vertex; and Represents a vertex Two-dimensional coordinates at time t; and Represents the coordinates of the center of mass in the x and y directions.

[0086] Specifically, the calculation formulas for the major and minor axes of the envelope are as follows:

[0087] ;

[0088] ;

[0089] in, represents the major axis length of the envelope, The minor axis length of the envelope; The unique identification index of the envelope vertex; Indicates the The velocity vector of each pedestrian at time t is used to measure the degree of motion fluctuation of an individual or group; Indicates the boundary of the envelope The projection value of each vertex in the long axis direction is the first principal direction projection determined by principal component analysis; Represents the transpose of a unit vector in a certain direction, used for the second principal direction in projection operations.

[0090] Specifically, the area calculation formula of the envelope shape is as follows:

[0091] ;

[0092] in, represents the area of ​​the envelope; represents the coordinates of the adjacent vertices of the envelope; s represents the total number of vertices on the boundary of the envelope; and Indicates the envelope The coordinates of the vertices at time t. Indicates that when the index k of the boundary vertex of the envelope is the total number of vertices s (that is, the last vertex), the attribute of its next adjacent vertex (index k+1) at time t is the same as the attribute of the first vertex at time t. The attributes of the moments are consistent, realizing the closed connection between the beginning and the end of the envelope vertices.

[0093] Specifically, the calculation formula of the envelope deformation rate is as follows:

[0094] ;

[0095] in, express Momentary envelope shape change rate; represents the time interval; s represents the total number of vertices on the boundary of the envelope; Indicates the envelope Vertices in Velocity vector at time.

[0096] Specifically, the calculation formula of the envelope density is as follows:

[0097] ;

[0098] in, Indicates the density of the envelope, which is used to indicate the number of pedestrians per unit area of ​​the envelope; The number of pedestrians within the envelope; Represents the area of ​​the envelope.

[0099] S3, based on the transformer model, predicts the dynamic model of the crowd envelope to obtain the future geometric features of the crowd envelope.

[0100] Its execution logic is as follows Figure 3 Trajectory prediction and correction are the core steps for accurately outputting the group's future motion trends. This includes two processes: Transformer initial prediction and residual correction trajectory prediction.

[0101] Specifically, the crowd envelope dynamic model is predicted based on the transformer model, and the calculation formula is as follows:

[0102] ;

[0103] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and E represents the input feature matrix, which is used to input the geometric features of the crowd envelope dynamic model at the time step; It represents the adaptive capture of long-range dependencies in the input sequence (such as envelope time step features) by associating the features (Q) at each position with the features (K) at all positions. Represents the scaling factor to avoid the softmax gradient disappearing due to excessive dot product; Normalize the attention weights to highlight the contribution of key features.

[0104] S4: The residual correction module updates the future geometric features of the crowd envelope to obtain the updated future geometric features of the crowd envelope. Pedestrian trajectories are then predicted based on the updated future geometric features of the crowd envelope. Specifically, this embodiment treats the pedestrian group as a whole (envelope) and constructs a dynamic envelope model to predict the movement trajectory of the entire crowd and monitor abnormal individual behavior within the group. Predicting the entire crowd reduces computational targets and complexity, while leveraging the repeatability and tracking characteristics of crowd trajectories to improve prediction efficiency and accuracy. Crowd behaviors such as gathering and spreading are represented by the expansion and contraction of the envelope and changes in its geometric shape. The positional changes and displacements of the envelope outline are used to represent and predict the overall movement trajectory of the crowd. Furthermore, by predicting changes in the envelope line, crowd trajectories and abnormal individual behavior can be predicted. Abnormal behavior is converted into a mathematical problem, and its impact on the envelope is quantified, allowing the generation of an envelope whose shape changes due to abnormal behavior to be generated in advance.

[0105] Specifically, the overall execution logic is as follows Figure 3 As shown in the figure, the process involves acquiring RGB images using a binocular camera, extracting bounding boxes through 3D object detection, and then reducing the dimensionality of the 3D information and projecting it onto a 2D bird's-eye view. The 3D object detection information includes each pedestrian's spatial coordinates, depth, bounding box, and category parameters. The system then uses DBSCAN clustering and the α-shape method to construct a dynamic envelope model and detect abnormal behavior. The Transformer is then used for preliminary trajectory prediction, with residual correction improving prediction accuracy. Finally, the predicted future crowd envelope trajectory is output, triggering an anomaly signal warning to enable timely detection and handling of potential abnormal events.

[0106] Specifically, in this embodiment, the geometric and motion feature parameters such as centroid time series, axis length features, area and density, and deformation rate are spliced ​​into an input matrix according to the time dimension as the input of the Transformer neural network, and feature time series dependency learning is performed through the multi-head attention mechanism and position encoding: through the changing trend of the long axis and short axis length, the Transformer model determines whether the crowd is in a gathering or diffusion state; through the changing trend of the area and density, the Transformer model determines the movement trend of the crowd; through the deformation rate, the Transformer model measures the dynamic fluctuation of the crowd shape; through the similarity of historical trajectories, the Transformer model determines whether there are abnormal individuals in the crowd that are inconsistent with the overall movement trend of the crowd or have large differences with the historical movement trajectory. When predicting and generating the future crowd envelope dynamic model, the residual correction module is used to update the preliminary prediction results to improve the prediction accuracy. Its execution logic is as follows: Figure 3 As shown in the residual correction trajectory prediction, the deviation between the preliminary predicted trajectory and the historical true trajectory is used as input, the error pattern is learned and a correction value is generated, which is superimposed on the preliminary predicted trajectory to obtain a high-precision final predicted trajectory.

[0107] The correction process involves predicting a sequence of N frames into the future at time point t. When the system reaches time point t+k (where 1≤k<N), the true value at the current moment is obtained and used to replace the corresponding frame in the original predicted sequence. Subsequent frames are then re-predicted based on this value. Finally, the updated predicted frame is inserted into the original sequence. When the difference between the envelope area and deformation rate of the true observation and the predicted value exceeds a threshold, the residual correction module prioritizes adjusting the prediction weight of the corresponding feature to prevent error accumulation.

[0108] Specifically, the implementation ideas of the present invention can be summarized as calling the built-in 3D target detection function of the binocular camera driver SDK to obtain the pedestrian 3D bounding box and category label, combining depth information to filter false detections (such as distinguishing pedestrians from similar objects), and removing overlapping bounding boxes from the detection results; constructing multi-dimensional feature vectors, such as velocity variance, density gradient, envelope deformation rate, historical trajectory similarity, etc.; grouping and clustering the crowd, and performing envelope dynamic modeling for each different group; quickly identifying abnormal individuals, such as individuals with sudden changes in direction, speed, and envelope position, and quantifying their impact on group movement; predicting envelope deformation and trajectory evolution based on the results of anomaly detection, and correcting the predicted trajectory through the residual correction module. Improved group prediction efficiency and accuracy: By treating the pedestrian group as a whole for prediction, the calculation targets are reduced, the calculation complexity is reduced, and the problem of excessive calculation amount of traditional methods when the number of individuals is large is avoided; at the same time, the repeatability and followability of the crowd trajectory are utilized to improve the prediction accuracy. Improved accuracy and dynamic adaptability of group geometric representation: The present invention breaks through the limitations of the traditional convex hull model through a dynamic envelope modeling strategy, achieving accurate representation of the complex forms of dense crowds (such as concave structures and dynamic deformations). This method can capture the spatial distribution and morphological changes of the group (such as aggregation, diffusion, splitting, etc.) in real time, providing an intuitive and accurate geometric basis for trajectory prediction, and effectively solving the problems of blurred boundaries and incomplete structural representation in dense scenes. Effectiveness of group motion pattern recognition and abnormal behavior detection: The motion pattern analysis method based on multi-dimensional feature fusion and dynamic clustering of the present invention can efficiently distinguish typical group behaviors such as following, diffusion, and crossing, providing prior knowledge for trajectory prediction. The abnormal individual detection module can quickly identify sudden behaviors such as sudden changes in direction and abnormal speed by quantifying behavioral disturbance parameters, and evaluate their impact on group motion, providing key warning information for real-time decision-making, and improving the system's adaptability to complex scenarios. Optimization of trajectory prediction accuracy and long-term stability: This invention introduces a residual correction module, learns historical prediction deviations, and dynamically compensates for the cumulative errors that may occur during the long-term prediction process. This makes the trajectory evolution trend more consistent with the actual pedestrian movement patterns, effectively improves the reliability of the prediction results, and provides a stable trajectory reference for real-time obstacle avoidance and path planning of smart devices.

[0109] Specifically, in this embodiment, the hardware and data acquisition uses a ZED 2i industrial binocular camera as the image acquisition device, connected to an edge computing unit via a USB 3.0 interface, to acquire RGB images (1280×720 resolution) and depth maps (32-bit precision) in real time. The camera is mounted at a height of 1.5-2 meters and covers a field of view of approximately 120°, ensuring complete imaging of pedestrians. Dynamic clustering and grouping: Pedestrians are grouped using a density and spatial distance clustering algorithm (or the DBSCAN algorithm based on trajectory similarity). Pedestrians with similar spatial distances and high density within a specific area are grouped together (e.g., following mode groups or crossing mode groups). Pedestrians within each group are ensured to meet motion trend consistency constraints (e.g., individual velocity deviation within a group is less than a preset threshold). Group envelope generation: The α-shape algorithm is applied independently to each cluster group to generate a 3D envelope model containing the concave structure. Geometric parameters such as the center of mass position, major / minor axis directions, and envelope area are calculated for each group. It also analyzes the spatial relationship between envelopes of different groups (such as the degree of overlap and relative movement trends of adjacent groups), fuses multiple sets of envelope parameters through Kalman filtering, constructs a global group dynamic model, and updates the position and morphological changes of each group of envelopes in real time. Abnormal individual detection: Real-time monitoring of individual behavioral parameters, including movement direction, speed, etc., quickly detects abnormal individuals based on preset abnormal behavior definitions, quantifies their behavioral disturbance parameters, and evaluates their impact on group movement. Trajectory prediction and correction: Based on the sequence modeling logic of the Transformer encoder, combined with the geometric parameters of the envelope and the group movement trend, the overall trajectory of the pedestrian group is predicted in the short term; at the same time, a residual correction module is introduced to learn historical prediction deviations and dynamically compensate for the accumulated errors that may occur in the long-term prediction process, thereby improving the accuracy and long-term stability of trajectory prediction.

[0110] like Figure 4 As shown, this embodiment also discloses a device for evaluating screen content video quality based on deep and shallow spatiotemporal features, including:

[0111] The crowd dynamic motion feature capture module 41 performs 3D object detection on the original images captured by the binocular camera to obtain 3D object detection information. Based on the spatial coordinate information and depth information in the 3D object detection information, it detects bounding boxes and category labels for pedestrians, removes overlapping bounding boxes in the detection results caused by crowd occlusion, and reduces the dimension of the 3D object detection information and projects it into a 2D bird's-eye view image to obtain 2D data with a unified scale representation of pedestrians at different distances. Based on this 2D data, the dynamic motion features of the crowd are captured in real time.

[0112] The crowd envelope dynamic model acquisition module 42 extracts the speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector from the crowd dynamic motion characteristics, and classifies the pedestrian motion patterns based on the speed variance feature vector, density gradient feature vector and historical trajectory similarity feature vector. Pedestrians with the same motion pattern and a distance less than a set threshold are grouped into the same cluster. The algorithm dynamically models the geometric figures of the envelope of different cluster groups in a 2D bird's-eye view to obtain a dynamic model of the crowd envelope;

[0113] The envelope geometric feature prediction module 43 predicts the crowd envelope dynamic model based on the transformer model to obtain the future geometric features of the crowd envelope;

[0114] The pedestrian trajectory prediction module 44 updates the future geometric features of the crowd envelope through the residual correction module to obtain the updated future geometric features of the crowd envelope, and predicts the pedestrian trajectory based on the updated future geometric features of the crowd envelope.

[0115] The specific implementation of the pedestrian trajectory prediction system based on the dynamic envelope is the same as the pedestrian trajectory prediction method based on the dynamic envelope, and will not be repeated in this embodiment.

[0116] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A pedestrian trajectory prediction method based on dynamic envelope, characterized in that: include: S1 performs 3D object detection on the original images captured by the binocular camera to obtain 3D object detection information. Based on the spatial coordinate information and depth information in the 3D object detection information, pedestrian bounding box and category label detection are performed. The overlapping bounding boxes in the detection results caused by crowd occlusion are removed, and the 3D object detection information is projected into a 2D bird's-eye view image to obtain 2D data with a unified scale representation of pedestrians at different distances. The dynamic motion characteristics of the crowd are captured in real time based on the 2D data. S2, extracting velocity variance eigenvectors, density gradient eigenvectors, and historical trajectory similarity eigenvectors from the crowd's dynamic motion characteristics, classifying pedestrian motion patterns based on these eigenvectors, density gradient eigenvectors, and historical trajectory similarity eigenvectors. Pedestrians with the same motion pattern and a distance less than a set threshold are grouped into the same cluster. Dynamically modeling the envelope geometry of different clusters in a 2D bird's-eye view using the α-shape algorithm to obtain a crowd envelope dynamic model. The velocity variance feature vector is extracted from the dynamic motion characteristics of the crowd. The calculation formula is as follows: in, The speed variance feature vector reflects the speed fluctuation degree of pedestrian i in the historical time period; n represents the number of historical frames used to calculate the speed variance; k represents the time step index, which ranges from 1 to n and represents the kth frame traced back from the current moment; t represents the current moment; i represents the unique identification index of the pedestrian to distinguish different pedestrians; v i,t-k represents the velocity vector of pedestrian i at time tk; represents the average velocity vector of pedestrian i in the past n frames; The density gradient feature vector is extracted from the dynamic motion characteristics of the crowd. The calculation formula is as follows: in, represents the neighborhood set of pedestrian i; The number of pedestrians in the neighborhood set; represents the density gradient feature vector, reflecting the density change rate between the location of pedestrian i and the neighborhood; j represents the unique identification index of the pedestrian in the neighborhood set; ρ j represents the crowd density at the location of pedestrian i; ρ i represents the coordinates of pedestrian i in the 2D bird's-eye view; ρ j represents the coordinates of pedestrian j in the 2D bird's-eye view; ||p j -p i || represents the Euclidean distance between pedestrian j and pedestrian i; ε represents a minimum constant; The formula for extracting the historical trajectory similarity feature vector from the dynamic motion characteristics of the crowd is as follows: Among them, T i and T j represents the trajectory vector composed of the position sequence of pedestrians i and j in the past m frames; s ij represents the cosine similarity between the two trajectories, which is used to measure the consistency of the historical motion direction; m represents the number of historical frames used to calculate the trajectory similarity; p i,t-m represents the 2D coordinates of pedestrian i at time tm; ||T i || represents the trajectory vector T i L2 norm of ||T j ||Trajectory vector T j The L2 norm of S3, based on the transformer model, predicts the dynamic model of the crowd envelope to obtain the future geometric features of the crowd envelope; S4, updating the future geometric features of the crowd envelope through the residual correction module to obtain the updated future geometric features of the crowd envelope, and predicting the pedestrian trajectory based on the updated future geometric features of the crowd envelope.

2. The pedestrian trajectory prediction method based on dynamic envelope according to claim 1, characterized in that: In S1, the 3D target detection information is projected into the 2D bird's-eye view image using the following calculation formula: Among them, x bev and y bev Respectively represent the horizontal and vertical coordinates of the 2D bird's-eye view plane coordinates; x w 、y w and z w They represent the horizontal coordinate, vertical coordinate, and depth coordinate in the 3D coordinate system in the world coordinate system, respectively.

3. The pedestrian trajectory prediction method based on dynamic envelope according to claim 1, characterized in that: In S2, the parameters of the crowd envelope dynamic model include: the coordinates of the centroid of the envelope used to describe the average displacement trend of the entire crowd, the major and minor axes of the envelope used to describe the density of the crowd distribution, the area of ​​the envelope shape used to indicate the size of the area covered by the envelope, the envelope deformation rate used to reflect the degree of change of the crowd boundary, and the envelope density used to indicate the number of pedestrians per unit area of ​​the envelope.

4. The pedestrian trajectory prediction method based on dynamic envelope according to claim 3, characterized in that: The calculation formula of the center of mass coordinates of the envelope is as follows: Among them, C t represents the mass center of the crowd envelope at time t of the dynamic model; s is the total number of vertices on the envelope boundary; i represents the unique identification index of the envelope vertex; x i,t and y i,t represents the two-dimensional coordinates of vertex i at time t; c x,t and c y,t Represents the coordinates of the center of mass in the x and y directions; The calculation formulas for the major and minor axes of the envelope are as follows: L major,t =max(s i )-min(s i ); Among them, L major,t Indicates the major axis length of the envelope, L minor,t The short axis length of the envelope; i represents the unique identification index of the envelope vertex; v i,t represents the velocity vector of the i-th pedestrian at time t, which is used to measure the degree of motion fluctuation of an individual or group; i Represents the projection value of the i-th vertex of the envelope boundary in the long axis direction, which is the first principal direction projection determined by principal component analysis; Represents the transpose of a unit vector in a certain direction, used for the second principal direction in projection operations.

5. The pedestrian trajectory prediction method based on dynamic envelope according to claim 3, characterized in that: The area calculation formula of the envelope shape is as follows: Among them, A t represents the area of ​​the envelope; x i,t y i+1,t represents the coordinates of the adjacent vertices of the envelope; s represents the total number of vertices on the envelope boundary; x i,t and y i,t Represents the coordinates of the i-th vertex of the envelope at time t.

6. The pedestrian trajectory prediction method based on dynamic envelope according to claim 3, characterized in that: The calculation formula of the envelope deformation rate is as follows: Among them, D t represents the deformation rate of the envelope at time t; Δt represents the time interval; s represents the total number of vertices on the envelope boundary; v i,t Represents the velocity vector of the i-th vertex of the envelope at time t.

7. The pedestrian trajectory prediction method based on dynamic envelope according to claim 3, characterized in that: The calculation formula of the envelope density is as follows: Among them, ρ t Indicates the density of the envelope, which is used to indicate the number of pedestrians per unit area of ​​the envelope; N t The number of pedestrians within the envelope; A t Represents the area of ​​the envelope.

8. The pedestrian trajectory prediction method based on dynamic envelope according to claim 1, characterized in that, in S3, the crowd envelope dynamic model is predicted based on the transformer model, and the calculation formula is as follows: in, Q represents the query vector, K represents the key vector, V represents the value vector, and E represents the input feature matrix, which is used to input the geometric features of the crowd envelope dynamic model at the time step; Attention(Q,K,V) means that the feature Q of each position is associated with the feature K of all positions to adaptively capture the long-distance dependency in the envelope time step features; d k Represents the scaling factor, which is used to prevent the softmax gradient from disappearing due to excessive dot product; softmax() represents the normalization of attention weights.

9. A pedestrian trajectory prediction system based on dynamic envelope, characterized in that: include: The crowd dynamic motion feature capture module performs 3D target detection on the original images captured by the binocular camera to obtain 3D target detection information. Based on the spatial coordinate information and depth information in the 3D target detection information, it detects pedestrian bounding boxes and category labels, removes overlapping bounding boxes in the detection results caused by crowd occlusion, and reduces the dimension of the 3D target detection information and projects it into a 2D bird's-eye view image to obtain 2D data with a unified scale representation of pedestrians at different distances. Based on this 2D data, the dynamic motion features of the crowd are captured in real time. The crowd envelope dynamic model acquisition module extracts the velocity variance eigenvector, density gradient eigenvector, and historical trajectory similarity eigenvector from the crowd's dynamic motion characteristics. Pedestrians' motion patterns are classified based on the velocity variance eigenvector, density gradient eigenvector, and historical trajectory similarity eigenvector. Pedestrians with the same motion pattern and a distance less than a set threshold are grouped into the same cluster. The α-shape algorithm is used to dynamically model the envelope geometry of different clusters in a 2D bird's-eye view to obtain the crowd envelope dynamic model. The velocity variance eigenvector is extracted from the crowd's dynamic motion characteristics using the following calculation formula: in, The speed variance feature vector reflects the speed fluctuation degree of pedestrian i in the historical time period; n represents the number of historical frames used to calculate the speed variance; k represents the time step index, which ranges from 1 to n and represents the kth frame traced back from the current moment; t represents the current moment; i represents the unique identification index of the pedestrian to distinguish different pedestrians; v i,t-k represents the velocity vector of pedestrian i at time tk; represents the average velocity vector of pedestrian i in the past n frames; The density gradient feature vector is extracted from the dynamic motion characteristics of the crowd. The calculation formula is as follows: in, represents the neighborhood set of pedestrian i; The number of pedestrians in the neighborhood set; represents the density gradient feature vector, reflecting the density change rate between the location of pedestrian i and the neighborhood; j represents the unique identification index of the pedestrian in the neighborhood set; ρ j represents the crowd density at the location of pedestrian i; ρ i represents the coordinates of pedestrian i in the 2D bird's-eye view; ρ j represents the coordinates of pedestrian j in the 2D bird's-eye view; ||p j -p i || represents the Euclidean distance between pedestrian j and pedestrian i; ε represents a minimum constant; The formula for extracting the historical trajectory similarity feature vector from the dynamic motion characteristics of the crowd is as follows: Among them, T i and T j represents the trajectory vector composed of the position sequence of pedestrians i and j in the past m frames; s ij represents the cosine similarity between the two trajectories, which is used to measure the consistency of the historical motion direction; m represents the number of historical frames used to calculate the trajectory similarity; p i,t-m represents the 2D coordinates of pedestrian i at time tm; ||T i || represents the trajectory vector T i L2 norm of ||T j ||Trajectory vector T j The L2 norm of The envelope geometric feature prediction module predicts the crowd envelope dynamic model based on the transformer model to obtain the future geometric features of the crowd envelope; The pedestrian trajectory prediction module updates the future geometric features of the crowd envelope through the residual correction module, obtains the updated future geometric features of the crowd envelope, and predicts the pedestrian trajectory based on the updated future geometric features of the crowd envelope.

Images