Pedestrian abnormal behavior detection method, device and storage medium in mixed environment of human and vehicle

Abstract

The application relates to the field of artificial intelligence, and provides a pedestrian abnormal behavior detection method in a mixed pedestrian and vehicle environment, which comprises the following steps: acquiring continuous image frames of a mixed pedestrian and vehicle scene in real time; detecting pedestrians in the continuous image frames, extracting pedestrian positions and appearance features, and generating initial pedestrian trajectories; dynamically tracking the pedestrian trajectories based on a multi-modal motion state, and generating real-time updated pedestrian tracking trajectories; constructing a space-time interaction graph network, predicting a future trajectory of the pedestrian, and generating a trajectory confidence; determining a pedestrian behavior abnormality level according to a space-time deviation amount of the future trajectory of the pedestrian and the tracking trajectory of the pedestrian, a trajectory confidence, and an interaction relationship mutation feature; and when the pedestrian behavior abnormality level exceeds a preset threshold, triggering a graded early warning signal and generating avoidance guidance information. The technical scheme can effectively improve tracking continuity, prediction accuracy and early warning timeliness.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, device and storage medium for detecting abnormal pedestrian behavior in mixed pedestrian and vehicle traffic environments. Background Technology

[0002] As urban traffic scenarios become increasingly complex, the need for safety management in environments where pedestrians and vehicles coexist is becoming more urgent. In such scenarios, the uncertainty of pedestrian behavior (such as sudden changes of direction or vehicular entry into lanes) and the mutual interference of vehicle dynamics make anomaly detection face problems such as trajectory drift and difficulty in modeling interaction relationships, thus requiring reliable technical means to achieve accurate early warning.

[0003] Currently, existing technologies for pedestrian anomaly detection are mainly based on single-modal tracking detection methods and / or prediction methods based on simple spatiotemporal models. However, these existing technologies suffer from key shortcomings such as insufficient robustness in dynamic tracking, lack of interaction relationship modeling, and rigid early warning mechanisms. Summary of the Invention

[0004] This application provides a method, device, and storage medium for detecting abnormal pedestrian behavior in mixed pedestrian and vehicle traffic environments, which can effectively improve tracking continuity, prediction accuracy, and early warning timeliness.

[0005] On the one hand, this application provides a method for detecting abnormal pedestrian behavior in a mixed pedestrian and vehicle environment, the method comprising:

[0006] Real-time acquisition of continuous image frames of mixed pedestrian and vehicle traffic scenes;

[0007] Pedestrians are detected in the continuous image frames, their positions and appearance features are extracted, and initial pedestrian trajectories are generated.

[0008] Dynamic tracking of pedestrian trajectories is performed based on multimodal motion states to generate real-time updated pedestrian tracking trajectories;

[0009] Construct a spatiotemporal interaction graph network to predict future pedestrian trajectories and generate trajectory confidence scores;

[0010] The level of pedestrian behavior abnormality is determined based on the spatiotemporal deviation between the future trajectory of the pedestrian and the tracking trajectory of the pedestrian, the trajectory confidence, and the abrupt change characteristics of the interaction relationship.

[0011] When the level of abnormal pedestrian behavior exceeds a preset threshold, a graded early warning signal is triggered and avoidance guidance information is generated.

[0012] On the other hand, this application provides a pedestrian abnormal behavior detection device in a mixed pedestrian and vehicle traffic environment, the device comprising:

[0013] The acquisition module is used to acquire continuous image frames in real time of mixed pedestrian and vehicle traffic scenes;

[0014] The first generation module is used to detect pedestrians in the continuous image frames, extract pedestrian positions and appearance features, and generate initial pedestrian trajectories;

[0015] The second generation module is used to dynamically track pedestrian trajectories based on multimodal motion states and generate real-time updated pedestrian tracking trajectories.

[0016] The prediction module is used to construct a spatiotemporal interaction graph network, predict the future trajectory of pedestrians, and generate trajectory confidence scores.

[0017] The determination module is used to determine the level of abnormal pedestrian behavior based on the spatiotemporal deviation between the future trajectory of the pedestrian and the tracking trajectory of the pedestrian, the trajectory confidence, and the abrupt change characteristics of the interaction relationship.

[0018] The triggering module is used to trigger a graded early warning signal and generate avoidance guidance information when the pedestrian behavior abnormality level exceeds a preset threshold.

[0019] Thirdly, this application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the technical solution of the pedestrian abnormal behavior detection method in the above-described mixed pedestrian and vehicle environment.

[0020] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for detecting abnormal pedestrian behavior in mixed pedestrian and vehicle traffic environments.

[0021] As can be seen from the technical solution provided in this application, on the one hand, by fusing multimodal motion states of position prediction error, appearance similarity, and scene geometric constraints, robust tracking trajectories are generated in occluded or complex motion scenarios, reducing the risk of misjudgment caused by trajectory breakage; on the other hand, by constructing a spatiotemporal interaction graph network, the spatiotemporal characteristics of pedestrian historical trajectories and their interaction with vehicles are fused, improving the accuracy of future trajectory prediction and providing a reliable confidence basis for anomaly determination; thirdly, based on the spatiotemporal deviation between the predicted trajectory and the tracking trajectory, trajectory confidence, and abrupt change characteristics of interaction relationships, the anomaly level is dynamically determined and a graded early warning signal is triggered, significantly improving the adaptability and timeliness of anomaly detection. In summary, the technical solution of this application can effectively improve tracking continuity, prediction accuracy, and early warning timeliness. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the pedestrian abnormal behavior detection method provided in the embodiments of this application in a mixed pedestrian and vehicle traffic environment;

[0024] Figure 2 This is a schematic diagram of the pedestrian abnormal behavior detection device provided in the embodiments of this application in a mixed pedestrian and vehicle environment;

[0025] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element, component, or step (etc.) should not be construed as limited to only one element, component, or step, but may include one or more of the elements, components, or steps, etc.

[0028] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0029] As urban traffic scenarios become increasingly complex, the need for safety management in environments where pedestrians and vehicles coexist is becoming more urgent. In such scenarios, the uncertainty of pedestrian behavior (such as sudden changes of direction or vehicular entry into lanes) and the mutual interference of vehicle dynamics make anomaly detection face problems such as trajectory drift and difficulty in modeling interaction relationships, thus requiring reliable technical means to achieve accurate early warning.

[0030] Currently, existing technologies for pedestrian anomaly detection are mainly based on the following two types of methods:

[0031] 1) Detection methods based on single-modal tracking: Tracking is performed using a single feature of the pedestrian's appearance or movement trajectory, and anomalies are determined by combining the trajectory deviation threshold. For example, a Kalman filter is used to predict the pedestrian's position, and an alarm is triggered based on the deviation between the trajectory and the predicted position.

[0032] 2) Prediction method based on simple spatiotemporal model: Use linear regression or RNN model to predict pedestrian trajectory in the short term, and combine it with preset rules to judge behavioral risk.

[0033] However, the aforementioned existing technologies have the following key drawbacks: 1) Insufficient robustness of dynamic tracking: Single-modal tracking methods are prone to trajectory breakage or mismatch in complex scenarios such as occlusion and changes in lighting, and cannot stably generate continuous pedestrian motion state representations; 2) Lack of interaction relationship modeling: Existing methods do not fully consider the spatiotemporal interaction influence between pedestrians and vehicles, resulting in low confidence in trajectory prediction; 3) Rigid early warning mechanism: The judgment method based on fixed thresholds is difficult to adapt to changes in risk levels in dynamic scenarios, and is prone to missed or false alarms.

[0034] To address the aforementioned problems in existing technologies, this application proposes a method for detecting abnormal pedestrian behavior in mixed pedestrian and vehicle traffic environments, the flowchart of which is attached. Figure 1 As shown, the main steps include S101 to S106, which are detailed below:

[0035] Step S101: Acquire continuous image frames of the mixed pedestrian and vehicle traffic scene in real time.

[0036] It should be noted that, in the embodiments of this application, continuous image frames of pedestrian and vehicle mixed traffic scenarios can be acquired using a single video acquisition device, such as a camera in real time, or by using a video acquisition device together with other sensors such as radar and LiDAR, i.e., radar vision devices. The difference is that the former avoids the problem of inconsistent trajectory data formats caused by differences in sensor types, while the latter may require preprocessing such as alignment of images acquired by different devices before being used in subsequent processing stages.

[0037] Step S102: Detect pedestrians in consecutive image frames, extract pedestrian position and appearance features, and generate initial pedestrian trajectories.

[0038] In this embodiment, pedestrian appearance features include color / texture features (e.g., clothing color, pattern, local texture, etc.), depth features (i.e., high-dimensional semantic features), and pose / contour features (i.e., pedestrian skeleton keypoints or contour shape features), etc. In cross-frame tracking, these appearance features are used to address short-term occlusion or similar motion interference problems that motion prediction (e.g., Kalman filtering) cannot handle. For pedestrian position features, they can be directly generated and extracted using the center coordinates and size of the detection box in the object detection model, while pedestrian appearance features can be represented by a high-dimensional feature vector output from a pre-trained ReID model. As for generating the initial pedestrian trajectory, it can be done by binding detected pedestrian image patches with pedestrian appearance features, calculating the motion similarity between the new detection box and the existing trajectory, and then performing matching using the Hungarian algorithm.

[0039] Step S103: Dynamically track pedestrian trajectories based on multimodal motion states to generate real-time updated pedestrian tracking trajectories.

[0040] Considering that a single modality is insufficient to handle complex scenarios such as occlusion and changes in lighting, this embodiment of the application can dynamically track pedestrian trajectories based on multimodal motion states, generating real-time updated pedestrian tracking trajectories, thereby solving the trajectory breakage problem caused by tracking mismatch in the prior art. The aforementioned multimodal motion states mainly include position prediction error, appearance similarity, and scene geometric constraints, etc. Among them, position prediction error reflects kinematic continuity; for example, when the smoothness assumption fails, it is an occlusion signal. Appearance similarity can solve the problem of target identity loss caused by short-term occlusion, and scene geometric constraints can use physical space limitations (such as inaccessible areas) to correct erroneous tracking areas.

[0041] In general, this application utilizes computer vision technology (deepsort) to track pedestrian trajectories and generate real-time updated pedestrian tracking trajectories. The deepsort detector detects bounding boxes and generates detection targets. Kalman filtering is then used to predict the trajectory from the previous frame in the current frame. The cost matrices for the tracked and detected targets are calculated using Mahalanobis distance based on appearance features. Cascaded matching and cross-union matching are then performed on the tracked and detected targets sequentially, finally obtaining all matching pairs, unmatched tracked targets, and unmatched detected targets in the current frame. For each successfully matched tracked target, its corresponding detected target is used for updating, and unmatched tracked and detected targets are processed. Here, unmatched tracked targets may include occluded detected targets that are not detected. Conventional processing methods would discard these, but this application's solution determines whether the detected target has moved beyond the image boundary based on its distance from the image boundary. If not, it is considered occluded and marked as masked, and Kalman filtering prediction continues until a matching detected target appears or the detected target moves beyond the image boundary. It should be noted that the detection target mentioned in the above embodiments refers to the pedestrian / vehicle target instance identified in the current frame image by the target detection model (e.g., YOLO, Faster R-CNN), which includes location, confidence level, and may also include appearance features. The tracking target refers to the tracking target instance maintained across frames, which includes information such as motion state, historical trajectory, appearance features, occlusion state, and lifespan. As can be seen from the above description, the detection target mainly refers to the detection result of the current frame, i.e., the new target, while the tracking target mainly refers to the tracking state of historical frames, i.e., the persistent target.

[0042] To address the trajectory breakage issue in scenarios with prolonged occlusion and prevent mismatches caused by brief occlusion or detection noise, as an embodiment of this application, dynamic tracking of pedestrian trajectories based on multimodal motion states is performed to generate real-time updated pedestrian tracking trajectories. This can be achieved through steps S1031 to S1034, as detailed below:

[0043] Step S1031: Predict the state of the pedestrian position using a Kalman filter to generate a predicted bounding box and covariance matrix.

[0044] Kalman filtering utilizes the state equations of a linear system to make an optimal estimate of the system state using observed input and output data. Kalman's recursive optimal estimation theory employs a state-space description method, enabling it to handle multidimensional and non-stationary stochastic processes. Kalman filtering mainly consists of two stages: prediction and update. Prediction involves...

[0045]

[0046] During the prediction process, this application obtains the predicted values ​​based on the model. Covariance Update, that is:

[0047]

[0048] in, This represents the final state of the vehicle, i.e., the predicted vehicle state. Values ​​to be detected A and B are transformation matrices, y k The vehicle state prediction is obtained by multiplying the transformation matrix C by the vehicle state prediction value. K k This is called Kalman gain.

[0049] Step S1032: Based on the predicted bounding box and covariance matrix, calculate the Mahalanobis distance between the detection result of the current frame and the historical trajectory, and generate a comprehensive matching cost matrix by combining the cosine similarity of appearance features.

[0050] In this embodiment, the deepsort detector uses a simple convolutional neural network to extract the appearance features of the detected objects. After each detection and tracking, the object's appearance features are extracted and saved. In each subsequent step, a similarity calculation is performed between the appearance features of the detected object in the current frame and the previously stored appearance features. Mahalanobis distance is used to calculate both trajectory and appearance features. It avoids the risk of different variances in data features encountered in Euclidean distance calculations. A covariance matrix is ​​added to the calculation to normalize the variance, making the so-called "distance" more consistent with the data features and practical meaning. The formula for calculating the distance d(x,y) is as follows:

[0051]

[0052] The cosine distance is used to calculate the difference in direction, dist(x,y), and its calculation formula is as follows:

[0053]

[0054] Mahalanobis distance calculation, which yields trajectory features, provides potential information for object localization and is highly effective in short-term prediction. Cosine distance calculation, which yields appearance features, is effective in restoring the target's ID after prolonged occlusion, reducing ID switching frequency. A weighted average is then used to combine these two features to generate a comprehensive matching cost matrix.

[0055] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0056] Where, d (1)Let d be the Mahalanobis distance. (2) Let λ be the cosine distance and λ be the weighting coefficient.

[0057] Step S1033: Based on the comprehensive matching cost matrix, a cascaded matching strategy is adopted to prioritize the association of short-term mismatched trajectories, and cross-frame motion continuity verification is performed on the unmatched trajectories.

[0058] When a target is occluded for an extended period, the uncertainty of Kalman filter predictions increases significantly, and the observability of the state space decreases considerably. If two trackers compete for the matching right of the same detection result, the track with the longer occlusion time often has a smaller Mahalanobis distance, making the detection result more likely to be associated with that track. This undesirable effect often disrupts tracking persistence. Theoretically, assuming the covariance matrix is ​​initially a normal distribution, continuous prediction without updates will cause the variance of this normal distribution to increase. Points far from the mean Euclidean distance may acquire the same Mahalanobis distance value as points previously closer in the distribution. Therefore, cascade matching prioritizes targets that appear more frequently. Specifically, a tracker is assigned to each detector, and each tracker has a `time_since_update` parameter. If a tracker completes a match and updates, this parameter is reset to 0; otherwise, it is incremented by 1. In essence, cascade matching is matching with different priorities. In cascade matching, the tracker priority is determined by this parameter, with tracks with smaller parameters matching first and those with larger parameters matching later. Based on the above facts, in order to prioritize high-probability associated targets, avoid computational redundancy in global matching, and use spatial distribution features in low-priority queues to fill in missing matches and solve the trajectory intersection problem in dense scenes, after obtaining the comprehensive matching cost matrix, a cascaded matching strategy can be adopted to prioritize the association of short-term mismatched trajectories based on the comprehensive matching cost matrix.

[0059] Specifically, based on the comprehensive matching cost matrix, the cascaded matching strategy for prioritizing the association of short-term mismatched trajectories can be implemented as follows: Matching priorities are dynamically adjusted according to the trajectory mismatch duration, allocating high-priority matching queues to trajectories with mismatch durations less than a first threshold; in the high-priority queue, the Hungarian algorithm is used for bidirectional optimal matching to generate a first matching result and a set of unmatched trajectories; the set of unmatched trajectories and trajectories with mismatch durations exceeding the first threshold are transferred to a low-priority queue; in the low-priority queue, a secondary matching is performed based on a region density clustering algorithm to generate the final matching result. As can be seen from the above embodiments, on the one hand, priority hierarchical management reduces computational complexity while supplementing matching with density clustering, balancing real-time performance and accuracy; on the other hand, spatial distribution feature constraints reduce the identity exchange error rate in densely populated pedestrian areas.

[0060] Step S1034: If the continuous mismatch trajectory exceeds the preset number of frames, start the occlusion recovery mechanism for the trajectory with continuous mismatch exceeding the preset number of frames, and correct the tracking state through the prediction of the scene edge position.

[0061] Specifically, starting the occlusion recovery mechanism and correcting the tracking state through the prediction of the scene edge position includes the input of preconditions and steps S1 to S4, which are described as follows:

[0062] Preconditions: 1) The trajectory with continuous mismatch: time_since_update (i.e., the number of frames of the continuous mismatch trajectory) > the preset threshold N (e.g., N = 5); 2) The boundary range of the current frame image: [x_min, y_min, x_max, y_max]; 3) The position of the last valid detection box of this trajectory: last_bbox = (x, y, w, h).

[0063] Step S1: Determine whether the trajectory may leave the scene, including: 1) Calculate the distance from the trajectory to the image boundary: the center coordinates of the trajectory prediction box (output of Kalman filter): (x_pred, y_pred); the boundary safety distance threshold: d_threshold (e.g., 10 pixels); 2) Decision logic: If x_pred < x_min + d_threshold or x_pred > x_max - d_threshold, the target may leave from the left and right boundaries; if y_pred < y_min + d_threshold or y_pred > y_max - d_threshold, the target may leave from the upper and lower boundaries; otherwise, the target is occluded.

[0064] Step S2: Prediction of the trajectory in the occlusion state, including: 1) Continuously use Kalman filter to predict the position: update the prediction state for each frame, but do not update the detection box (no matching detection); the confidence of the prediction box decays over time (e.g., covariance matrix inflation) 2) Prediction termination conditions include: the trajectory prediction box completely exceeds the image boundary, that is, the target leaves the scene, terminate tracking, or re-match to the detection box, exit the occlusion state, or the occlusion lasts for more than the maximum number of frames (e.g., N_max = 30 frames), the target tracking fails, terminate the trajectory.

[0065] Step S3: Cross-frame motion continuity verification, including: 1) Motion consistency verification: For the predicted trajectory during occlusion, check whether its motion direction is consistent with the historical trajectory before mismatch: calculate the historical average velocity vector vhist; calculate the current predicted velocity vector vpred; if cos(vhist, vpred) < θthresh (e.g., 0.7), determine that the motion is abnormal and terminate the trajectory; 2) Scene geometric constraint: If the prediction box enters an impassable area (e.g., building area), forcefully terminate the trajectory.

[0066] Step S4: Correct the state after occlusion recovery, including: when a detection box is rematched, calculate the Mahalanobis distance and appearance similarity between the detection box and the predicted box. If the match is successful, update the trajectory state using the detection box, reset time_since_update = 0, and exit the "occlusion" state.

[0067] As can be seen from steps S1031 to S1034 of the above embodiments, by cascaded matching to preferentially associate short-term mismatched trajectories, the loss of trajectory identity caused by short-term occlusion is reduced, and the tracking state is corrected based on the prediction of the scene edge position, solving the problem of tracking termination caused by the long-term disappearance of the target in the prior art.

[0068] Step S104: Construct a spatio-temporal interaction graph network to predict the future trajectory of pedestrians and generate trajectory confidence.

[0069] In the embodiment of the present application, the spatio-temporal interaction graph network is a graph structure used to model the spatial and temporal dependencies in pedestrian trajectory prediction. It combines the spatial interactions of pedestrians (such as the relative positions and interactions between pedestrians) and temporal dynamics (such as the movement history of pedestrians) to more comprehensively capture the complexity of pedestrian behavior. Briefly, the spatio-temporal interaction graph network predicts the future trajectory of pedestrians and generates trajectory confidence by fusing the spatio-temporal features of the historical trajectories of pedestrians and their interaction relationships with vehicles. As an embodiment of the present application, constructing the spatio-temporal interaction graph network can be achieved through steps S1041 to S1043, which are described in detail as follows:

[0070] Step S1041: Divide the historical trajectories of pedestrians into multiple overlapping spatio-temporal patches according to time windows.

[0071] If the total time length to be analyzed is represented by T obs and the duration of each time window is represented by L, and the sliding interval of the time window is represented by the step size S, then the number K of spatio-temporal patches into which the historical trajectories of pedestrians are divided can be expressed as:

[0072]

[0073] where the symbol represents rounding down. It should be noted that in the embodiment of the present application, S and L can be designed such that S < L. This design can ensure the information continuity between adjacent time windows and avoid key interaction events being split by the boundaries of time windows. Further, each spatio-temporal patch contains a local region of the input data, and these regions can be continuous or overlapping. The main purpose of spatio-temporal patches is to decompose large chunks of data into smaller units for more efficient processing and analysis.

[0074] Step S1042: Construct a bigraph structure of node graph and edge graph within each spatiotemporal patch.

[0075] In this embodiment of the application, the node graph constructed within the spatiotemporal patch integrates speed, direction angle, and vehicle distance features, while the edge graph transforms the original edges into nodes through a first-order boundary operator to construct the edge graph adjacency relationship.

[0076] In this embodiment, each node of the node graph is generated as follows: the pedestrian velocity vector at each time step is normalized to generate a direction encoding matrix; the direction encoding matrix is ​​nonlinearly mapped to Euclidean space coordinates using a multilayer perceptron to generate a high-dimensional embedding feature; vehicle position information is injected into the node graph as an external node, and the repulsive force field weights of pedestrian and vehicle nodes are calculated based on the high-dimensional embedding feature; the high-dimensional embedding feature and the repulsive force field weights are concatenated into a node representation and input into the subsequent graph attention network.

[0077] As mentioned earlier, the edge graph transforms the original edges into nodes using a first-order boundary operator, constructing the edge graph adjacency relationship. In other words, the edge graph is constructed by converting the original edges into edge graph nodes using a first-order boundary operator, and constructing the edge graph adjacency relationship based on the consistency of pedestrian movement trends. Here, edge graph nodes represent the interaction relationships between pedestrians, and the adjacency relationship reflects the spatiotemporal evolution of the interaction intensity. The edge graph adjacency relationship update is achieved in the following way: calculating the topological features of the edge graph nodes using a Hodge Laplacian convolution kernel; and dynamically adjusting the weight distribution of the adjacency matrix according to the curvature change rate of the pedestrian trajectory.

[0078] Specifically, in order to construct the edge graph, this application uses the first-order boundary operator B1 to convert it into the corresponding undirected edge graph. Where, ε k This represents the node characteristics in the edge graph. This operator represents a new adjacency relationship. It defines the connection between nodes in the original graph (i.e., the edge in the original graph) as a node in the new graph, and creates an edge between these new nodes if they share a common node in the original graph.

[0079] To analyze and update the feature propagation of each edge graph, this application uses the first-order Hodge-Laplacian Laguerre operator to analyze and learn the dynamics within these edge graphs, namely:

[0080]

[0081] Where L1 represents the first-order Hodge Laplace operator, Used to capture and enhance edge relationships, focusing on direct interactions, B2 is typically associated with higher-dimensional structures. This application performs edge convolution by adapting Hodge-Laplace-Laguerre convolution to obtain high-dimensional edge embeddings for the k-th edge graph.

[0082]

[0083] Where h1 is an L1-based spectral filter used to update the edge features ε k θ j Γ represents the learnable parameters. j (·) denotes a Laguerre polynomial function.

[0084] Finally, the embedded node features of the k-th unified spatiotemporal interaction graph are obtained. Sum of edge features Subsequently, this application utilizes a fused graph convolutional network to integrate node and edge embeddings, enhancing the understanding of graph dynamics. Specifically, this application incorporates normalized edge embeddings as weights into the aggregation process of the GCN.

[0085]

[0086] Each node i on the spatiotemporal interaction graph network is defined as follows:

[0087]

[0088] Where θ(·) and φ(·) are linear transformations of node and edge features, and σ(·) is the activation function.

[0089] Step S1043: Dynamically calculate the interaction weights between nodes in the spatiotemporal interaction graph network using the graph attention mechanism.

[0090] In a spatiotemporal interaction graph network, the interaction weights between nodes, also known as interaction strengths, are a concept used to quantify the strength of interactions between pedestrians. They reflect spatial relationships, social relationships, or other forms of mutual influence between pedestrians. The calculation of interaction strength or interaction weights is typically based on the pedestrian's position, speed, direction, or other characteristics, used to model the dynamic relationships between pedestrians. In one embodiment of this application, dynamically calculating the interaction weights between nodes in the spatiotemporal interaction graph using a graph attention mechanism can be achieved by: generating initial spatial association weights based on the cosine of the angle between the pedestrian's movement directions; generating temporal consistency weights by statistically analyzing the Pearson correlation coefficient of the pedestrian's speed change rate using a time sliding window; and performing gating fusion on the spatial association weights and temporal consistency weights to generate the final interaction weight matrix.

[0091] In the above embodiments, the generation of temporal consistency weights by statistically analyzing the Pearson correlation coefficient of pedestrian speed change rates using a time-sliding window mainly includes stages such as extracting speed change sequences, dividing the time-sliding window, calculating the Pearson correlation coefficient, and aggregating the time window results. Specifically, for each pedestrian, the speed sequence within the time-sliding window is extracted, for example, the speed values ​​per second v1, v2, ..., v... T Calculate the rate of change of velocity between adjacent time steps, i.e., Δv t =v t+1 -v t This yields the sequence Δv1, Δv2, ..., Δv T-1 Dividing the time window can be done by segmenting the velocity change rate sequence into multiple overlapping subsequences according to a fixed-length time window (e.g., a window length of 5 seconds and a step size of 2 seconds). When calculating the Pearson correlation coefficient, specifically, for any two pedestrians (denoted as A and B) within the same time window, the Pearson correlation coefficient of their velocity change rate sequences can be calculated. Where cov is the covariance, σ is the standard deviation, and r AB The value ∈ [-1,1] indicates that the larger the value, the more consistent the trend of speed change. The aggregated time window result can be obtained by averaging (or weighted averaging) the Pearson coefficients within all time windows to obtain the time consistency weight between the two lines. For example: r in window 1 AB =0.8, r in window 2 AB =0.6, then

[0092] In the above embodiments, the gating fusion of spatial correlation weights and temporal consistency weights to generate the final interaction weight matrix mainly includes stages such as weight normalization, gating function design, weighted fusion, and generation of the interaction weight matrix. The normalized weights are obtained by gating the spatial correlation weights... (Based on direction cosine) and time consistency weight Normalized to the range [0, 1], when designing the gating function, a gating parameter g∈[0, 1] can be defined to control the fusion ratio of the two types of weights. g can be obtained through training or preset according to the scenario (e.g., dense areas rely more on spatial weights). Weighted fusion can be achieved by dynamically mixing the two types of weights according to the gating parameter to generate the final interactive weights. For example, if g = 0.6, the final weights will be biased towards spatial correlation (60% spatial weight + 40% temporal weight). As for generating the interaction weight matrix, this can be achieved by repeating the above steps for each pair of pedestrians to form matrix W. final Matrix elements

[0093] In another embodiment, each node of the node graph The definition is as follows:

[0094]

[0095] Where Θ(·) is the transformation function, Γ(·) and σ(·) represent the activation functions, N(·) is the neighborhood set of node i, and a T The learnable parameter is the attention coefficient. This represents the interaction weights between two nodes i and j. During training, these interaction weights are dynamically updated to reflect the importance of each node's contribution to its neighbors.

[0096] As an embodiment of this application, predicting the future trajectory of a pedestrian can be achieved through steps S'1041 to S'1043, as detailed below:

[0097] Step S'1041: In the Transformer encoder, initialize the learnable future trajectory placeholders, concatenate them with historical trajectory features along the time dimension, and generate the concatenated feature tensor.

[0098] In this embodiment, a trajectory predictor based on a Transformer encoder is employed. By encoding future trajectories into learnable parameters and concatenating them with historical trajectories, past and future information are modeled in a unified manner. This allows the model to fully utilize global temporal dependencies, thereby achieving more accurate predictions. This application embeds the graph output from the spatiotemporal interaction graph network into H... k Stack all patches together to obtain the integrated feature representation H as follows:

[0099]

[0100] This application performs time-averaged pooling on the L channel (the L channel refers to the portion of time series data that is related to a specific time step or feature dimension when processing time series data), and outputs... As historical input markers, this application then initializes a learnable future trajectory placeholder, forming the filled future markers. Adjust the prediction time step T of these markers pred To match the prediction range of this embodiment. Finally, the input labels of the Transformer encoder are formed by concatenating the learned historical input labels H and the padded future labels F, resulting in the concatenated feature tensor. Where N represents the number of pedestrians and D represents the feature dimension. In the above embodiment, learnable future trajectory placeholders are used to initialize the output of future time steps in the prediction task. These placeholders are progressively updated and optimized during training to better represent future trajectories, thereby improving the model's predictive ability.

[0101] Step S'1042: Apply learnable additive position embeddings to the concatenated feature tensors to generate temporal position encoded features.

[0102] Furthermore, through a learnable additive position embedding These markers are then enhanced and applied throughout the concatenated sequence to generate temporal position encoded features. Subsequently, the Transformer encoder encodes features based on temporal location. The process is then performed. It should be noted that the learnable additive positional embedding P mentioned above helps the model capture positional information by adding a learnable embedding vector to each time step or position, thereby better understanding the temporal or spatial dependencies in the sequence.

[0103] Step S'1043: Input the temporal position encoding features into the multi-head self-attention mechanism to generate hidden state features, and generate multimodal trajectory distribution parameters based on the hidden state features.

[0104] The representation used to generate the predicted sequence, namely:

[0105]

[0106] Here, [·||·] represents a concatenation operation along the time dimension. It is important to note that... This represents the complete output of the encoder, with a length of K+T. pred Only the last T pred Each time step is used as a representation of the predicted trajectory, corresponding to the filled future label F. This embodiment employs a binary Gaussian distribution loss function Li. prediction To optimize trajectory prediction:

[0107]

[0108] in, and These are the mean and variance of a bivariate Gaussian distribution, while This represents the correlation coefficient.

[0109] Furthermore, to optimize the model specifically for false positives / false negatives, improve the scenario adaptability of anomaly detection, and avoid difficulties in edge device deployment due to model bloat, the method in the above embodiments also includes dynamically adjusting the network structure based on node contribution. Specifically, this involves online updating of the spatiotemporal interaction graph network, including: real-time collection of trajectory data from false positives and false negatives to construct an incremental learning sample set; transferring the offline training model parameters of the spatiotemporal interaction graph network to the online lightweight model using knowledge distillation; sparsifying the spatiotemporal interaction graph network in the online lightweight model using a dynamic weight pruning strategy; and deploying the updated online lightweight model to edge computing nodes while simultaneously updating the hierarchical warning thresholds. The sample set includes pedestrian tracking trajectories and prediction errors in abnormal scenarios, and the pruning strategy dynamically adjusts the network structure based on node contribution. As can be seen from the above embodiments' technical solution for online updating of the spatiotemporal interaction graph network, not only is the anomaly detection performance continuously optimized through incremental learning to adapt to the time-varying characteristics of traffic scenarios, but dynamic pruning also reduces the number of model parameters, ensuring efficient deployment of the lightweight model on edge devices.

[0110] In the above embodiments, the sparsification of the spatiotemporal interaction graph network in the online lightweight model using a dynamic weight pruning strategy can be achieved by: statistically analyzing the variance of the graph attention weights in the spatiotemporal interaction graph network over time, and marking nodes with variance values ​​below a first threshold as redundant nodes; performing gradient masking on redundant nodes to suppress their parameter updates during backpropagation; removing the redundant node and its connecting edges from the spatiotemporal interaction graph network when the frequency of its occurrence within a continuous time window exceeds a second threshold; and recalculating the attention weight distribution of the remaining nodes based on the network topology after node removal.

[0111] As can be seen from the technical solution of step S104 in the above embodiment, by constructing a spatiotemporal interaction graph network and integrating the spatiotemporal characteristics of pedestrian historical trajectories and their interaction with vehicles, the accuracy of future trajectory prediction is improved, providing a reliable confidence basis for anomaly determination.

[0112] Step S105: Determine the level of pedestrian behavior abnormality based on the spatiotemporal deviation between the pedestrian's future trajectory and the pedestrian tracking trajectory, the trajectory confidence, and the abrupt change characteristics of the interaction relationship.

[0113] Considering that a single judgment dimension cannot distinguish between real anomalies and sensor noise, this application embodiment can determine the level of pedestrian behavior anomaly based on the spatiotemporal deviation between the pedestrian's future trajectory and the pedestrian tracking trajectory, trajectory confidence, and abrupt change characteristics of interaction relationships. This addresses the high false alarm rate problem caused by fixed threshold rules in existing technologies. The spatiotemporal deviation reflects the macroscopic deviation between the predicted trajectory and the actual trajectory, while the judgment weight can be adjusted based on the trajectory confidence to avoid false alarms triggered by low-confidence predictions. The abrupt change characteristics of interaction relationships can capture behavioral anomalies in sudden scenarios such as vehicle approach.

[0114] As one embodiment of this application, determining the pedestrian behavior anomaly level based on the spatiotemporal deviation between the pedestrian's future trajectory and the pedestrian tracking trajectory, trajectory confidence, and abrupt change characteristics of the interaction relationship can be as follows: The Hausdorff distance between the predicted trajectory and the pedestrian tracking trajectory is calculated as the spatiotemporal deviation; the number of abrupt changes in the trajectory curvature rate is counted as the behavior anomaly coefficient; the spatiotemporal deviation is dynamically weighted and corrected according to the trajectory confidence; and a comprehensive anomaly score is output by fusing the weighted spatiotemporal deviation and the behavior anomaly coefficient through a logistic regression model. Specifically, the dynamic weighted correction of the spatiotemporal deviation based on the trajectory confidence can be as follows: when the trajectory confidence is below a first threshold, the spatiotemporal deviation is multiplied by a first gain coefficient; when the trajectory confidence is above a second threshold, the spatiotemporal deviation is smoothed using a sliding window averaging method; and the range of the first gain coefficient is dynamically adjusted according to vehicle density.

[0115] Step S106: When the level of abnormal pedestrian behavior exceeds the preset threshold, a graded early warning signal is triggered and avoidance guidance information is generated.

[0116] Specifically, when the pedestrian behavior anomaly level exceeds a preset threshold, triggering a graded warning signal can be as follows: when the pedestrian behavior anomaly level is at a preset first level and the trajectory confidence is higher than a preset threshold, a visual warning signal is triggered and a risk area is marked on an electronic map; when the pedestrian behavior anomaly level is at a preset second level or the trajectory confidence is lower than a preset threshold, an audible and visual alarm device is activated and a pedestrian avoidance path plan is generated; when the pedestrian behavior anomaly level is at a preset third level, an emergency braking command is sent to nearby vehicles and the traffic signal control system is activated. In the above embodiment, generating a pedestrian avoidance path plan includes: generating a dynamic safety zone boundary based on pedestrian movement trend prediction; calculating the optimal avoidance distance between the vehicle's travel path and the safety zone boundary using a potential field algorithm, generating a candidate set of avoidance paths; encoding the avoidance path into a vehicle control command based on the optimal avoidance distance and vehicle dynamics constraints; verifying the compliance of the vehicle control command with real-time traffic rules, and selecting the final execution command. The dynamic safety zone boundary is dynamically adjusted according to the pedestrian's position and speed, and the commands include deceleration gradient, steering angle, and braking timing.

[0117] As can be seen from the technical solutions of steps S105 and S106 in the above embodiments, based on the spatiotemporal deviation between the predicted trajectory and the tracked trajectory, the trajectory confidence and the abrupt change characteristics of the interaction relationship, the abnormality level is dynamically determined and a graded early warning signal is triggered instead of a single alarm mechanism. This avoids the interference or response delay caused by the "one-size-fits-all" alarm in the prior art, and significantly improves the adaptability of abnormality detection and the timeliness of alarm.

[0118] Furthermore, the method in the above embodiments also includes cross-device collaborative computing, that is, assigning pedestrian detection and trajectory tracking tasks to edge computing nodes for execution; uploading spatiotemporal map construction and trajectory prediction tasks to a cloud server for execution; and coordinating data exchange between edge nodes and the cloud through an asynchronous communication protocol to achieve dynamic load balancing. Specifically, coordinating data exchange between edge nodes and the cloud through an asynchronous communication protocol to achieve dynamic load balancing can be achieved by: setting up a circular buffer at the edge node to store image frames to be processed; triggering data compression when the buffer occupancy rate exceeds a threshold; performing differential encoding on the compressed data stream, transmitting only motion vector information between adjacent frames; deploying a decoder in the cloud to reconstruct the complete data stream; and achieving multi-node data fusion through timestamp alignment.

[0119] From the above appendix Figure 1 The example of the pedestrian abnormal behavior detection method in a mixed pedestrian-vehicle environment demonstrates that, on the one hand, by fusing multimodal motion states based on position prediction error, appearance similarity, and scene geometric constraints, robust tracking trajectories are generated in occluded or complex motion scenarios, reducing the risk of misjudgment due to trajectory breakage. On the other hand, by constructing a spatiotemporal interaction graph network, the spatiotemporal characteristics of pedestrian historical trajectories and their interaction relationships with vehicles are fused, improving the accuracy of future trajectory prediction and providing a reliable confidence basis for anomaly determination. Thirdly, based on the spatiotemporal deviation between the predicted trajectory and the tracking trajectory, trajectory confidence, and abrupt changes in interaction relationships, the anomaly level is dynamically determined and graded early warning signals are triggered, significantly improving the adaptability and timeliness of anomaly detection. In summary, the technical solution of this application can effectively improve tracking continuity, prediction accuracy, and early warning timeliness.

[0120] Please see the appendix Figure 2 This application provides a pedestrian abnormal behavior detection device in a mixed pedestrian and vehicle environment. The device may include an acquisition module 201, a first generation module 202, a second generation module 203, a prediction module 204, a judgment module 205, and a triggering module 206, as detailed below:

[0121] The acquisition module 201 is used to acquire continuous image frames of mixed pedestrian and vehicle traffic scenes in real time;

[0122] The first generation module 202 is used to detect pedestrians in consecutive image frames, extract pedestrian positions and appearance features, and generate initial pedestrian trajectories;

[0123] The second generation module 203 is used to dynamically track pedestrian trajectories based on multimodal motion states and generate real-time updated pedestrian tracking trajectories.

[0124] Prediction module 204 is used to construct a spatiotemporal interaction graph network, predict the future trajectory of pedestrians, and generate trajectory confidence scores.

[0125] The judgment module 205 is used to determine the abnormality level of pedestrian behavior based on the spatiotemporal deviation between the future trajectory of the pedestrian and the pedestrian tracking trajectory, the trajectory confidence, and the abrupt change characteristics of the interaction relationship.

[0126] The triggering module 206 is used to trigger a graded warning signal and generate avoidance guidance information when the abnormal level of pedestrian behavior exceeds a preset threshold.

[0127] From the above appendix Figure 2 As illustrated by the pedestrian abnormal behavior detection device in a mixed pedestrian-vehicle environment, on the one hand, by fusing multimodal motion states based on position prediction error, appearance similarity, and scene geometric constraints, robust tracking trajectories are generated in occluded or complex motion scenarios, reducing the risk of misjudgment due to trajectory breakage. On the other hand, by constructing a spatiotemporal interaction graph network, the spatiotemporal characteristics of pedestrian historical trajectories and their interaction relationships with vehicles are fused, improving the accuracy of future trajectory prediction and providing a reliable confidence basis for anomaly determination. Thirdly, based on the spatiotemporal deviation between the predicted trajectory and the tracking trajectory, trajectory confidence, and abrupt changes in interaction relationships, the anomaly level is dynamically determined and graded early warning signals are triggered, significantly improving the adaptability and timeliness of anomaly detection. In summary, the technical solution of this application can effectively improve tracking continuity, prediction accuracy, and early warning timeliness.

[0128] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for detecting abnormal pedestrian behavior in a mixed pedestrian-vehicle environment. When the processor 30 executes the computer program 32, it implements the steps in the above-described embodiment of the method for detecting abnormal pedestrian behavior in a mixed pedestrian-vehicle environment, for example... Figure 1 The steps S101 to S106 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the acquisition module 201, the first generation module 202, the second generation module 203, the prediction module 204, the judgment module 205, and the triggering module 206 are shown.

[0129] For example, the computer program 32 of the pedestrian abnormal behavior detection method in a mixed pedestrian and vehicle traffic environment mainly includes: real-time acquisition of continuous image frames of the mixed pedestrian and vehicle traffic scene; detection of pedestrians in the continuous image frames, extraction of pedestrian position and appearance features, and generation of initial pedestrian trajectory; dynamic tracking of pedestrian trajectory based on multimodal motion state, generating real-time updated pedestrian tracking trajectory; construction of spatiotemporal interaction graph network, prediction of future pedestrian trajectory and generation of trajectory confidence; determination of pedestrian behavior abnormality level based on the spatiotemporal deviation between the future pedestrian trajectory and the pedestrian tracking trajectory, trajectory confidence, and abrupt change characteristics of interaction relationship; and triggering a graded warning signal and generating avoidance guidance information when the pedestrian behavior abnormality level exceeds a preset threshold. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of computer program 32 in electronic device 3. For example, the computer program 32 can be divided into the functions of an acquisition module 201, a first generation module 202, a second generation module 203, a prediction module 204, a judgment module 205, and a trigger module 206 (a module in the virtual device). The specific functions of each module are as follows: the acquisition module 201 is used to acquire continuous image frames of mixed pedestrian and vehicle traffic scenes in real time; the first generation module 202 is used to detect pedestrians in the continuous image frames, extract pedestrian position and appearance features, and generate initial pedestrian trajectories; the second generation module 203 is used to dynamically track pedestrian trajectories based on multimodal motion states and generate real-time updated pedestrian tracking trajectories; the prediction module 204 is used to construct a spatiotemporal interaction graph network, predict future pedestrian trajectories, and generate trajectory confidence; the judgment module 205 is used to determine the abnormality level of pedestrian behavior based on the spatiotemporal deviation between the future pedestrian trajectory and the pedestrian tracking trajectory, trajectory confidence, and abrupt change characteristics of interaction relationships; the trigger module 206 is used to trigger a graded warning signal and generate avoidance guidance information when the abnormality level of pedestrian behavior exceeds a preset threshold.

[0130] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0131] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0132] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0134] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can also be implemented by a computer program instructing related hardware. The computer program for the pedestrian abnormal behavior detection method in a mixed pedestrian and vehicle environment can be stored in a storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned method embodiments, namely, acquiring continuous image frames of the mixed pedestrian and vehicle scene in real time; detecting pedestrians in the continuous image frames, extracting pedestrian position and appearance features, and generating an initial pedestrian trajectory; dynamically tracking the pedestrian trajectory based on multimodal motion state, generating a real-time updated pedestrian tracking trajectory; constructing a spatiotemporal interaction graph network, predicting the future trajectory of the pedestrian and generating trajectory confidence; determining the pedestrian behavior abnormality level based on the spatiotemporal deviation between the future pedestrian trajectory and the pedestrian tracking trajectory, trajectory confidence, and abrupt change characteristics of the interaction relationship; and triggering a graded warning signal and generating avoidance guidance information when the pedestrian behavior abnormality level exceeds a preset threshold. Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media do not include electrical carrier signals and telecommunication signals.

[0140] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this invention.

Claims

1. A method for detecting abnormal pedestrian behavior in a mixed pedestrian and vehicle environment, characterized in that, The method includes: Real-time acquisition of continuous image frames of mixed pedestrian and vehicle traffic scenes; Pedestrians are detected in the continuous image frames, their positions and appearance features are extracted, and initial pedestrian trajectories are generated. Dynamic tracking of pedestrian trajectories based on multimodal motion states generates real-time updated pedestrian tracking trajectories. This dynamic tracking includes: predicting pedestrian position state using a Kalman filter to generate a predicted bounding box and covariance matrix; calculating the Mahalanobis distance between the current frame detection result and historical trajectories based on the predicted bounding box and covariance matrix, and generating a comprehensive matching cost matrix by combining the cosine similarity of appearance features; using a cascaded matching strategy to prioritize associating short-term mismatched trajectories based on the comprehensive matching cost matrix, and performing cross-frame motion continuity verification on unmatched trajectories; if a continuously mismatched trajectory exceeds a preset number of frames, an occlusion recovery mechanism is activated for the trajectory that has been continuously mismatched for more than a preset number of frames, correcting the tracking state through scene edge position prediction. A spatiotemporal interaction graph network is constructed to predict the future trajectory of pedestrians and generate trajectory confidence. The construction of the spatiotemporal interaction graph network includes: dividing the historical trajectory of pedestrians into multiple overlapping spatiotemporal patches according to time windows; constructing a bi-graph structure of node graph and edge graph in each spatiotemporal patch; and dynamically calculating the interaction weights between nodes in the spatiotemporal interaction graph network through a graph attention mechanism. The level of pedestrian behavior abnormality is determined based on the spatiotemporal deviation between the future trajectory of the pedestrian and the tracking trajectory of the pedestrian, the trajectory confidence, and the abrupt change characteristics of the interaction relationship. When the level of abnormal pedestrian behavior exceeds a preset threshold, a graded early warning signal is triggered and avoidance guidance information is generated.

2. The method for detecting abnormal pedestrian behavior in a mixed pedestrian and vehicle environment according to claim 1, characterized in that, The step of prioritizing the association of short-term mismatch trajectories based on the comprehensive matching cost matrix includes: The matching priority is dynamically adjusted based on the trajectory mismatch duration, and a high-priority matching queue is assigned to trajectories with mismatch time less than the first threshold. In the high-priority queue, the Hungarian algorithm is used for bidirectional optimal matching to generate the first matching result and a set of unmatched trajectories. The set of unmatched trajectories and trajectories whose mismatch time exceeds the first threshold are transferred to a low-priority queue. In the low-priority queue, a secondary matching is performed based on the region density clustering algorithm to generate the final matching result.

3. The method for detecting abnormal pedestrian behavior in a mixed pedestrian and vehicle environment according to claim 1, characterized in that, The dynamic calculation of the interaction weights between nodes in the spatiotemporal interaction graph network through the graph attention mechanism includes: Initial spatial association weights are generated based on the cosine value of the angle between the pedestrian's direction of movement; The time consistency weight is generated by calculating the Pearson correlation coefficient of the rate of change of pedestrian speed through a time sliding window. The spatial correlation weights and temporal consistency weights are gated and fused to generate the final interaction weight matrix.

4. The method for detecting abnormal pedestrian behavior in a mixed pedestrian and vehicle environment according to claim 1, characterized in that, The predicted future trajectory of pedestrians includes: In the Transformer encoder, learnable future trajectory placeholders are initialized and concatenated with historical trajectory features along the time dimension to generate a concatenated feature tensor. Learnable additive position embeddings are applied to the concatenated feature tensor to generate temporal position encoded features; The temporal position encoding features are input into a multi-head self-attention mechanism to generate hidden state features, and multimodal trajectory distribution parameters are generated based on the hidden state features.

5. The method for detecting abnormal pedestrian behavior in a mixed pedestrian and vehicle environment according to claim 1, characterized in that, Also includes: Trajectory data of false alarms and missed alarms are collected in real time to construct an incremental learning sample set, which includes pedestrian tracking trajectories and prediction errors in abnormal scenarios. The offline training model parameters of the spatiotemporal interaction graph network are transferred to the online lightweight model using knowledge distillation technology. A dynamic weight pruning strategy is used to sparsify the spatiotemporal interaction graph network in the online lightweight model. The pruning strategy dynamically adjusts the network structure based on the node contribution. The updated online lightweight model is deployed to edge computing nodes, and the hierarchical early warning thresholds are updated synchronously.

6. A pedestrian abnormal behavior detection device in a mixed pedestrian and vehicle environment, characterized in that, The device includes: The acquisition module is used to acquire continuous image frames in real time of mixed pedestrian and vehicle traffic scenes; The first generation module is used to detect pedestrians in the continuous image frames, extract pedestrian positions and appearance features, and generate initial pedestrian trajectories; The second generation module is used to dynamically track pedestrian trajectories based on multimodal motion states, generating real-time updated pedestrian tracking trajectories. The dynamic tracking of pedestrian trajectories based on multimodal motion states includes: predicting pedestrian position state using a Kalman filter to generate a predicted bounding box and covariance matrix; calculating the Mahalanobis distance between the current frame detection result and historical trajectories based on the predicted bounding box and covariance matrix, and generating a comprehensive matching cost matrix by combining the cosine similarity of appearance features; using a cascaded matching strategy to prioritize associating short-term mismatched trajectories based on the comprehensive matching cost matrix, and performing cross-frame motion continuity verification on unmatched trajectories; if a continuously mismatched trajectory exceeds a preset number of frames, an occlusion recovery mechanism is activated for the trajectory that has been continuously mismatched for more than a preset number of frames, correcting the tracking state through scene edge position prediction. The prediction module is used to construct a spatiotemporal interaction graph network to predict the future trajectory of pedestrians and generate trajectory confidence. The construction of the spatiotemporal interaction graph network includes: dividing the historical trajectory of pedestrians into multiple overlapping spatiotemporal patches according to time windows; constructing a bi-graph structure of node graph and edge graph in each spatiotemporal patch; and dynamically calculating the interaction weights between nodes in the spatiotemporal interaction graph network through a graph attention mechanism. The determination module is used to determine the level of abnormal pedestrian behavior based on the spatiotemporal deviation between the future trajectory of the pedestrian and the tracking trajectory of the pedestrian, the trajectory confidence, and the abrupt change characteristics of the interaction relationship. The triggering module is used to trigger a graded early warning signal and generate avoidance guidance information when the pedestrian behavior abnormality level exceeds a preset threshold.

7. An electronic device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

8. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images