Single-person abnormal behavior identification method and system based on multi-modal skeleton feature fusion

Abstract

The invention discloses a single-person abnormal behavior identification method and system based on multi-modal skeleton feature fusion, and the method comprises the steps: S1, collecting continuous RGB images and infrared thermal imaging images in a monitoring video, carrying out the human body detection and key point estimation of visible light and infrared images through employing a multi-modal fusion model of YOLOv12 in combination with Transform, and constructing a single-person posture time series data set; s2, key point speed vectors are calculated for the continuous skeleton frame sequence of each target person, skeleton key point information and speed information are fused, and an action feature sequence is formed; s3, inputting the motion feature sequence into an MPED-RNN model, decomposing skeleton motion into a global displacement component and a local attitude deformation component, and performing joint coding, decoding and prediction through a dual-channel GRU network; and S4, calculating a prediction error and a reconstruction error according to a reconstruction result and a future skeleton key point prediction result, evaluating whether the current behavior deviates from a normal trajectory, and judging whether the current behavior is in an abnormal state. According to the invention, real-time identification of abnormal behaviors of a single person in a complex scene is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior detection, in particular to a single-person abnormal behavior recognition based on multi-modal skeleton feature fusion. BACKGROUND

[0002] With the continuous expansion of intelligent security systems, abnormal behavior early warning and other application scenarios, video-based abnormal behavior recognition technology has become an important part of smart city construction. In single-person behavior monitoring, how to quickly and accurately identify sudden abnormal behaviors such as falling, sitting down, and being motionless is of great significance to improve response efficiency and reduce accident risk.

[0003] Most of the current mainstream abnormal behavior recognition methods rely on single modal video information, which can achieve certain results under good lighting conditions, but still have problems such as target loss, recognition failure, false alarm rate, etc. in dim light background environment at night. Infrared thermal imaging technology can directly perceive human thermal radiation, and has high application potential in dim scenes, but the information clarity in details, textures, and contour edges is low.

[0004] In the behavior recognition task, in order to enhance the time series modeling capability, researchers have widely introduced recurrent neural network (RNN) structure to model the action sequence in recent years. Skeleton-driven anomaly detection method extracts human key point sequence, analyzes the position and speed of skeleton key points, and becomes a scheme with good effect and low computational overhead.

[0005] For example: a disease prediction system based on child abnormal behavior is disclosed in CN 112233800 B, which collects child behavior video information in real time through a camera, extracts behavior features and time sequence motion features from the video information, and constructs a prediction model to predict child abnormal behavior. However, this system only uses visible light video as a single modal information for monitoring. In the night or dim light environment, the imaging quality of visible light video will decrease significantly, making it difficult to accurately extract behavior features, and the monitoring result and behavior prediction are not accurate. SUMMARY

[0006] The purpose of the present application is a single-person abnormal behavior recognition method and system based on multi-modal skeleton feature fusion, which realizes abnormal behavior recognition by fusing multi-modal visual information and time series skeleton features, and provides effective and reliable basis for the rapid response and risk warning of intelligent security systems.

[0007] In order to achieve one of the above purposes, the present application adopts the following technical scheme:

[0008] A single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion, comprising the following steps:

[0009] S1: collect continuous RGB image frames and infrared thermal imaging image frames in the monitoring video, use a multi-modal fusion model combining YOLOv12 and Transformer to detect the skeleton key points of the human body in the extracted RGB image frames and infrared thermal imaging image frames, and construct a single-person posture time series dataset;

[0010] S2: construct a skeleton frame sequence of the target personnel according to the single-person posture time series dataset, calculate the skeleton key point velocity vector through the continuous skeleton frame sequence of each target personnel, fuse the skeleton key point information of the single-person posture time series dataset with the velocity vector, and form an action feature sequence;

[0011] S3: input the action feature sequence into an MPED-RNN model, split the skeleton frame sequence representation into two components of global displacement and local posture, and perform joint encoding, decoding and prediction through a double-channel GRU network to obtain a posture sequence reconstruction result and a future skeleton key point prediction result;

[0012] S4: calculate the prediction error and reconstruction error according to the reconstruction result and future skeleton key point prediction result, evaluate whether the current behavior deviates from the normal trajectory, and determine whether the current behavior is an abnormal state.

[0013] Further, the S1: collect continuous RGB image frames and infrared thermal imaging image frames in the monitoring video, use a multi-modal fusion model combining YOLOv12 and Transformer to detect the human body and estimate the skeleton key points thereof in the extracted RGB image frames and infrared images, and construct a single-person posture time series dataset; specifically comprising:

[0014] S101: collect visible light image frames and infrared thermal imaging image frames in a monitoring scene video, and perform time series alignment to construct a synchronous multi-modal video sequence;

[0015] S102: construct a multi-modal Transformer-YOLO double-flow backbone network architecture;

[0016] S103: realize cross-modal feature extraction by using the Transformer-YOLO double-flow backbone network architecture, extract the skeleton key point information of the target personnel in each frame, obtain a two-dimensional coordinate sequence (x, y, v) containing n skeleton key points of head, neck and limb joints, wherein v represents the skeleton key point visibility, and construct a single-person posture time series dataset;

[0017] S104: divide the single-person posture time series dataset into a training set, a validation set and a test set.

[0018] Further, the S101: collect visible light image frames and infrared thermal imaging image frames in the monitoring scene, and perform time sequence alignment to construct a synchronous multi-modal video sequence; specifically comprising:

[0019] In a complex environment such as low light, the dual-mode video data of abnormal behavior is collected, the timestamp synchronization technology is used to accurately synchronize the video streams of the two modalities of visible light and infrared, to ensure data alignment, control the inter-frame error within a certain time range, and form a time sequence accurately synchronized video sequence;

[0020] The positive samples of abnormal behavior and the negative samples of daily activities are cross-labeled by two people to construct a high-quality training data set.

[0021] Further, the S102: construct a multi-modal Transformer-YOLO dual-flow backbone network architecture; specifically comprising:

[0022] First, the RGB image frame and the infrared thermal imaging image frame are respectively sent into two independent YOLOv12 backbone networks, and the residual efficient layer aggregation network R-ELAN and the convolution layer inside the network are used to perform layer-by-layer abstraction on the input original pixel data, convert the image from simple edge, texture and other bottom information to more high-level semantic feature representation, and output visible light feature map and thermal imaging feature map ;

[0023] Second, the visible light feature map and the thermal imaging feature map are input, processed by 3-D weight and dot product fusion to obtain an intermediate feature map, the intermediate feature map is connected to generate a feature vector , and position embedding coding space information is added;

[0024] Then, the position information coded as the feature vector is processed by multiple Transformer modules to output a feature map , which strengthens the feature representation capability of multi-modal target detection;

[0025] Finally, the feature map is sent into three independent multi-scale detection heads Y1, Y2 and Y3 to perform prediction on feature maps of different sizes respectively, so as to effectively detect targets of different sizes, and finally integrate and output the required human body bounding box, class label and confidence score.

[0026] Further, the S102 further comprises:

[0027] Transformer-YOLO employs a dual-stream Transformer encoder, which captures the long-range dependencies between visible light and infrared features through a cross-attention mechanism, performs texture feature complementation, and enhances the robustness of target detection. The model combines the local details of R-ELAN with the global context of Transformer through a global-local feature collaboration strategy, improving the detection accuracy of small targets and occluded targets. It also maintains real-time performance through a lightweight design of layered Transformer window partitioning.

[0028] Further, S2: Constructing a skeleton frame sequence of the target person based on the single-person posture time-series dataset, calculating the key point velocity vector through the continuous skeleton frame sequence of each target person, and fusing the key point position information with the velocity information to form a motion feature sequence; specifically including:

[0029] S201: Perform cross-frame ID association for each target person in the single-person pose temporal dataset, and use the lightweight multi-target tracking algorithm OC-SORT to achieve continuous identity matching of the targets, construct the skeleton frame sequence of the target person, and form the original key point trajectory data structure [T,n,D]: where T is the frame number, n is the number of skeleton key points, and D corresponds to the coordinates and visibility of the skeleton key points. , It is divided into two cases: visible and occluded.

[0030] S202: The velocity characteristics of the skeleton key points are expressed as the displacement differences of corresponding skeleton key points in two consecutive frames, and the formula is as follows:

[0031]

[0032] in Key points of the skeleton at time t and the previous frame Displacement difference of coordinates Let be the displacement difference of the y-coordinate of the skeleton keypoint at time t compared to the previous frame. , The positions of the key skeleton points at time t;

[0033] S203: Concatenate the original skeleton keypoint coordinates (x, y) and velocity vector (dx, dy) of each frame to form the multi-dimensional fused motion features of each frame [n, m], where m corresponds to the coordinates of the keypoints and the velocity vector (dx, dy). Based on the frame number T, a single-person action feature sequence of [T, n, m] is formed.

[0034] Further, the S3: input the action feature sequence into the MPED-RNN model, split the skeleton frame sequence representation into global displacement and local posture two components, jointly encode, decode and predict through the double-channel GRU network, obtain the posture sequence reconstruction result and the future skeleton key point prediction result; specifically comprising:

[0035] S301: divide the action feature sequence into two independent components: wherein the global displacement component: , represents the moving track of the center point of the trunk in the time sequence, contains a two-dimensional coordinate; the local posture deformation component represents the offset of other key points relative to the global center point, and its formula is:

[0036]

[0037] wherein is the x-axis position of the key point t, is the global displacement component, represents the relative two-dimensional coordinates containing n skeleton key points;

[0038] S302: send the global displacement component and the local posture deformation component into two independent GRU encoders for time series modeling, respectively extract the hidden state features of the global trajectory and the local deformation through the double channels, and update the state information at each time step;

[0039] S303: based on the hidden state output by the encoder, two decoders respectively perform time series reconstruction and next step prediction on the global and local features, output the complete posture sequence reconstruction result and the future key point prediction result;

[0040] Further, the S4: calculate the prediction error and the reconstruction error according to the reconstruction result and the future skeleton key point prediction result, evaluate whether the current behavior deviates from the normal trajectory, judge whether the current behavior is an abnormal state, specifically comprising:

[0041] S401: compare the next frame skeleton key point coordinates predicted at the current time with the skeleton key point coordinates received at the next time, calculate the L2 distance between them, sum the L2 distances of n skeleton key points, and calculate the prediction error , and its formula is:

[0042]

[0043] wherein, is the real coordinate of the i-th key point in the next frame, is the predicted coordinate of the i-th key point in the next frame;

[0044] The reconstructed coordinates of the current frame skeleton keypoints are compared with the actual coordinates of the current frame skeleton keypoints. The reconstructed distance L2 between them is calculated. The L2 distances of the n skeleton keypoints are summed to calculate the reconstruction error of the model. Its formula is:

[0045]

[0046] in, Let i be the true coordinates of the i-th keypoint in the current frame. The reconstructed coordinates of the i-th key point in the current frame;

[0047] Set the fusion weight coefficient Calculate the anomaly score for each frame. Its formula is:

[0048]

[0049] in This represents the prediction error at time point t. This represents the reconstruction error at time t, which is the difference between the pose reconstructed by the model after observing the current frame and the true pose.

[0050] S402: Calculate the single-frame anomaly score from S401. Smoothing is performed in the time dimension for continuous data. The anomaly scores of the frames are statistically averaged to generate the final time-period anomaly risk score. Its formula is:

[0051]

[0052] in This represents the current time point after smoothing. The calculated final anomaly risk score, This represents the number of frames taken within the sliding time window. Represents the first in the window Single-frame anomaly scoring; time-based anomaly risk scoring for the current target person. When the threshold is exceeded, the person is marked as being in an abnormal state, triggering an alarm logic.

[0053] To achieve the second objective mentioned above, the present invention adopts the following technical solution:

[0054] A single-person abnormal behavior recognition system based on multimodal skeleton feature fusion includes a data acquisition module, a model training module, and an analysis and processing module.

[0055] The data acquisition module is used for acquiring continuous RGB image frames and infrared thermal imaging image frames in a monitoring video, and a multi-modal fusion model of YOLOv12 combined with a Transformer is used for skeleton key point detection on a human body in the extracted RGB image frames and infrared thermal imaging image frames, and a single-person posture time sequence dataset is constructed;

[0056] The model training module is used for constructing a deep learning algorithm model for single-person abnormal behavior recognition, constructing a skeleton frame sequence of a target person according to the single-person posture time sequence dataset, calculating a skeleton key point speed vector through continuous skeleton frame sequences of each target person, fusing skeleton key point information of the single-person posture time sequence dataset with the speed vector to form an action feature sequence, inputting the action feature sequence into an MPED-RNN model, splitting the skeleton frame sequence representation into two components of global displacement and local posture, and performing joint encoding, decoding and prediction through a double-channel GRU network to obtain a posture sequence reconstruction result and a future skeleton key point prediction result.

[0057] The analysis processing module is used for calculating a prediction error and a reconstruction error according to the reconstruction result and the future skeleton key point prediction result, evaluating whether a current behavior deviates from a normal trajectory, and judging whether the current behavior is an abnormal state.

[0058] The present application has the following advantages:

[0059] The present application integrates the collected multi-modal data about a single person in a public scene, including visible light RGB images and infrared thermal imaging IR images, and combines a front-end perception model of YOLOv12 combined with a Transformer with a back-end time sequence analysis model of MPED-RNN, to realize a multi-modal environment perception + deep time sequence behavior understanding mode. The method is applied to real-time recognition of single-person abnormal behavior, and through front-end multi-modal fusion, the rich texture details of visible light and the all-weather perception ability of infrared thermal imaging are effectively combined, and the detection difficulties that cannot be solved by traditional single-mode schemes such as night and low light are overcome. After training, the present application obtains a single-person abnormal behavior recognition model with high robustness and high accuracy, which can accurately judge whether a person is in a state of falling down, sitting down or abnormal stillness, and provide a quantitative abnormal score, to provide strong support for intelligent security and emergency response.

[0060] In view of the false alarm problem caused by the difficulty of distinguishing similar but different nature actions by static or shallow time sequence model in the prior art, the MPED-RNN time sequence model is introduced, the skeletal movement is creatively decomposed into two components of "global displacement" and "local posture deformation" for deep modeling.

[0061] The method of the present application is also applicable to other scenes that require real-time monitoring of the state of a single person. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a flowchart of embodiment 1 of the present application;

[0063] Figure 2 is a flowchart of university playground environment anomaly behavior recognition in embodiment 1 of the present application;

[0064] Figure 3 is a principle block diagram of embodiment 2 of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Embodiment 1:

[0067] Referring to Figure 1 and Figure 2 , a single person anomaly behavior recognition method based on multi-modal skeleton feature fusion includes the following steps:

[0068] S1: Collect continuous RGB images and infrared thermal imaging images in public scene monitoring video, use YOLOv12 combined with the multi-modal fusion model of Transformer to detect human bodies and estimate their skeleton key points, and construct a single person posture time sequence dataset;

[0069] In the field of multi-modal target detection, combining the high-efficiency detection ability of YOLO with the global semantic modeling advantage of Transformer has become a research hotspot. The multi-modal Transformer-YOLO proposed in this study breaks through the limitations of traditional single-modal detection by deeply integrating YOLOv12 and Transformer architecture.

[0070] Specifically includes:

[0071] Step S101: Collecting visible light images and infrared thermal imaging images in a monitored video scene, time-aligning the images, and constructing a synchronized multi-modal video sequence;

[0072] Specifically, it comprises:

[0073] In a complex environment such as low light, the dual-mode video data of abnormal behavior is collected, the timestamp synchronization technology is used to accurately synchronize the video streams of the two modalities of visible light and infrared to ensure data alignment, and the frame error is controlled within a certain time range. In the embodiment, the frame error is controlled within ±10ms, and a time-accurately synchronized video sequence is formed.

[0074] The positive samples of abnormal behavior and the negative samples of daily activities are cross-labeled by two people to construct a high-quality training data set.

[0075] This series of operations is a key technical prerequisite for building a robust behavior recognition system. Among them, the cooperative collection of dual-mode data aims to break through the perception bottleneck of a single visual sensor under non-ideal conditions, and by fusing the fine texture of visible light and the thermal radiation features of infrared, it provides a data source with richer information dimension and higher signal-to-noise ratio for the model, thereby ensuring the all-weather and all-scene applicability of the system; Strict sub-frame level synchronization control ensures the strong correlation of captured cross-modal features in transient changes when analyzing high-speed dynamic events.

[0076] Step S102: Constructing a multi-modal Transformer-YOLO dual-flow backbone network architecture;

[0077] Specifically, it comprises:

[0078] First, the RGB image frame and the infrared thermal imaging image frame are respectively sent into two independent YOLOv12 backbone networks, and the residual efficient layer aggregation network R-ELAN and the convolution layer inside the network are used to perform layer-by-layer abstraction on the input raw pixel data, converting the image from simple edge and texture bottom layer information to more high-level semantic feature representation, and outputting visible light feature map and thermal imaging feature map .

[0079] Second, the visible light feature map and the thermal imaging feature map are input, processed by 3-D weight processing and dot product fusion to obtain an intermediate feature map, and the channel connection of the intermediate feature map generates a feature vector , and position embedding coding space information is added;

[0080] Then, the position information coded as is processed by 8 Transformer modules to output feature map , to strengthen the feature representation capability of multi-modal target detection;

[0081] Finally, the feature map is sent to three independent multi-scale detection heads Y1, Y2 and Y3, which respectively perform prediction on feature maps of different sizes to effectively detect targets of different sizes, and finally integrate and output the required human body bounding box, class label and confidence score.

[0082] To realize cross-modal semantic fusion, the Transformer-YOLO designs a double-stream Transformer encoder to capture the long-distance dependency relationship between visible light and infrared features through cross-attention mechanism. For example, in a low-light night scene, the thermal radiation features of the infrared branch complement the texture features of the visible light branch through the Transformer, enhancing the robustness of target detection. In addition, the model combines the local details (such as edges) of R-ELAN with the global context (such as pedestrian trajectories) of the Transformer through a global-local feature collaboration strategy, improving the detection accuracy of small targets and occluded targets. The lightweight design of the model through hierarchical Transformer window division maintains real-time performance.

[0083] The Transformer architecture uses RGB feature maps FVI and thermal imaging feature maps FIR as inputs, performs 3-D attention weighting on the input feature maps to enhance feature representation capability, and especially for input FVI and FIR respectively through 3-D weight through dot product with itself. The flattened features F'VI and FIR are connected along the channel dimension to generate a feature vector F'C, and a learnable position embedding F'C is added to generate input sent to the Transformer module. The position embedding can encode the position information into F'C, which helps to distinguish the spatial information between different labels during training. The Transformer module consists of 8 transformer modules, each module has two normalization layers, a multi-head attention mechanism and an MLP, and the MLP consists of a two-layer fully connected feedforward network with GELU activation.

[0084] The S102 further includes that the Transformer-YOLO adopts a double-stream Transformer encoder to capture the long-distance dependency relationship between visible light and infrared features through cross-attention mechanism, performs texture feature complementation, and enhances the robustness of target detection; the model combines the local details of R-ELAN with the global context of the Transformer through a global-local feature collaboration strategy, improves the detection accuracy of small targets and occluded targets, and maintains real-time performance through the lightweight design of hierarchical Transformer window division.

[0085] Step S103: Cross-modal feature extraction is implemented using the Transformer-YOLO double-stream backbone network architecture to extract the skeleton key point information of the target person in each frame, including the two-dimensional coordinate sequence (x, y, v) of the head, neck, and n limb joints, where v represents the visibility of the skeleton key point. In this specific embodiment: 17 skeleton key points including head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left pelvis, right pelvis, left knee, right knee, left ankle, and right ankle.

[0086] Step S104: The data set is divided into a training set, a validation set, and a test set.

[0087] In this specific embodiment: Training set: Select most of the data in the data set, such as 70%, as the training set, to train the MPED-RNN model. Ensure that the training set contains samples of various types of abnormal behaviors.

[0088] Validation set: Select a small part of the data in the data set, such as 20%, as the validation set, to evaluate the model performance and adjust the hyperparameters during the training process.

[0089] Test set: Select the remaining data, such as 10%, as the test set, to evaluate the final performance of the model. The test set should contain some abnormal behavior types or special cases that have not appeared in the training set and validation set to test the generalization ability of the model.

[0090] To improve the generalization performance of the model, a data augmentation strategy is introduced during the training phase, such as applying random cropping or rotation transformations to the training samples to expand the diversity of the data. The optimization of model parameters is achieved through the backpropagation algorithm, using the training data set for iterative learning. After training is complete, an independent test set is used to test the final performance of the model, and the overall performance of the model is objectively evaluated by calculating key performance indicators such as accuracy and recall.

[0091] Step S2: Construct the skeleton frame sequence of the target person according to the single-person posture time series data set, calculate the skeleton key point velocity vector through the continuous skeleton frame sequence of each target person, and fuse the collected skeleton key point position information with the velocity vector to form an action feature sequence, which specifically includes:

[0092] Step S201: Cross-frame ID association is performed on each detected target person in the single-person posture time series data set, a lightweight multi-target tracking algorithm OC-SORT is used to realize continuous identity matching of the target, and a skeleton frame sequence of the person to be detected is constructed to form a raw key point trajectory data structure of [T, n, D]: where T is the number of frames, n is the number of skeleton key points, and D corresponds to the coordinates and visibility of the key points. ​There are two cases of visible and occlusion, n = 17, D = 3 in this embodiment;

[0093] Step S202: The speed feature of the skeleton key point is the displacement difference of the corresponding skeleton key point in the two consecutive frames, and the formula is:

[0094]

[0095] Wherein is the displacement difference of the x coordinate of the skeleton key point at time t and the skeleton key point of the last frame, is the displacement difference of the y coordinate of the skeleton key point at time t and the skeleton key point of the last frame, is the displacement difference of the x coordinate of the skeleton key point at time t and the skeleton key point of the last frame, , is the position of the skeleton key point at time t;

[0096] In the two-dimensional image space, due to the lack of depth, only x and y coordinates cannot well represent the real position of the person in the scene, in order to make up for this defect, the displacement difference of the skeleton key point is used to enhance the global component and to normalize the local component.

[0097] Step S203: The original key point coordinates (x, y) and the velocity vector (dx, dy) of each frame are spliced to form a multi-dimensional fusion action feature of each frame [n, m], m corresponds to the coordinate and velocity vector of the key point , according to the frame number T, a single person action feature sequence of [T, n, m] is formed. In this embodiment, 17 skeleton key points are collected, that is, n = 17, m = 4, a multi-dimensional fusion action feature of each frame [17, 4] is formed, 4 corresponds to the coordinate and velocity vector of the key point , a single person action feature sequence of [T, 17, 4] is formed.

[0098] S3: input the action feature sequence into the MPED-RNN model, split the skeleton frame sequence into global displacement and local posture two components, encode, decode and predict through double channel GRU network, get the posture sequence reconstruction result and the future skeleton key point prediction result;

[0099] MPED-RNN models the global and local components as two interacting subprocesses, the internal state of one process is used as an additional feature in the input of the other process. That is, the model consists of two recurrent encoder-decoder network branches, each branch is dedicated to one component.

[0100] Each branch of the model adopts a single-encoder-double-decoder structure, containing three RNNs: an encoder, a reconstruction decoder, and a prediction decoder. MPED-RNN not only models the dynamics of each component independently, but also models the interdependence between them through a cross-branch message passing mechanism.

[0101] The global dynamics and the local dynamics can be modeled as two concurrent sub-processes, respectively:

[0102] In regular video data, the two dynamic processes of pose evolution and global displacement can even be exhibited independently in some cases. For example, an individual can only move his limbs, while his global spatial position in the scene remains relatively static. Similarly, a motorcyclist can also move a large range of the scene while keeping his torso pose relatively stable.

[0103] However, if viewed in a specific context, the conventional human activity pattern usually exhibits significant internal correlation between the two motion components. Therefore, the destruction or absence of such cross-component expected correlation can often be regarded as a sign or indication of abnormal behavior. In the above example, if such limb-independent movement or pose-fixed movement behavior occurs in a typical pedestrian walking scene, they constitute an abnormal event of analytical value.

[0104] Specifically comprising:

[0105] Step S301: dividing the action feature sequence [T, 17, 4] into two independent components: a global displacement component: , representing the moving trajectory of the torso center point in the time sequence, containing a two-dimensional coordinate; and a local pose deformation component , representing the offset of other key points relative to the global center point, whose formula is:

[0106]

[0107] wherein is the x-axis position of the key point at time t, is the global displacement component, , representing the relative two-dimensional coordinates of the n skeleton key points; in this specific embodiment, it contains the relative two-dimensional coordinates of 17 points;

[0108] Step S302: sending the global displacement component and the local pose deformation component into two independent GRU encoders for time series modeling, respectively, and extracting the hidden state features of the global trajectory and the local deformation in two channels, and updating the state information at each time step;

[0109] Step S303: Based on the hidden state of the encoder output, two decoders respectively reconstruct the global and local features in time sequence and predict the next step, output the complete pose sequence reconstruction result and the future key point prediction result;

[0110] The single-encoder-double-decoder architecture is the core design essence of the model. By setting two parallel decoding tasks of "reconstruction" and "prediction", the encoder is forced to learn a compact representation that is highly condensed and information-rich for regular behavior patterns. Among them, the reconstruction task requires the model to accurately remember and reproduce the just observed action sequence, while the prediction task requires the model to reasonably infer the future action based on the learned motion law. This double constraint makes the model's definition of "normal" very deep. Therefore, when an "abnormal" behavior sequence that has not been learned and does not conform to the regular dynamic characteristics is input, the model will not be able to effectively reconstruct and predict it, resulting in significant errors in these two tasks. This comprehensive error value constitutes the core quantitative basis for subsequent anomaly discrimination. The larger the value, the higher the degree of deviation of the current behavior from the normal pattern.

[0111] S4: Calculate the prediction error and reconstruction error according to the reconstruction result and future skeleton key point prediction result, evaluate whether the current behavior deviates from the normal trajectory, and determine whether the current behavior is an abnormal state;

[0112] Specifically, it includes:

[0113] Step S401: Compare the predicted next frame skeleton key point coordinates at the current time with the received next frame skeleton key point coordinates in the video sequence, calculate the distance L2 between them, sum the distance L2 of n skeleton key points, and calculate the prediction error The formula is:

[0114]

[0115] Wherein, is the real coordinate of the i-th key point in the next frame, is the predicted coordinate of the i-th key point in the next frame;

[0116] Compare the reconstructed current frame skeleton key point coordinates with the real current frame skeleton key point coordinates, calculate the reconstruction distance L2 between them, sum the distance L2 of n skeleton key points, and calculate the reconstruction error of the model The formula is:

[0117]

[0118] Wherein, is the real coordinate of the i-th key point in the current frame, The i-th key point in the reconstructed coordinate of the current frame;

[0119] Setting the fusion weight coefficient , calculating the anomaly score of each frame , the formula of which is:

[0120]

[0121] Wherein represents the prediction error at time point t, represents the reconstruction error at time point t, that is, the difference between the posture reconstructed by the model after observing the current frame and the real posture;

[0122] Step S402: Calculate the single-frame anomaly score of S401 Smooth in the time dimension, and statistically average the anomaly scores of consecutive frames to generate the final time period anomaly risk score , the formula of which is:

[0123]

[0124] Wherein represents the final anomaly risk score calculated at the current time point after smoothing, represents the number of frames taken by the sliding time window, represents the single-frame anomaly score of the i-th frame in the window; when the time period anomaly risk score of the current target person exceeds the set threshold, the person is marked as being in an abnormal state, triggering the alarm logic.

[0125] Referring to FIG. 1, for example: in a university playground environment, a plurality of groups of monitoring devices integrated with a visible light camera Sony FS5 and an infrared thermal imager FLIR A655sc are deployed along the edge of the playground. The devices collect dual-mode video data of student activities in the playground all day through timestamp synchronization technology, and transmit the synchronized RGB and IR video streams to the “single-person abnormal behavior recognition system” based on the present application deployed in the campus security center in real time. Figure 2 At night, a student falls down while running in the playground, and the working process of the system is as follows:

[0126] Step S1: Collect the RGB and infrared video of the campus playground at night, and detect the running student and extract its skeleton using the multi-modal fusion model YOLOv12+Transformer.

[0127]

[0128] ​​​The data acquisition receives the video stream of the area where the student is located. Due to insufficient lighting, the visible light image quality is low, but the infrared image clearly captures the thermal signal of the human body. The system's built-in YOLOv12 combined with the Transformer multi-modal fusion model successfully fuses the features of the two modalities, accurately detects the student, and real-time extracts the 17 skeleton key point sequence.

[0129] Step S2: Continuously track the student, calculate the key point speed, and fuse with the position information to construct a [T, 17, 4] feature sequence describing his running action.

[0130] Through the OC-SORT tracking algorithm, a unique ID is assigned to the student, and his skeleton changes are continuously tracked: the speed vector of the key points is calculated in real time, and according to the formula The speed vector of the key points is calculated, and the coordinate information is spliced to form a [T, 17, 4] multi-dimensional action feature sequence describing the process from "running" to "falling".

[0131] Step S3: When the student falls, his action sequence is sent to the MPED-RNN model. The model identifies that the action deviates from the learned normal pattern, resulting in a large error in reconstruction and prediction.

[0132] The action feature sequence is sent to the MPED-RNN model in the analysis processing module for time series analysis. In the running stage, the student's global displacement, i.e. the body's forward progress, and the local posture, i.e. the swing height of the limbs, are coordinated, which conforms to the normal pattern learned by the model during training. The model can accurately reconstruct and predict the action, with reconstruction error and prediction error both maintaining at a very low level.

[0133] At the moment of falling, the global displacement component changes dramatically, the center of gravity height drops rapidly, and the local posture deformation component presents a different pattern from normal running. The abnormal pattern that the model has not learned during training causes the two decoders built into the model to fail to accurately reconstruct and predict the posture, resulting in prediction error and reconstruction error rising sharply.

[0134] Step S4: According to the error produced in S3, a high abnormal score is calculated, and after sliding window smoothing, the risk score exceeds the threshold, determining "abnormal state" and triggering an alarm.

[0135] First, use the formula to integrate the sharply rising prediction and reconstruction errors into a very high single-frame abnormal score ;

[0136] Second, to avoid single frame jitter caused by misjudgment, the system adopts sliding time window, according to the formula:

[0137]

[0138] Smooth statistics on the abnormal scores of continuous multiple frames to obtain the final time period abnormal risk score , for continuous high score, more than the preset threshold, the system finally determines that the student is in an abnormal state.

[0139] Finally, determine Exceeding the threshold, immediately trigger the alarm logic. The specific performance is: the backend service generates a warning signal, and the front-end visual monitoring interface immediately pops up an alarm window, highlights the specific location of the event, and is accompanied by an audible and visual prompt to notify the security personnel to arrive at the scene to handle it in the first time.

[0140] This embodiment fully proves that the application can identify sudden abnormal behaviors such as falling in a campus actual scene with insufficient light in real time and accurately, and realize the intelligent early warning function of intelligent campus safety monitoring. Specific embodiment 2:

[0142] Referring to Figure 3 A single-person abnormal behavior recognition system based on multi-modal skeleton feature fusion, comprising a data acquisition module 1, a model training module 2 and an analysis processing module 3,

[0143] The data acquisition module is used to acquire continuous RGB image frames and infrared thermal imaging image frames in a monitoring video, a multi-modal fusion model of YOLOv12 combined with Transformer is adopted to detect skeleton key points of a human body in the extracted RGB image frames and the infrared thermal imaging image frames, and a single-person posture time sequence dataset is constructed; a timestamp synchronization technology is adopted for data acquisition, frame error is reduced, and a multi-modal video stream with accurate time sequence alignment is formed,

[0144] The video stream is input into the multi-modal fusion model of YOLOv12 combined with Transformer, a lightweight multi-target tracking algorithm OC-SORT is adopted to associate the target personnel across frames, human body detection and skeleton key point estimation are performed on each frame of image, a two-dimensional coordinate sequence containing 17 skeleton key points is extracted, a displacement difference of corresponding key points between two continuous frames is calculated as a velocity vector, and a single-person action feature sequence with a structure of [T, 17, 4] (T is the number of frames, 17 is the number of key points, and 4 is x, y coordinates and dx, dy velocity) is generated;

[0145] The model training module is configured to construct a deep learning algorithm model for single-person abnormal behavior recognition, construct a skeleton frame sequence of a target person according to the single-person posture time-series data set, calculate a skeleton key point speed vector through the continuous skeleton frame sequence of each target person, fuse the skeleton key point information of the single-person posture time-series data set with the speed vector, and form a motion feature sequence; the deep learning algorithm model for single-person abnormal behavior recognition specifically includes: a multi-modal fusion model of YOLOv12 combined with a Transformer is used to process the synchronized multi-modal video sequence, realize target detection and human body skeleton key point estimation of a single person in the video, calculate a skeleton key point speed vector for the extracted continuous skeleton frame sequence, and fuse the position information and speed information of the skeleton key point to form a motion feature sequence containing time-series dynamics.

[0146] The motion feature sequence is input into an MPED-RNN model, which decomposes the skeleton frame sequence into two components of global displacement and local posture, and learns the spatiotemporal dependence relationship under the normal behavior mode through joint encoding, decoding and prediction by a double-channel GRU network to obtain a posture sequence reconstruction result and a future skeleton key point prediction result.

[0147] The analysis processing module calculates a prediction error and a reconstruction error according to the reconstruction result and the future skeleton key point prediction result, evaluates whether the current behavior deviates from the normal trajectory, judges whether the current behavior is an abnormal state, and triggers an early warning when the error score exceeds a preset threshold.

[0148] The roles of the data acquisition module 1, the model training module 2 and the analysis processing module 3 in the system are the same as those in the single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion in Embodiment 1, and thus this embodiment is omitted here.

[0149] The technical solutions provided by the present application are described in detail above. The principles and implementation modes of the present application are described by applying specific examples, and the above examples are only used to help understand the method and core idea of the present application. It should be noted that those skilled in the art can make some improvements and modifications to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion, characterized in that, The method comprises the following steps: S1: Collecting continuous RGB image frames and infrared thermal imaging image frames in the monitoring video, using a multi-modal fusion model of YOLOv12 combined with Transformer to detect the human body in the extracted RGB image frames and infrared thermal imaging image frames, and constructing a single-person posture time series dataset; S2: Constructing a deep learning algorithm model for single-person abnormal behavior recognition, constructing a skeleton frame sequence of the target personnel according to the single-person posture time series dataset, calculating the skeleton key point velocity vector through the continuous skeleton frame sequence of each target personnel, fusing the skeleton key point information of the single-person posture time series dataset with the velocity vector to form an action feature sequence; S3: Inputting the action feature sequence into an MPED-RNN model, splitting the skeleton frame sequence into two components of global displacement and local posture, and performing joint encoding, decoding and prediction through a double-channel GRU network to obtain a posture sequence reconstruction result and a future skeleton key point prediction result; S4: Calculating the prediction error and reconstruction error according to the reconstruction result and future skeleton key point prediction result, evaluating whether the current behavior deviates from the normal trajectory, and judging whether the current behavior is an abnormal state.

2. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 1, characterized in that: The S1 specifically comprises: S101: Collecting visible light image frames and infrared thermal imaging image frames in the monitoring scene video, and performing time series alignment to construct a synchronous multi-modal video sequence; S102: Constructing a multi-modal Transformer-YOLO double-flow backbone network architecture; S103: Realizing cross-modal feature extraction by using the Transformer-YOLO double-flow backbone network architecture, extracting the skeleton key point information of the target personnel in each frame, obtaining a two-dimensional coordinate sequence (x, y, v) containing n skeleton key points of head, neck and limb joints, wherein v represents the skeleton key point visibility, and constructing a single-person posture time series dataset; S104: Dividing the single-person posture time series dataset into a training set, a validation set and a test set.

3. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 2, characterized in that: The S101: collecting visible light image frames and infrared thermal imaging image frames in the monitoring scene, and performing time series alignment to construct a synchronous multi-modal video sequence; specifically comprises: Collecting dual-modal video data of abnormal behaviors, using timestamp synchronization technology to accurately synchronize the video streams of two modalities of visible light and infrared to ensure data alignment, controlling the frame error within a certain time range to form a time series accurately synchronized video sequence; Double-person cross-labeling is performed on the positive samples of abnormal behaviors and negative samples of daily activities to construct a high-quality training dataset.

4. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 2, characterized in that: The S102: constructing a multi-modal Transformer-YOLO double-flow backbone network architecture; specifically comprises: Firstly, the RGB image frame and the infrared thermal imaging image frame are respectively sent into two independent YOLOv12 backbone networks, and the residual efficient layer aggregation network R-ELAN and the convolution layer in the network are used to perform layer-by-layer abstraction on the input original pixel data, and the image is converted from simple edge and texture bottom layer information to more high-level semantic feature representation, and the visible light feature map is output and the thermal imaging feature map ; Second, input visible light feature map And thermal imaging feature map , 3-D weight processing and dot product fusion, get intermediate feature map, connect the intermediate feature map channel to generate feature vector Add position embedding coding space information; Then, the position information encoded as a feature vector is processed by multiple Transformer modules to output a feature map , enhancing the feature representation capability of the multi-modal object detection. Finally, the feature map is sent to three independent multi-scale detection heads Y1, Y2 and Y3, which respectively make predictions on feature maps of different sizes to effectively detect targets of different sizes, and finally integrate the output of the required human body bounding box, class label and confidence score.

5. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 2 or 4, characterized in that: The S102 further comprises: Transformer-YOLO adopts a dual-stream Transformer encoder to capture long-range dependencies between visible light and infrared features through cross-attention mechanisms, complement texture features, and enhance the robustness of target detection. The model combines the local details of R-ELAN and the global context of Transformer through a global-local feature collaboration strategy to improve the detection accuracy of small targets and occluded targets while maintaining real-time performance through a lightweight design based on hierarchical Transformer window division.

6. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 1, characterized in that: The S2 specifically comprises: S201: Construct a deep learning algorithm model for single-person abnormal behavior recognition, perform cross-frame ID association on each target person in the single-person pose time series data set, use a lightweight multi-target tracking algorithm OC-SORT to realize continuous identity matching of the target, construct the skeleton frame sequence of the target person, and form the original key point trajectory data structure [T, n, D]: where T is the frame number, n is the number of skeleton key points, and D corresponds to the coordinates and visibility of the skeleton key points (visible and occluded) , ​ S202: The speed feature of the skeleton key point is the displacement difference of the corresponding skeleton key point in the two consecutive frames, and the formula is: ； wherein is the displacement difference of the x coordinate of the skeleton key point at time t and the previous frame skeleton key point is the displacement difference of the x coordinate of the skeleton key point at time t and the previous frame skeleton key point is the displacement difference of the y coordinate of the skeleton key point at time t and the previous frame skeleton key point , is the skeleton key point position at time t S203: Splice the original skeleton key point coordinate (x, y) and the velocity vector (dx, dy) of each frame to form a multi-dimensional fusion action feature of each frame [n, m], m corresponds to the coordinate and velocity vector of the key point According to the frame number T, a single-person action feature sequence [T, n, m] is formed.

7. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 1, characterized in that: The S3 specifically comprises: S301: divide the action feature sequence into two independent components: where the global displacement component: represents the moving trajectory of the center point of the torso in the time sequence, containing a two-dimensional coordinate; the local posture deformation component represents the offset of other key points relative to the global center point, and its formula is: ； wherein is the key point x-axis position at time t, is the global displacement component, denotes the relative two-dimensional coordinates of the n skeleton key points; S302: The global displacement component and the local posture deformation component are respectively input into two independent GRU encoders for time series modeling. The double channels extract the hidden state features of the global trajectory and the local deformation, and the state information is updated at each time step. S303: Based on the hidden state output by the encoder, two decoders respectively reconstruct and predict the next step of the global and local features, output the complete posture sequence reconstruction result and the future key point prediction result.

8. The single-person abnormal behavior recognition method based on multi-modal skeleton feature fusion according to claim 1, characterized in that: The S4 specifically comprises: S401: compare the predicted next frame skeleton key point coordinates at the current time with the received skeleton key point coordinates at the next time, calculate the predicted distance L2 therebetween, sum the L2 distances of the n skeleton key points, and calculate the prediction error The formula is: ； wherein, is the true coordinate of the i-th keypoint in the next frame, is the predicted coordinate of the i-th keypoint in the next frame. The reconstructed current frame skeleton key point coordinates are compared with the real current frame skeleton key point coordinates, the reconstruction distance L2 between them is calculated, the L2 distances of the n skeleton key points are summed, and the reconstruction error of the model is calculated The formula is: ； wherein, is the real coordinate of the i-th key point in the current frame, is the reconstructed coordinate of the i-th key point in the current frame; Setting fusion weight coefficients , calculate the anomaly score of each frame , the formula is: ； wherein represents the prediction error at time point t, represents the reconstruction error at time point t, i.e. the difference between the pose reconstructed by the model after observing the current frame and the true pose; S402: Calculate the single-frame anomaly score in S401 In the time dimension, the anomaly scores of consecutive frames are statistically averaged to generate the final time period anomaly risk score In the time dimension, the anomaly scores of consecutive frames are statistically averaged to generate the final time period anomaly risk score The formula is: ； wherein represent the frame number taken by the sliding time window, represent the frame number taken by the sliding time window, represent the frame number taken by the sliding time window, represent the frame number taken by the sliding time window, represent the frame number taken by the sliding time window, represent the frame number taken by the sliding time window, 9. A single-person abnormal behavior recognition system based on multi-modal skeleton feature fusion, characterized in that: It includes a data acquisition module, a model training module and an analysis processing module, The data acquisition module is used to acquire continuous RGB image frames and infrared thermal imaging image frames in the monitoring video, and a multi-modal fusion model combining YOLOv12 and Transformer is used to detect the skeleton key points of the human body in the extracted RGB image frames and infrared thermal imaging image frames, and to construct a single-person posture time series dataset; The model training module is used to construct a deep learning algorithm model for single-person abnormal behavior recognition, construct a skeleton frame sequence of the target personnel according to the single-person posture time series dataset, calculate the skeleton key point velocity vector through the continuous skeleton frame sequence of each target personnel, fuse the skeleton key point information of the single-person posture time series dataset with the velocity vector to form an action feature sequence, input the action feature sequence into the MPED-RNN model, split the skeleton frame sequence into global displacement and local posture two components, and perform joint encoding, decoding and prediction through a double-channel GRU network to obtain the posture sequence reconstruction result and the future skeleton key point prediction result; The analysis processing module calculates the prediction error and reconstruction error according to the reconstruction result and future skeleton key point prediction result, evaluates whether the current behavior deviates from the normal trajectory, and judges whether the current behavior is an abnormal state.

Images