Abnormal behavior recognition method based on visual large model and cognitive Agent
By combining the YOLOv12 network and the Mamba model, the challenges of efficiency in long-term modeling and environmental semantic understanding are addressed, achieving efficient and reliable abnormal behavior identification and generating alarms with environmental semantic interpretation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHWEST UNIV
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies face challenges in long-term continuous motion monitoring and abnormal behavior recognition in complex environments, including efficiency and memory bottlenecks in long-term modeling, as well as a lack of environmental semantic understanding and physical consistency constraints, resulting in jitter in predicted trajectories and high false alarm rates.
The YOLOv12 network is used to extract human key point sequences, combined with the Mamba model for long-term time series modeling, and action smoothing is achieved through selective state space mechanism and residual prediction architecture. A cognitive agent is introduced for semantic analysis to generate natural language warnings.
It improves the efficiency of long-term modeling, reduces the false alarm rate, enhances the environmental perception capability and decision reliability of intelligent security systems, and provides highly interpretable alarms.
Smart Images
Figure CN122024319A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to an abnormal behavior recognition method based on a large visual model and a cognitive agent. Background Technology
[0002] Anomaly detection is a core task of intelligent security monitoring systems, aiming to detect and issue early warnings of emergencies in real time from complex video streams. With the advancement of smart city construction, this technology has extremely wide application value in fields such as public safety and elderly care.
[0003] Despite the progress made in existing deep learning-based action recognition methods, two major challenges remain when dealing with long-term continuous action monitoring and semantic understanding in complex environments: The first major challenge is the efficiency and memory bottleneck of long-term modeling. Traditional recurrent neural networks are prone to gradient vanishing and forgetting when processing long sequences. While Transformer-based models have strong memory capabilities, their computational complexity increases quadratically with the sequence length, resulting in slow inference speed and difficulty in meeting the real-time requirements of edge devices. The second major challenge lies in the lack of environmental semantic understanding and physical consistency constraints. Existing technologies mostly rely on pure skeleton coordinate regression, and the generated predicted trajectories often exhibit jitter or first-frame jumps, which do not conform to the physical inertia of human movement. At the same time, current systems lack "cognitive" capabilities, only able to output anomaly probability scores, and cannot combine the surrounding environment for logical reasoning.
[0004] For example, the publication CN117690192A, titled "Method and Device for Abnormal Behavior Recognition Based on Multi-View Instance-Semantic Consensus Mining," describes a method that extracts features by constructing instance encoders and semantic encoders respectively, and combines contrastive learning and semantic distillation mechanisms to maximize the consistency of multi-view data in the feature space. However, this method mainly focuses on solving the spatial alignment and consensus problem of static image features under multiple perspectives, lacking modeling of the long-term dynamic evolution process of human actions, making it difficult to capture the evolution trend and temporal logic of actions. Summary of the Invention
[0005] To address the aforementioned shortcomings in existing technologies, this invention provides an abnormal behavior recognition method based on a large visual model and a cognitive agent, which solves two major challenges faced by existing technologies.
[0006] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: an abnormal behavior recognition method based on a large visual model and a cognitive agent, comprising the following steps: S1: Acquire RGB images and infrared thermal imaging (IR) images from the monitored scene, and extract human key point sequences using the YOLOv12 network; S2: Perform temporal difference processing on the keypoint sequence to construct a skeleton velocity vector containing motion trends; S3: Input the skeleton velocity vector into the Mamba model and use the Selective State Space (SSM) mechanism to adjust the model parameters to perform long-term time-series modeling of the action sequence; S4: Use a residual prediction architecture to predict the velocity change in the next frame and reconstruct the human posture, and achieve smooth motion through physical consistency constraints; S5: Calculate the anomaly score based on the prediction error. When the score exceeds the preset threshold, trigger the cognitive agent, call the visual big data model (VLM) to perform semantic analysis on the current scene environment, and generate a natural language warning containing the anomaly type and environmental trigger.
[0007] Further, step S1 includes the following sub-steps: S11: Construct a dual-modal acquisition front end to simultaneously acquire visible light RGB and infrared thermal IR video streams, and eliminate the timing deviation between modes through timestamp alignment technology to form an aligned dual-modal frame sequence; S12: The aligned bimodal frame sequence is input into the YOLOv12 object detection network, and the texture features of RGB and the thermal radiation features of IR are fused through the cross-modal attention mechanism; S13: Based on the fusion results, perform human detection and pose estimation on each frame of the image, extract two-dimensional coordinate data containing human key points, and form the original skeleton position sequence.
[0008] Furthermore, the step of inputting the aligned bimodal frame sequence into the YOLOv12 object detection network and fusing RGB texture features with IR thermal radiation features through a cross-modal attention mechanism includes the following sub-steps: S121: Construct the backbone network of YOLOv12 based on the CSPDarknet cross-stage local network architecture. The backbone network consists of several cascaded CSP bottleneck modules and downsampling convolutional layers, which stitch the input bimodal images in the channel dimension to form a composite input tensor. S122: Input the composite input tensor into the first convolutional module of the backbone network, perform initial downsampling and channel expansion on it, map the image data into an initial feature map, and complete the transformation from image space to feature space; S123: A CBS module is constructed by cascading convolutional layers, BN layers, and SiLU activation functions. The CBS module and the CSP bottleneck module perform continuous convolution operations, alternately extracting features from the initial feature map and downsampling with stride convolutions, sequentially outputting a resolution equal to the original... Figure 1The system outputs a shallow texture feature map F1 with a resolution of 1 / 8 and a mid-level contour feature map F2 with a resolution of 1 / 16. Finally, the spatial pyramid pooling (SPPF) module is introduced to perform max pooling aggregation on the deep features, and the output is a deep semantic feature map F3 with a resolution of 1 / 32. S124: Using the feature maps F1, F2, and F3 output by the backbone network as input, a path aggregation network PANet containing a bidirectional fusion path is constructed as the neck network of YOLOv12. The deep semantic feature map F3 is upsampled by nearest neighbor interpolation and then concatenated with the mid-level feature map F2 along the channel dimension. The fused features are then upsampled again and concatenated with the shallow feature map F1 along the channel dimension to construct a top-down semantic propagation path. Subsequently, the fused features are downsampled using convolution operations and then concatenated with the deep features a second time to construct a bottom-up localization enhancement path, outputting a fused feature pyramid containing rich semantic and location information.
[0009] Furthermore, based on the fusion results, human detection and pose estimation are performed on each frame of the image, and two-dimensional coordinate data containing human key points are extracted to form the original skeleton position sequence, including the following steps: S131: The fused feature pyramid is input into the decoupled detection head of YOLOv12. The detection head is designed as a decoupled architecture, which includes three parallel branches: classification, bounding box regression and key point regression. In the key point regression branch, a direct regression strategy based on heatmap features is adopted. For each detected target anchor box, the network uses the grid cell of the current feature map as a local reference system to directly predict the coordinate offset of the human skeleton key points relative to the geometric center of the grid cell of the current feature map, covering the main joints of the head, torso and limbs. S132: Based on the confidence score of each predicted box output by the classification branch in the detection head, it represents the probability value that the current predicted target belongs to the human body category; S133: Stack the skeleton coordinates of consecutive frames in the time dimension according to the timestamp order of the video frames to construct the original skeleton position sequence.
[0010] Furthermore, step S2 includes the following sub-steps: S21: Perform cross-frame identity association on the detected target and construct a single person continuous skeleton trajectory; S22: To address the discontinuity issue in the first frame of motion prediction, the displacement difference of key points between two consecutive frames is calculated as a velocity feature. S23: Concatenate the absolute position coordinates of the key points with the velocity features to construct a high-dimensional motion feature vector.
[0011] Furthermore, the step of performing cross-frame identity association on the detected target and constructing a single person's continuous skeleton trajectory includes the following sub-steps: S211: Establish the Kalman filter state vector and error covariance matrix for each detected target; S212: In the current frame, calculate the intersection-union ratio between the predicted box position of the Kalman filter and the detection box position output by YOLOv12, and construct the association cost matrix; S213: Use the Hungarian algorithm to perform global optimal bipartite graph matching on the correlation cost matrix, and assign the current frame detection box to the corresponding historical trajectory sequence based on the optimal matching result; S214: For targets that are obscured or briefly lost and then reappear, the observation center momentum recovery mechanism is used to correct the cumulative error of the Kalman filter. S215: The corrected velocity vector is directly assigned to the velocity component in the Kalman filter state vector at the current moment, forcibly correcting the state estimation. The error covariance matrix is reset by locating the diagonal element in the matrix corresponding to the velocity state variable and resetting its value to a preset value representing high uncertainty. At the same time, the off-diagonal covariance elements related to velocity are set to zero. Finally, the target-specific time continuous skeleton coordinate sequence is output.
[0012] Furthermore, step S3 includes the following sub-steps: S31: Construct a sequence modeling layer based on the Mamba architecture, which includes an input-dependent selective state space mechanism (SSM) to dynamically generate state space parameters based on the action features at the current moment. S32: The input motion features are mapped to time step parameters, control matrix, and output matrix through a linear projection layer; S33: The continuous motion dynamics parameters are transformed into discrete state parameters using a discretization formula, and the long sequence is recursively calculated using a parallel scanning algorithm to capture the dependencies between different frames.
[0013] Furthermore, step S4 includes the following sub-steps: S41: Use a sequence-to-sequence Seq2Seq residual prediction head to predict the velocity residual at the next moment; S42: Based on the continuity constraints of physical motion and velocity residuals, the attitude at the next moment is reconstructed through integration, thus eliminating jitter in the predicted motion.
[0014] Furthermore, step S5 includes the following sub-steps: S51: Calculate the mean square error between the skeleton sequence predicted by the model and the actual observed sequence, and generate a normalized motion anomaly score. S52: When the motion anomaly score exceeds the preset safety threshold, activate the cognitive agent module and extract the RGB keyframes at the moment the anomaly occurs and input them into the visual large model (VLM). S53: Construct structured prompts, instruct the Visual Model (VLM) to identify environmental risk elements in the scene, and combine motion anomaly scores to perform logical reasoning to generate natural language warnings that include anomaly types and environmental triggers.
[0015] An abnormal behavior recognition system based on a large visual model and a cognitive agent, the system comprising: Data acquisition module: used to acquire RGB images and infrared thermal imaging (IR) images in the monitoring scene, and extract human key point sequences using the YOLOv12 network; perform temporal difference processing on the key point sequences to construct a skeleton velocity vector containing motion trends; The temporal modeling module is used to input the skeleton velocity vector into the Mamba model, adjust the model parameters using the Selective State Space (SSM) mechanism, and perform long-term temporal modeling of the action sequence. It uses a residual prediction architecture to predict the velocity change in the next frame and reconstruct the human posture, and achieves motion smoothing through physical consistency constraints. Cognitive Analysis Module: This module calculates anomaly scores based on prediction errors. When a score exceeds a preset threshold, it triggers the Cognitive Agent, which calls the Visual Model (VLM) to perform semantic analysis on the current scene environment and generates a natural language warning containing the anomaly type and environmental trigger.
[0016] The beneficial effects of this invention are: This invention introduces a selective state-space model based on the Mamba architecture, overcoming computational bottlenecks and memory limitations. Utilizing a hardware-aware parallel scanning algorithm and input-dependent dynamic parameter adjustment, it achieves linear complexity processing of action sequences lasting thousands of frames, significantly improving the efficiency and memory depth of long-term modeling. Employing a physically constrained residual velocity prediction mechanism, it abandons the traditional method of directly regressing absolute coordinates, effectively eliminating jitter in predicted actions and discontinuities in the first frame. This ensures that the generated skeleton trajectory conforms to human motion inertia, significantly reducing false alarm rates and improving the smoothness of action prediction. Innovatively, it integrates a cognitive agent decision layer, upgrading single numerical alarms to semantic-level reasoning. Combining scene risk with logical judgment provides highly interpretable natural language alarms, enhancing the environmental awareness and decision reliability of intelligent security systems. This invention's method is also applicable to other monitoring scenarios requiring long-term analysis and semantic understanding of human actions. Attached Figure Description
[0017] Figure 1 This is a flowchart of the abnormal behavior recognition method based on a large visual model and a cognitive agent according to the present invention. Detailed Implementation
[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0019] Example 1, such as Figure 1 As shown, an abnormal behavior recognition method based on a large visual model and a cognitive agent includes the following steps: S1: Acquire RGB images and infrared thermal imaging (IR) images from the monitored scene, and extract human key point sequences using the YOLOv12 network; S1 includes the following steps: S11: Construct a dual-modal acquisition front end to simultaneously acquire visible light RGB and infrared thermal IR video streams, and eliminate the timing deviation between modes through timestamp alignment technology to form an aligned dual-modal frame sequence; S12: The aligned bimodal frame sequence is input into the YOLOv12 object detection network, and the texture features of RGB and the thermal radiation features of IR are fused through the cross-modal attention mechanism; The aligned bimodal video sequence is ,in The number of image frame indices. For the corresponding image. Input the YOLOv12 network and use it to extract multi-scale feature maps containing human structural information; specifically including: S121: Construct a YOLOv12 backbone network based on the CSPDarknet cross-stage local network architecture. This backbone network consists of several cascaded CSP bottleneck modules and downsampling convolutional layers, processing the input bimodal image. and The input tensor is spliced along the channel dimension to form a composite input tensor. S122: Input the composite input tensor into the first convolutional module of the backbone network, perform initial downsampling and channel expansion on it, map the image data into an initial feature map, and complete the transformation from image space to feature space; S123: A CBS module is constructed by cascading convolutional layers, BN layers, and SiLU activation functions. The CBS module and the CSP bottleneck module perform continuous convolution operations, alternately extracting features from the initial feature map and downsampling with stride convolutions, sequentially outputting a resolution equal to the original... Figure 1 The system outputs a shallow texture feature map F1 with a resolution of 1 / 8 and a mid-level contour feature map F2 with a resolution of 1 / 16. Finally, the spatial pyramid pooling (SPPF) module is introduced to perform max pooling aggregation on the deep features, and the output is a deep semantic feature map F3 with a resolution of 1 / 32. S124: Using the feature maps F1, F2, and F3 output by the backbone network as input, a path aggregation network PANet containing a bidirectional fusion path is constructed as the neck network of YOLOv12. The deep semantic feature map F3 is upsampled by nearest neighbor interpolation and then concatenated with the mid-level feature map F2 along the channel dimension. The fused features are then upsampled again and concatenated with the shallow feature map F1 along the channel dimension to construct a top-down semantic propagation path. Subsequently, the fused features are downsampled using convolution operations and then concatenated with the deep features a second time to construct a bottom-up localization enhancement path. The final output is a fused feature pyramid composed of tensors of three different scales. , respectively corresponding to the original Figure 1 Downsampling step sizes of 8, 1 / 16, and 1 / 32 are used to enhance the network's ability to locate the edges of thermal radiation regions in infrared images.
[0020] S13: Based on the fusion results, perform human detection and pose estimation on each frame of the image, and extract the human body and pose estimation functions. Two-dimensional coordinate data of key points on an individual body form the original skeleton position sequence. ,in , To determine the maximum value of an image frame sequence, the following steps are involved: S131: The fused feature pyramid is input into the decoupled detection head of YOLOv12. This detection head is designed with a decoupled architecture, containing three parallel branches: classification, bounding box regression, and keypoint regression. In the keypoint regression branch, a direct regression strategy based on heatmap features is adopted. For each detected target anchor box, the network directly predicts the human body using the grid cells of the current feature map as a local reference system. Each skeletal keypoint is relative to the geometric center of the current feature map mesh unit. coordinate offset It covers the major joint locations of the head, torso, and limbs; The predicted normalized offset is then applied to the grid center coordinates using numerical mapping constraints. The result is multiplied by the downsampling step size of the current feature map, and the relative feature map coordinates are mapped back to the original image space to resolve the absolute pixel positions of the main joints covering the head, torso and limbs. S132: Obtain the confidence score of each prediction box. This score is directly output by the classification branch in the detection head described in S131, representing the probability value that the current predicted target belongs to the human body category. Candidate boxes with confidence scores below a preset threshold are filtered out. The remaining candidate boxes are sorted from highest to lowest confidence score. The predicted box with the highest confidence score is selected as the baseline, and redundant boxes with an Intersection over Union (IoU) exceeding the overlap threshold are removed. This process is repeated until all boxes have been processed. The target single-person object with the highest confidence score in the current scene is located, and the pose data is parsed to obtain each frame. The set of normalized pixel coordinates of all key points in the middle ; S133: According to the timestamp order of video frames , will continue The skeleton coordinates of the frames are stacked in the time dimension to construct the original skeleton position sequence. ,in This serves as the basic input for subsequently constructing the velocity vector.
[0021] S2: Perform temporal difference processing on the keypoint sequence to construct a skeleton velocity vector containing motion trends; S2 includes the following steps: S21: Perform cross-frame identity association on the detected target and construct a single person's continuous skeleton trajectory, including the following steps: S211: Establish a Kalman filter state vector for each detected target. With error covariance matrix Define the state vector as ,in Center pixel coordinates of the target bounding box The area of the bounding box. Aspect ratio, These represent the corresponding rates of change; Define the error covariance matrix P as diagonal matrix, Its diagonal elements correspond to the estimation error variance of each variable in the state vector, which are used to characterize the degree of uncertainty of the filter's estimation of the current state; Obtain the original coordinates of the target bounding box output by the YOLOv12 detection network, and record them as the top-left corner coordinates. and the coordinates of the bottom right corner The observed components in the state vector are calculated using the following geometric formula: Midpoint of the horizontal direction of the bounding box ; Midpoint of the bounding box in the vertical direction ; bounding box width ; Bounding box height ; Boundary box area ; The ratio of the bounding box width to its height. ; S212: In the current frame In order to quantify the first Kalman filter prediction box With the YOLOv12 detection frame The degree of spatial overlap between the predicted box position of the Kalman filter and the detection box position output by YOLOv12 is calculated to measure the geometric similarity between the predicted trajectory and the current observation. The higher the IoU value, the greater the probability that the two belong to the same target. Based on this, an association cost matrix C is constructed, where the matrix elements are formulated as follows:
[0022] In object detection, IoU is a metric that measures the degree of overlap between two bounding boxes. It is calculated by dividing the intersection area of the two boxes by the union area. The superscript indicates the prediction box of the Kalman filter. Indicates the first One prediction box, This represents the detection bounding box output by YOLOv12, indicated by the superscript. Indicates the first One detection box; S213: The Hungarian algorithm is used to perform global optimal bipartite graph matching on the associated cost matrix C. The algorithm iteratively transforms the cost matrix to find a set of independent element combinations in the matrix, ensuring that at most one element is selected in each row (historical prediction state) and each column (current observation data), and the sum of the costs corresponding to the selected elements reaches the global minimum. Based on this optimal matching result, the detection box of the current frame is assigned to the corresponding historical trajectory sequence. S214: For targets that are obscured or briefly lost and then reappear, the observation center momentum recovery mechanism is used to correct the cumulative error of the Kalman filter. If the target is The last time it was observed, and at the current time. Re-observed, at a time interval of Then the corrected velocity vector can be calculated directly using the observed values. :
[0023] in and These are the center coordinate observation values for the current frame and the previous observation frame, respectively; S215: Correct the velocity vector The velocity components are directly assigned to the Kalman filter state vector x at the current time step. The forced correction of state estimation involves resetting the error covariance matrix P, locating the diagonal elements in the matrix corresponding to the velocity state variables, resetting their values to preset large values representing high uncertainty, and simultaneously setting the off-diagonal covariance elements related to velocity to zero. The result is the output of a target-specific time-continuous skeleton coordinate sequence S. .
[0024] S22: To address the discontinuity issue in the first frame of motion prediction, the displacement difference of key points between two consecutive frames is calculated as a velocity feature. The original skeleton coordinate sequence is processed using a first-order backward difference method to eliminate position drift and first-frame jump phenomena caused by absolute coordinate modeling; for the first frame of the sequence... Frame (where) ), calculate its relationship with the first The displacement difference of frame -1 is used as the instantaneous velocity feature, and the calculation formula is as follows:
[0025] in, , indicating the first The velocity vector of the frame, and These are the skeleton coordinates of the current frame and the previous frame, respectively; for the starting frame of the sequence, its velocity vector is... Initialize to a zero vector, thereby transforming a non-stationary sequence of absolute positions into a stationary sequence of relative velocities.
[0026] S23: Set the absolute position coordinates of the key points With speed characteristics The features are concatenated to construct a high-dimensional action feature vector. ; Perform a concatenation operation along the feature channel dimension to merge the absolute position coordinates of the key points. With the calculated velocity vector Parallel connections are made along the feature channel dimension to construct a high-dimensional action feature vector. Its mathematical expression is:
[0027] The first two channels carry spatial attitude information, while the last two channels carry instantaneous motion trend information. These are used as inputs to the Mamba model, enabling the model to simultaneously perceive both spatial attitude and instantaneous motion trend.
[0028] S3: Input the skeleton velocity vector into the Mamba model and use the Selective State Space (SSM) mechanism to adjust the model parameters to perform long-term time-series modeling of the action sequence; In this embodiment, the action feature vector Arranged in chronological order, forming a sequence of action features. This data is used as input to the Mamba model, and after temporal feature extraction by the Mamba layer, the output is a sequence of latent variables containing context dependencies. This is used for subsequent action prediction; S3 includes the following steps: S31: Construct a sequence modeling layer based on the Mamba architecture. This layer includes an input-dependent selective state-space mechanism, Selective SSM, which breaks the parameter fixed constraint of traditional linear time-invariant systems and is used to model actions based on the characteristics of the current time step. Dynamically generate state space parameters; S32: Transform the input action features through a linear projection layer. Mapped to time step parameters Control matrix and output matrix ; Action feature vector The formula is as follows, using three independent linear projection layers:
[0029]
[0030]
[0031] in, Indicates the output dimension is A linear fully connected layer, The activation function ensures that the time step is non-negative; because , , All are inputs The function's processing parameters change for each frame in the action sequence. Through this mechanism, the model can generate larger values when sudden anomalies are detected. To update the state quickly, while generating smaller values during smooth movements. To maintain long-term sequential memory.
[0032] S33: Using the discretization formula Continuous motion dynamics parameters are transformed into discrete state parameters, and a parallel scanning algorithm is used to recursively calculate long sequences to capture the dependencies between different frames; specifically including: The continuous-time state transition matrix Initialized as a HiPPO-LegS matrix, its 6th... Line number The formula for the elements of the column is:
[0033] Combined with the time-varying step size generated by S32 , will continuous parameters Convert to discrete-time recursive parameters The calculation formula is:
[0034]
[0035] in, For the state transition matrix Identity matrices of the same dimension.
[0036] The continuous physical dynamics system is mapped to a discrete digital signal processing system adapted to the video frame rate. Based on the aforementioned discrete parameters, a hardware-aware parallel correlated scanning algorithm is employed to perform linear recursion, in order to... The hidden state of each frame can be calculated in linear time complexity. Through the output matrix Obtain the final features :
[0037]
[0038] This results in a significant improvement in inference speed and memory efficiency for long-term anomaly detection tasks; In this embodiment, the Mamba model is used for end-to-end joint training. The Mamba model, as a feature extractor, forms an end-to-end network with the residual prediction head. The training objective is to minimize the prediction error of the velocity in the next frame, and the mean squared error (MSE) loss function is constructed as follows:
[0039] in For the model based on Predicted next frame speed As a true velocity label, the gradient of the loss L with respect to the linear projection weights within the Mamba layer and the initialization parameters of the SSM is calculated using the backpropagation algorithm. The Adam optimizer is then used to update the network weights, enabling the network to learn the ability to infer future motion trends from historical action sequences.
[0040] S4: Use a residual prediction architecture to predict the velocity change in the next frame and reconstruct the human posture, and achieve smooth motion through physical consistency constraints; S4 includes the following sub-steps: S41: Employing a sequence-to-sequence (Seq2Seq) residual prediction head, the model does not directly output the absolute coordinates of the next time step, but instead predicts the velocity residual of the next time step. ; The current time when receiving the output of the Mamba sequence modeling layer in step S33. High-dimensional hidden state vector , where D is the feature channel dimension, and this vector is rich in spatiotemporal context information extracted from historical action sequences; A fully connected linear layer is established as the prediction head, and its weight matrix is defined as follows. Its bias vector is defined as The bias vector is a learnable parameter generated during network initialization that needs to be updated during training; High-dimensional state features By mapping back to the original motion feature space through linear projection, the displacement vector of the next frame relative to the current frame is regressed. The prediction formula is as follows:
[0041] in , indicating the predicted Key skeletal points are at arrive The x-axis and y-axis displacement increments within a +1 time interval. This design forces the model to learn the first derivative motion trend of the action, solving the problem of discontinuity in the first frame during prediction caused by traditional RNNs.
[0042] S42: Based on the continuity constraints of physical motion and velocity residuals, the attitude at the next moment is reconstructed through integration. The reconstruction formula is as follows: This eliminates jitter in the predicted movements, ensuring that the generated skeleton sequence is smooth and conforms to human motion inertia; Obtain the actual skeleton position coordinates at the current moment. The coordinates are derived from the recursive prediction output of the previous moment in steps S1 and S2. Based on physical inertia, human motion cannot undergo instantaneous spatial teleportation in a very short time. The Euler integral method is used to superimpose the predicted velocity residual onto the reference attitude to reconstruct the absolute coordinates for the next moment. The reconstructed formula is as follows:
[0043] in Current position The predicted speed is output in step S41; Through this residual connection structure, the system ensures that the generated skeleton trajectory is continuously differentiable in geometric space, eliminating the jitter phenomenon commonly seen in direct coordinate prediction. This makes the generated abnormal behavior conform to the real human kinematics, providing high-quality data support for the subsequent accurate identification by the Agent.
[0044] In addition, a sampling-based loss function is introduced during the training phase, allowing the model to gradually use its own generated predictions as subsequent inputs during training, thereby improving the model's robustness in long-term time-series predictions.
[0045] S5: Calculate the anomaly score based on the prediction error. When the score exceeds the preset threshold, trigger the cognitive agent, call the visual big data model (VLM) to perform semantic analysis on the current scene environment, and generate a natural language warning containing the anomaly type and environmental trigger.
[0046] S5 includes the following steps: S51: Calculate the mean squared error between the skeleton sequence predicted by the model and the actual observed sequence, and generate a normalized motion anomaly score. ; S52: When the motion anomaly score exceeds the preset safety threshold, activate the cognitive agent module and extract the RGB keyframes at the moment the anomaly occurs and input them into the visual large model (VLM). Set anomaly detection threshold This threshold is determined based on the distribution statistics of historical normal behavior data; the system monitors the normalized anomaly score output in step S51 in real time. Once detected An interrupt signal is immediately generated to activate the cognitive agent module, which is in a dormant or low-power listening state. After responding to the interruption signal, the cognitive agent accesses the circular video buffer in step S1 and locks the timestamp of the anomaly. ;by Centered on, looking back Frame and extend backward A frame is a segment of RGB visible light video containing a complete action sequence or a set of keyframes. ; Simultaneously, the corresponding infrared thermal imaging frames are extracted as auxiliary visual information, and these image data are packaged as the visual input tensor of the Visual Language Model (VLM).
[0047] S53: Construct structured prompts, instruct the Visual Model (VLM) to identify environmental risk elements in the scene, and combine motion anomaly scoring for logical reasoning to generate natural language warnings containing anomaly types and environmental triggers, specifically including: The system dynamically constructs text prompt templates containing three logical levels. These are, respectively, environment scanning instructions, attitude semantic association instructions, and causal reasoning instructions; The captured keyframe image With pre-constructed structured prompts Simultaneously input into a pre-trained visual language model (VLM), and output a text containing reasoning logic; The agent parses the text, extracts key information entities, and automatically generates the final natural language warning signaling according to the predefined standard format "[Anomaly Category: X] + [Location of Occurrence: Y] + [Inferred Cause: Z] + [Suggested Measures: W]", and pushes it to the monitoring center.
[0048] Example 2: An abnormal behavior recognition system based on a large visual model and a cognitive agent, the system comprising: Data acquisition module: used to acquire RGB images and infrared thermal imaging (IR) images in the monitoring scene, and extract human key point sequences using the YOLOv12 network; perform temporal difference processing on the key point sequences to construct a skeleton velocity vector containing motion trends; The temporal modeling module is used to input the skeleton velocity vector into the Mamba model, adjust the model parameters using the Selective State Space (SSM) mechanism, and perform long-term temporal modeling of the action sequence. It uses a residual prediction architecture to predict the velocity change in the next frame and reconstruct the human posture, and achieves motion smoothing through physical consistency constraints. Cognitive Analysis Module: This module calculates anomaly scores based on prediction errors. When a score exceeds a preset threshold, it triggers the Cognitive Agent, which calls the Visual Model (VLM) to perform semantic analysis on the current scene environment and generates a natural language warning containing the anomaly type and environmental trigger.
[0049] In one embodiment of the present invention, a binocular camera module integrating a visible light camera and an infrared thermal imager is deployed in a hospital inpatient corridor. The corridor is dimly lit at night, and the floor has just been cleaned, with yellow warning signs placed on it. The device uses timestamp synchronization technology to collect synchronous dual-modal video data of this scene around the clock and transmits it in real time to the "abnormal behavior recognition system" based on the present invention.
[0050] During a nighttime monitoring task, a nurse on duty (Personnel A) was making routine rounds when a patient in a hospital gown (Personnel B) staggered out of the ward and suddenly slipped and fell on a wet surface. The system aims to accurately identify the anomaly and provide an interpretable alarm in this low-light, high-sudden-event environment. The system's workflow is as follows: Step S1: The system performs multimodal perception and velocity feature construction.
[0051] At the beginning of the mission, both personnel A and B were in motion. Due to the dim lighting in the corridor, the YOLOv12 front-end of the system primarily relied on the clear outlines provided by infrared thermal imaging for human detection. The system accurately extracted the key point sequences of the skeletons of the two individuals and immediately performed temporal difference calculations. Unlike traditional methods that only focus on position, this system constructed high-dimensional features including "instantaneous velocity vectors," keenly capturing the slight instability in personnel B's gait, while personnel A's velocity characteristics remained stable.
[0052] Step S2: The Mamba model performs selective temporal modeling of sudden fall actions.
[0053] As person B steps onto the slippery area and loses balance, their skeletal structure undergoes a violent downward acceleration. Traditional models, due to their fixed time steps, often fail to respond promptly to such millisecond-level rapid changes, leading to the loss of keyframe information or excessive smoothing. This invention utilizes a selection mechanism, where the model dynamically generates a larger time step parameter based on the magnitude of the current input's violent movement. This allows the system to record the subtle postural changes of person B at the moment of falling with high-precision discretization encoding, while for nurse A walking normally next to her, the model maintains a small step size to maintain long-term temporal memory, thus achieving dynamic attention to input dependence.
[0054] Step S3: The system uses the residual prediction head to reconstruct physically consistent motion trajectories.
[0055] The model predicts the change in velocity of person B using a Seq2Seq residual prediction head. This is achieved through an integral reconstruction formula. The system generated a smooth fall trajectory that conformed to the laws of gravitational acceleration. The system calculations revealed that the predicted trajectory of person B differed significantly from the prior pattern of normal upright walking, resulting in a large mean square error (MSE) and thus generating an extremely high motion anomaly score. .
[0056] Step S4: The cognitive agent intervenes to perform environmental semantic reasoning and decision-making.
[0057] In this embodiment, a high anomaly score triggers the cognitive agent to capture the RGB keyframes at the moment of the fall and call the visual large model (VLM) to perform joint "environment-attitude" analysis.
[0058] VLM identified key semantic elements in the image: "falling posture," "patient's clothing," and the adjacent "yellow slippery warning sign." Based on this, the Agent performed logical reasoning, ruling out the possibilities of "voluntary lying down" or "fainting due to illness," and finally generated a structured natural language alert: "[High-risk anomaly] A patient was detected to have fallen in corridor 3. Environmental analysis shows that the slippery ground is the direct cause. Please send someone to help immediately." This embodiment fully demonstrates that the present invention can capture instantaneous anomalies in complex scenarios with low light and high suddenness through dynamic modeling of Mamba, and provide logical explanations through semantic reasoning of the Agent, effectively solving the core pain points of behavior recognition systems that "cannot see fast actions" and "do not understand environmental logic".
[0059] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.
Claims
1. An abnormal behavior recognition method based on a large visual model and a cognitive agent, characterized in that, Includes the following steps: S1: Acquire RGB images and infrared thermal imaging (IR) images from the monitored scene, and extract human key point sequences using the YOLOv12 network; S2: Perform temporal difference processing on the keypoint sequence to construct a skeleton velocity vector containing motion trends; S3: Input the skeleton velocity vector into the Mamba model and use the Selective State Space (SSM) mechanism to adjust the model parameters to perform long-term time-series modeling of the action sequence; S3 includes the following steps: S31: Construct a sequence modeling layer based on the Mamba architecture, which includes an input-dependent selective state space mechanism (SSM) to dynamically generate state space parameters based on the action features at the current moment. S32: The input motion features are mapped to time step parameters, control matrix, and output matrix through a linear projection layer; S33: The continuous motion dynamics parameters are transformed into discrete state parameters using the discretization formula, and the long sequence is recursively calculated using a parallel scanning algorithm to capture the dependencies between different frames. S4: Use a residual prediction architecture to predict the velocity change in the next frame and reconstruct the human posture, and achieve smooth motion through physical consistency constraints; S5: Calculate the anomaly score based on the prediction error. When the score exceeds the preset threshold, trigger the cognitive agent, call the visual big data model (VLM) to perform semantic analysis on the current scene environment, and generate a natural language warning containing the anomaly type and environmental cause. S5 includes the following steps: S51: Calculate the mean square error between the skeleton sequence predicted by the model and the actual observed sequence, and generate a normalized motion anomaly score. S52: When the motion anomaly score exceeds the preset safety threshold, activate the cognitive agent module and extract the RGB keyframes at the moment the anomaly occurs and input them into the visual large model (VLM). S53: Construct structured prompts, instruct the Visual Model (VLM) to identify environmental risk elements in the scene, and combine motion anomaly scores to perform logical reasoning to generate natural language warnings that include anomaly types and environmental triggers.
2. The abnormal behavior recognition method based on a large visual model and a cognitive agent according to claim 1, characterized in that, S1 includes the following steps: S11: Construct a dual-modal acquisition front end to simultaneously acquire visible light RGB and infrared thermal IR video streams, and eliminate the timing deviation between modes through timestamp alignment technology to form an aligned dual-modal frame sequence; S12: The aligned bimodal frame sequence is input into the YOLOv12 object detection network, and the texture features of RGB and the thermal radiation features of IR are fused through the cross-modal attention mechanism; S13: Based on the fusion results, perform human detection and pose estimation on each frame of the image, extract two-dimensional coordinate data containing human key points, and form the original skeleton position sequence.
3. The abnormal behavior recognition method based on a large visual model and a cognitive agent according to claim 1, characterized in that, The aligned bimodal frame sequence is input into the YOLOv12 object detection network, and the RGB texture features and IR thermal radiation features are fused through a cross-modal attention mechanism. It includes the following steps: S121: Construct the backbone network of YOLOv12 based on the CSPDarknet cross-stage local network architecture. The backbone network consists of several cascaded CSP bottleneck modules and downsampling convolutional layers, which stitch the input bimodal images in the channel dimension to form a composite input tensor. S122: Input the composite input tensor into the first convolutional module of the backbone network, perform initial downsampling and channel expansion on it, map the image data into an initial feature map, and complete the transformation from image space to feature space; S123: A CBS module is constructed by cascading convolutional layers, BN layers, and SiLU activation functions. The CBS module and the CSP bottleneck module are used to perform continuous convolution operations to extract features and perform stride convolution downsampling on the initial feature map. The shallow texture feature map F1 with a resolution of 1 / 8 of the original image and the mid-level contour feature map F2 with a resolution of 1 / 16 are output sequentially. Finally, the spatial pyramid pooling SPPF module is introduced to perform max pooling aggregation on the deep features and output the deep semantic feature map F3 with a resolution of 1 / 32. S124: Using the feature maps F1, F2, and F3 output by the backbone network as input, a path aggregation network PANet containing a bidirectional fusion path is constructed as the neck network of YOLOv12. The deep semantic feature map F3 is upsampled by nearest neighbor interpolation and then concatenated with the mid-level feature map F2 along the channel dimension. The fused features are then upsampled again and concatenated with the shallow feature map F1 along the channel dimension to construct a top-down semantic propagation path. Subsequently, the fused features are downsampled using convolution operations and then concatenated with the deep features a second time to construct a bottom-up localization enhancement path, outputting a fused feature pyramid containing rich semantic and location information.
4. The abnormal behavior recognition method based on a large visual model and a cognitive agent according to claim 3, characterized in that, Based on the fusion results, human detection and pose estimation are performed on each frame of the image, and two-dimensional coordinate data containing human key points are extracted to form the original skeleton position sequence, including the following steps: S131: The fused feature pyramid is input into the decoupled detection head of YOLOv12. The detection head is designed as a decoupled architecture, which includes three parallel branches: classification, bounding box regression and key point regression. In the key point regression branch, a direct regression strategy based on heatmap features is adopted. For each detected target anchor box, the network uses the grid cell of the current feature map as a local reference system to directly predict the coordinate offset of the human skeleton key points relative to the geometric center of the grid cell of the current feature map, covering the main joints of the head, torso and limbs. S132: Based on the confidence score of each predicted box output by the classification branch in the detection head, it represents the probability value that the current predicted target belongs to the human body category; S133: Stack the skeleton coordinates of consecutive frames in the time dimension according to the timestamp order of the video frames to construct the original skeleton position sequence.
5. The abnormal behavior recognition method based on a large visual model and a cognitive agent according to claim 4, characterized in that, S2 includes the following steps: S21: Perform cross-frame identity association on the detected target and construct a single person continuous skeleton trajectory; S22: To address the discontinuity issue in the first frame of motion prediction, the displacement difference of key points between two consecutive frames is calculated as a velocity feature. S23: Concatenate the absolute position coordinates of the key points with the velocity features to construct a high-dimensional motion feature vector.
6. The abnormal behavior recognition method based on a large visual model and a cognitive agent according to claim 5, characterized in that, The step of performing cross-frame identity association on the detected target and constructing a single person's continuous skeleton trajectory includes the following steps: S211: Establish the Kalman filter state vector and error covariance matrix for each detected target; S212: In the current frame, calculate the intersection-union ratio between the predicted box position of the Kalman filter and the detection box position output by YOLOv12, and construct the association cost matrix; S213: Use the Hungarian algorithm to perform global optimal bipartite graph matching on the correlation cost matrix, and assign the current frame detection box to the corresponding historical trajectory sequence based on the optimal matching result; S214: For targets that are obscured or briefly lost and then reappear, the observation center momentum recovery mechanism is used to correct the cumulative error of the Kalman filter. S215: The corrected velocity vector is directly assigned to the velocity component in the Kalman filter state vector at the current moment, forcibly correcting the state estimation. The error covariance matrix is reset by locating the diagonal element in the matrix corresponding to the velocity state variable and resetting its value to a preset value representing high uncertainty. At the same time, the off-diagonal covariance elements related to velocity are set to zero. Finally, the target-specific time continuous skeleton coordinate sequence is output.
7. The abnormal behavior recognition method based on a large visual model and a cognitive agent according to claim 1, characterized in that, S4 includes the following steps: S41: Use a sequence-to-sequence Seq2Seq residual prediction head to predict the velocity residual at the next moment; S42: Based on the continuity constraints of physical motion and velocity residuals, the attitude at the next moment is reconstructed through integration, thus eliminating jitter in the predicted motion.
8. A system for identifying abnormal behavior based on a large visual model and a cognitive agent as described in any one of claims 1-7, characterized in that, The system includes: Data acquisition module: used to acquire RGB images and infrared thermal imaging (IR) images in the monitoring scene, and extract human key point sequences using the YOLOv12 network; perform temporal difference processing on the key point sequences to construct a skeleton velocity vector containing motion trends; The temporal modeling module is used to input the skeleton velocity vector into the Mamba model, adjust the model parameters using the Selective State Space (SSM) mechanism, and perform long-term temporal modeling of the action sequence. It uses a residual prediction architecture to predict the velocity change in the next frame and reconstruct the human posture, and achieves motion smoothing through physical consistency constraints. Cognitive Analysis Module: This module calculates anomaly scores based on prediction errors. When a score exceeds a preset threshold, it triggers the Cognitive Agent, which calls the Visual Model (VLM) to perform semantic analysis on the current scene environment and generates a natural language warning containing the anomaly type and environmental trigger.