A multi-modal human abnormality detection method based on large model semantics and kinematic coordinates

Abstract

The application discloses a kind of multi-modal human body anomaly detection methods based on semantic and kinematic characteristics, to accurately identify the unconventional or dangerous behavior of human by combining unsupervised error reconstruction with semantic reasoning based on visual language model.The method is divided into two branches: in time-kinematic branch, multi-granularity spatiotemporal graph is extracted, and mask Transformer autoencoder is used to obtain physical kinematic level anomaly by calculating reconstruction error.Secondly, in space-semantic branch, multi-dimensional semantic anomaly is quantified by inputting multi-modal data to visual language model, effectively alleviating the limitations of pure kinematics.Experimental results show that the proposed multi-modal dual-view framework performs excellently in anomaly detection tasks in complex scenes, significantly improves the robustness of the system compared with existing methods.

Description

Technical Field

[0001] This invention relates to graph convolutional networks, visual language models, and their applications. Specifically, it relates to a multimodal bi-branch anomaly detection method based on skeletal graphs, semantic and kinematic coordinates of large models, which aims to accurately determine human anomalies by using kinematic coordinate data and semantic information of large models. Background Technology

[0002] Anomaly human pose detection is a crucial task in computer vision, aiming to identify potential dangers or unusual behavioral patterns in complex scenes. This is of great significance for intelligent security and health protection. In existing research, skeletal sequence-based methods have shown strong robustness to influencing factors such as lighting changes and complex backgrounds, and have become the mainstream approach. Furthermore, due to the unpredictability of anomalous events such as abnormal poses, current mainstream methods typically employ unsupervised learning. Unsupervised learning means that the model is trained using data without anomalies, while samples with high reconstruction errors are judged as anomalous during the inference and validation phases. In human anomaly pose detection, variants of traditional graph convolutional networks or models like the spatiotemporal Transformer have demonstrated remarkable capabilities. However, these methods rely entirely on physical coordinates for pure kinematic modeling. This drawback often leads to misjudgments when faced with actions that have the same motion trajectory but completely different semantic properties. For example, taking a book out of a backpack and taking a controlled knife in a public place have almost identical changes in skeletal topology, but the nature of the actions is completely different. Meanwhile, in recent years, a large number of works on video anomaly inference based on large models have emerged. These methods have demonstrated powerful zero-shot inference capabilities. However, these methods typically convert video frame images containing dense pixel information into discrete text descriptions for processing, leading to the loss of high-frequency and micro-detail information. Furthermore, the illusion problem of large models can also affect the accuracy of anomaly detection to some extent. Therefore, a new method is needed to address the lack of global semantics in kinematic coordinates and the tendency of global semantics to lose high-frequency and micro-detail details, while also solving the problem of cross-modal data alignment. Summary of the Invention

[0003] In view of this, and to overcome the shortcomings of existing technologies, this invention provides a multimodal, dual-view human abnormal pose detection method. This method aims to achieve a cross-modal combination of unsupervised reconstruction based on multi-granularity skeletons and large-model visual semantic reasoning, fully utilizing kinematic coordinate information and large-model global semantic information, thereby significantly improving the performance of human abnormal pose detection.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A multimodal human anomaly detection method based on large model semantics and kinematic coordinates, characterized by the following steps:

[0006] 1. Spatiotemporal Skeleton Extraction and Preprocessing: To construct a robust network input, a multi-target tracking algorithm with motion compensation was employed. Spatially, the bounding box was adaptively expanded to prevent limb truncation. Subsequently, a centroid matching strategy was combined to extract human key points to filter out background noise interference. Finally, coordinate normalization, masking, and sequence slicing strategies were used to process the coordinate data to obtain the final normalized coordinates. With mask matrix .

[0007] 2. Construction and Feature Extraction of Multi-Granularity Spatiotemporal Maps: Based on the spatiotemporal map of each character, three spatiotemporal skeleton maps are constructed at different scales: for joint-level spatiotemporal maps... Preserve the original topology; for limb-level spatiotemporal graphs The 18 joints are aggregated into six limb-level nodes (head, torso, and limbs) using a topological pooling matrix; for the body-level spatiotemporal graph... Its nodes include only neck, left hip, and right hip nodes. Simultaneously, to enhance the spatial representation, the angle between the torso vector (from the center of the pelvis to the neck) and the gravity axis is introduced, and its sine and cosine values ​​are concatenated into the node features to enhance the features. For each granularity, features are extracted from the spatiotemporal skeleton map using independent channel-level topology refinement graph convolutions.

[0008] 3. Cross-granularity attention fusion and nonlinear self-expressive reconstruction: Node features are flattened along the time axis to eliminate spatial dimension, and then unified to the same target dimension through linear mapping. After obtaining sequence features at three granularities, they are concatenated along the channel direction to obtain concatenated features. After obtaining the splicing features, due to the expansion of its channel dimension, therefore... The data is fed into an explicit information bottleneck layer for feature compression. Finally, the mask is converted into a penalty term and a multi-head self-attention mechanism is introduced to obtain a cross-granularity comprehensive representation sequence. .Will The sequence is fed into a Transformer encoder to learn a latent representation of the normal kinematic pattern, and then reconstructed using a Transformer decoder based on this latent representation. Finally, the initial anomaly score of the time-kinematic branch is obtained based on the mean squared error. .

[0009] 4. Multi-granularity semantic aggregation of the visual language model: The enhanced keyframes and carefully designed cue words are fed into the visual language model (VLM), which outputs integer evaluation values ​​from 1 to 10 in these three aspects respectively. These evaluation values ​​are then weighted and normalized outside the VLM system to obtain the anomaly scores for the spatial-semantic branches. .

[0010] 5. Dual-Stream Score Fusion and Anomaly Decision-Making: The anomaly scores from the two branches exhibit inconsistent scales and a significant misalignment in the definition of the normal state. A probabilistic mapping mechanism based on a right-tailed semi-normal distribution is used to process the anomaly scores in the time-kinematic branch, truncating normal fluctuations below the mean and strictly shifting and aligning its probability space to a zero-based starting point. Based on the weighted fusion of the two branches, a comprehensive anomaly score for each person in the video frame is obtained. Finally, a frame-level pooling scheme is used to obtain the final anomaly score for the current frame. And compare it with the set threshold.

[0011] Preferably, in step 2, when extracting features from the spatiotemporal graphs of the three granularities, adaptive convolution is used to better aggregate the features of the corresponding spatiotemporal graphs.

[0012] Preferably, in step 3, a sliding window is used to slice the bone sequence, while maintaining a large overlap step size. This approach expands the diversity of the training data while preserving the original motion frequencies. After the sliding window mechanism and global mapping, a smoothed error score can be obtained. .

[0013] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multimodal human anomaly detection method based on large model semantics and kinematic coordinates. The beneficial effects of the present invention are as follows:

[0014] 1. Multimodal dual-view anomaly detection: Couples multi-granularity skeleton nonlinear self-expression reconstruction with VLM-driven high-level semantic reasoning, bridging the gap between skeleton-based models lacking contextual semantics and VLMs lacking kinematic details.

[0015] 2. Multi-granularity attention combination mechanism: It uses a cross-granularity attention mechanism to model the topology of long program sequences, which achieves higher sensitivity to subtle, local micro kinematic anomalies.

[0016] 3. Harmonize semantic alignment across different metric spaces: The right-tailed semi-normal mapping mechanism is used to calibrate and fuse heterogeneous bi-branch anomaly scores, effectively mitigating misclassification of actions that exhibit similar kinematic trajectories but different contextual semantics, and enhancing detection robustness in complex scenes. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is the overall flowchart of the model of the present invention; Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] This invention discloses a multimodal human anomaly detection method based on semantics and kinematics, the specific steps of which include the following:

[0021] Step 1: Spatiotemporal Graph Construction and Initialization. Skeletal point data is obtained from public datasets (such as Ubnormal, ShanghaiTech, etc.). The BoT-SORT algorithm with motion compensation is employed and integrated with YOLOv8 for multi-target tracking. OpenPose is used to extract 18 human keypoints, and a spatiotemporal skeleton graph with three granularities is constructed based on the keypoint data and an attention mask. Specifically, a dynamic scale scaling factor is introduced to normalize the coordinates. The calculation formula is defined as follows: .

[0022] The final coordinate space normalization formula is defined as follows: .

[0023] Step 2: Multi-granularity spatiotemporal graph feature extraction. Based on the 18-joint topological skeleton graph corresponding to the COCO model, three spatiotemporal skeleton graphs with complementary properties were constructed in parallel at different scales: the joint-level spatiotemporal graph retains the original 18-node topology; the limb-level spatiotemporal graph aggregates the 18 joints into 6 high-order limb-level nodes (head, trunk, and limbs); and the body-level spatiotemporal graph includes only three core trunk nodes: neck, left hip, and right hip. It introduces the angle between the trunk vector from the pelvic center to the neck and the gravity axis (vertical axis), and concatenates the sine (sin) and cosine (cos) values ​​of this angle into the node features to obtain the enhanced augmented features. For each granularity, an independent channel-level topology-optimized graph convolutional network is used to perform deep spatiotemporal feature extraction on the spatiotemporal skeleton graph. Specifically, the single-layer spatiotemporal graph convolution calculation formula is defined as: .

[0024] After multi-granularity convolution extraction, high-order graph feature tensors of each granularity are obtained, and the final output channel number of the three granularities is kept consistent after adjustment.

[0025] Step 3: Graph-to-Sequence Mapping and Cross-Granularity Attention Fusion. Because the topological spatial dimensions of joint-level, limb-level, and body-level spatiotemporal graphs are completely inconsistent, direct cross-modal or cross-scale matrix operations are not possible. The node features at each granularity are flattened on the time axis to completely eliminate the spatial topological dimension. Then, a fully connected linear mapping layer unifies the feature vector dimensions of each granularity to a completely identical target high-dimensional space. .

[0026] After obtaining standard sequence features at three granularities, they are concatenated and fed into the information bottleneck layer. The mathematical expression for this is: .

[0027] To achieve deep, dynamic, cross-granularity semantic interaction, a multi-head self-attention mechanism was subsequently introduced and combined with a mask to process features. The process can be represented as follows: .

[0028] Step 4: Calculate the kinematic branch anomaly score. Combine the cross-granularity representation sequence. The data is fed into a standard Transformer encoder to learn a low-dimensional latent manifold representation of the normal kinematic pattern. Then, a symmetric Transformer decoder is used to attempt to reconstruct the original synthetic representation sequence from this low-dimensional latent representation. Finally, the initial anomaly score for the kinematic branches is defined based on the mean squared error. Because a sliding window slicing mechanism is used, a one-dimensional Gaussian window is used to perform distance-weighted calculation of the frame scores within a segment to obtain a global smoothing error score.

[0029] Step 5: Calculation of Spatial-Semantic Branch Anomaly Score Based on Visual Language Model (VLM). By designing highly structured prompts within the system, VLM is forced to rationally deconstruct and analyze the target person at three independent and progressively fine-grained levels. The prompts restrict VLM to provide discrete integer ratings from 1 to 10 for each of the three aspects. Finally, these structured texts output by the large model are intercepted outside the VLM system and fed into a classical mathematical computation layer for weighted normalization to obtain the spatial-semantic branch anomaly score. : .

[0030] Step Six: Dual-stream Score Fusion and Anomaly Decision-Making. For anomalous scores in the kinematic branch, normal reconstruction fluctuations below the mean μ are forcibly truncated and filtered out using a right-tailed semi-normal mapping function. For the remaining scores, their probability space is strictly translated, aligned, and scaled starting from μ. .

[0031] Finally, the anomaly scores from the two branches are weighted and fused, and then frame-level max pooling is used to obtain the final anomaly score for the current frame. This score is then compared with a preset threshold to determine whether an anomaly has occurred in the video.

[0032] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A multimodal human anomaly detection method based on large model semantics and kinematic coordinates, characterized in that, The specific steps include the following: Data input and preprocessing module: Input video sequence and use motion compensation algorithm and centroid matching strategy to process video frames to generate augmented keyframes and multi-granularity spatiotemporal graphs, and then align them for dual-branch processing; Cross-granularity attention fusion module: Uses mask attention mechanism to process features of multi-granularity spatiotemporal graphs, and maps the features after the battle to a unified target dimension through linear mapping; Nonlinear self-expression reconstruction module: The temporal-kinematic branch based on sequence reconstruction adopts an unsupervised paradigm. The comprehensive representation sequence is fed into the Transformer encoder to learn the latent representation of the normal kinematic pattern. Then, the Transformer decoder is used to reconstruct the sequence based on the latent representation to obtain the preliminary score of the temporal-kinematic branch. VLM multi-granularity semantic fusion module: It adopts a multi-granularity semantic aggregation method to ensure the interpretability of spatial-semantic branch anomaly scores. By designing structured prompts, VLM outputs integer evaluation values ​​from 1 to 10 for three aspects. Finally, weighted normalization is performed outside the VLM system. Mapping and weighted fusion module: The right-tailed semi-normal mapping cuts off the normal fluctuations below the mean and strictly shifts and aligns its probability space to start from 0. Finally, the two branches are weighted and fused.

2. The multimodal human anomaly detection method based on large model semantics and kinematic coordinates according to claim 1, characterized in that, The formula for calculating the multi-head attention of the mask is: .

3. The multimodal human anomaly detection method based on large model semantics and kinematic coordinates according to claim 1, characterized in that, The formula for calculating the time-kinematic branch fraction is as follows: .

4. The multimodal human anomaly detection method based on large model semantics and kinematic coordinates according to claim 1, characterized in that, The weighted normalization strategy is denoted as: .

5. The multimodal human anomaly detection method based on large model semantics and kinematic coordinates according to claim 1, characterized in that, The formula for the right-tailed semi-normal mapping is: .

Images