Method for detecting abnormal behavior of figure in video based on emotion enhancement

Abstract

The invention discloses a video figure abnormal behavior detection method based on emotion enhancement, which innovatively introduces an emotion feature as a core supplement, and provides deep semantic interpretation for abnormal detection by analyzing an emotion motivation (such as anger, panic and attack tendency) behind a behavior. The emotion features provide high-level semantic constraints for the motion features (flow features) and the appearance features (RGB features), and the motion features and the appearance features provide direct and morphological abnormal clues; through the multi-level feature information fusion, the video anomaly detection return method based on emotion enhancement can make accurate judgment for different types of abnormal behaviors (such as irregular actions, emergencies, abnormal crowd behaviors and the like) in a complex video scene, so that the overall anomaly detection performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video detection, and in particular to a method for detecting abnormal behavior of people in videos based on emotion enhancement. Background Art

[0002] Existing video anomaly detection methods are primarily limited by how the dataset is annotated. These methods typically fall into two categories: using only normal samples for training, and using anomaly samples with video-level annotations. Based on these data annotation methods, video anomaly detection methods can be generally categorized into two types: one-class classification (OCC) and weakly supervised anomaly detection.

[0003] In the one-class classification (OCC) method, the training data only contains normal samples. In other words, the model only learns the characteristics of normal behavior and constructs anomaly detection boundaries based on these characteristics. The core idea of ​​this method is that when new data exceeds the range of normal samples, the model will classify it as anomaly. However, since no abnormal samples are exposed during the training process, the OCC method often has difficulty in effectively identifying unseen abnormal situations. Therefore, the limitation of the OCC method is that it can only recognize normal behavior in the training set and performs poorly for abnormal behavior outside this boundary. To solve this problem, researchers have proposed a method to generate pseudo-anomaly samples from normal training data in an attempt to make up for the lack of abnormal samples in the training data. However, the effectiveness of this method depends on the robustness of the pseudo-anomaly generator, so its performance is limited by the quality of the generator.

[0004] Unlike OCC methods, weakly supervised anomaly detection methods use both normal and abnormal samples during training. However, the annotation of abnormal samples is limited to the video level, rather than detailed annotation of each frame. Several studies have proposed applying self-supervised learning and multi-task learning at the object level to detect anomalies in videos. These methods enhance the model's learning capabilities by introducing multiple auxiliary tasks, typically including four proxy tasks, three of which are based on self-supervised learning and one based on knowledge distillation. Ultimately, the model determines the final anomaly score for each object in the video frame by calculating and averaging the anomaly scores generated by each proxy task. Through multi-task learning, the model integrates the learning results of multiple tasks, effectively improving the accuracy and robustness of anomaly detection.

[0005] To further improve anomaly detection performance, some studies have proposed detecting abnormal behaviors through fine-grained spatiotemporal location, employing a dual-branch neural network model. This model can simultaneously extract both motion and spatiotemporal features from videos. However, while this model can identify certain specific actions, it often misclassifies them as abnormal when these actions occur at inappropriate spatiotemporal locations. This demonstrates that modeling motion features in videos alone is insufficient for anomaly detection. Therefore, existing methods have not effectively addressed the problem of human-centric abnormal behavior detection and require further research and improvement.

[0006] Existing video anomaly detection technologies still have some significant flaws when facing complex scenarios in the real world, including the following aspects:

[0007] A. Lack of sufficient anomaly training data: In real-world applications, the frequency of abnormal events is much lower than that of normal events, which often leads to data imbalance in anomaly detection tasks. Many existing methods rely on limited labeled data, especially when labeled anomaly data is very scarce. This makes it difficult to train models with good generalization capabilities.

[0008] B. Dependence on Anomalous Context: Existing anomaly detection methods mostly focus on temporal comparison of video content, i.e., identifying anomalies by identifying temporal differences from normal video content. These methods often assume that anomalies are caused by abnormalities in object appearance or motion, while ignoring the underlying drivers behind these abnormal behaviors. For example, distinguishing abnormal behaviors solely by motion characteristics (e.g., similar movements such as running, injured running, and running away) is insufficient, as the driving motivations of these behaviors may differ.

[0009] C. Ignoring the emotional drivers of behavior: Existing methods often focus on modeling the appearance or motion characteristics of videos, neglecting the emotional factors behind abnormal behavior. In reality, emotions play a crucial role in driving abnormal behavior. For example, strong emotions such as anger, violence, or aggression can be the primary driving force behind abnormal behavior, but existing methods fail to effectively account for this. Emotions are closely related to human behavior, perception, and physiological responses. Failure to incorporate emotional factors into models can lead to inaccurate abnormal behavior identification.

[0010] D. Inability to capture complex spatiotemporal information: Although some methods have adopted modeling based on spatiotemporal information (such as motion characteristics and background changes in videos), existing technologies are still limited in their performance when dealing with complex spatiotemporal information and emotional changes. For example, spatiotemporal-based video transformers and self-attention mechanisms, while performing well in certain tasks, lack effective integration and interaction of emotional and motion features when dealing with multi-level spatiotemporal dynamics and emotion-driven behaviors.

[0011] In order to solve the above technical problems, the present invention provides a method for detecting abnormal behavior of people in videos based on emotion enhancement.

[0012] A method for detecting abnormal behavior of people in a video based on emotion enhancement includes the following steps:

[0013] Step 1: Extract the emotional features H of the characters from the video data E ;

[0014] Step 2: Decompose each RGB image frame of the video data into N non-overlapping patches, and use the pre-trained TimeSformer model to extract features from the patches in each RGB image frame of the video; then perform a separate time-space attention operation on the features of the extracted RGB image to obtain the RGB feature H that combines the relationship between the time dimension and the spatial dimension. R ;

[0015] Step 3: Divide the video data into several continuous, non-overlapping small video segments, each containing T frames; for each video segment, use the pre-trained optical flow estimation model to calculate the optical flow between adjacent frames to generate an optical flow map sequence containing T-1 optical flow maps; each optical flow map contains horizontal and vertical optical flow components, which are represented as a tensor of size H×W×3 after downsampling or convolutional encoding; then decompose each optical flow map in the encoded optical flow map sequence into N non-overlapping patches, and use the pre-trained TimeSformer model to extract features from the patches in the optical flow map; then perform a separated time-space attention operation on the extracted optical flow map features to obtain the flow feature H that combines the relationship between the time dimension and the spatial dimension. F ;

[0016] Step 4: Adopt cross-modal interaction strategy to fuse flow features H F and RGB feature H R , get the RGB features after flow enhancement

[0017] Step 5: Adopt cross-modal interaction strategy to fuse flow features H F and emotional perception features H E , get the emotion-enhanced flow features

[0018] Step 6: Enhance the RGB features based on the obtained flow and emotion-enhanced flow characteristics Generate anomaly scores through the video character abnormal behavior detection model.

[0019] In the above technical solution, the processing method of step 1 is as follows:

[0020] Step 1.1: Use the object detection model to detect the bounding box information of the person in the video frame and identify the position of the person in each frame of the video image; then input the bounding box information extracted by the object detection model into the OpenPose pose recognition model to extract the key point information of the person's pose;

[0021] Step 1.2: Generate posture features using the posture feature generation model based on the extracted posture key point information. The posture feature generation model consists of ST-GCN, a convolutional layer, and three fully connected layers.

[0022] Step 1.3: Use an autoencoder model to learn and process the latent emotion space of the obtained posture features. The autoencoder model compresses the posture features into a lower-dimensional emotion representation vector, that is, the character's emotion feature H is obtained. E .

[0023] In the above technical solution, the total loss of the video character abnormal behavior detection model includes: binary classification loss, reconstruction loss and sorting loss.

[0024] In the above technical solution, the loss function of the sorting loss is expressed as follows:

[0025]

[0026] Among them, B a and B n are positive sample set and negative sample set respectively, a i and n i are the individual in the positive sample set and the negative sample set respectively, f θ (a i ) and f θ (n i ) is the abnormality score for the positive sample set and the negative sample set.

[0027] The present invention has the following beneficial effects:

[0028] This invention innovatively introduces emotional features as a core supplement, providing deep semantic interpretation for anomaly detection by analyzing the emotional motivations behind behaviors (such as anger, panic, and aggressive tendencies). Emotional features provide high-level semantic constraints for motion features (flow features) and appearance features (RGB features), while motion features and appearance features provide direct, morphological clues to anomalies. Through this multi-level feature information fusion, the emotion-enhanced video anomaly detection system can accurately judge different types of abnormal behaviors (such as irregular movements, emergencies, abnormal crowd behavior, etc.) in complex video scenes, thereby effectively improving the overall anomaly detection performance.

[0029] This method processes video frames by separately calculating temporal and spatial attention. Spatial attention focuses on details within a single frame, such as object edges and textures, improving understanding of static features. Temporal attention focuses on dynamic changes between frames, such as motion trajectories, improving understanding of dynamic features. This separation mechanism enables the model to more finely extract spatial and temporal features, thereby more accurately capturing key features in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the process of the present invention.

[0031] For ordinary technicians in this field, other relevant drawings can be obtained based on the above drawings without any creative work. DETAILED DESCRIPTION

[0032] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described to better illustrate the principles of the invention and its practical application, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for specific applications.

[0033] The present invention provides a method for detecting abnormal behavior of people in videos based on emotion enhancement, which is as follows:

[0034] Step 1: Extract the emotional features H of the characters from the video data E .

[0035] Specifically, step 1 is handled as follows:

[0036] Step 1.1: Use an object detection model (preferably a YOLOv8 model) to detect the bounding box information of the person in the video frame, thereby identifying the position of the person in each frame of the video image; then input the person bounding box information extracted by the YOLOv8 object detection model into the OpenPose pose recognition model to extract the person's pose key point information.

[0037] Step 1.2: Generate pose features using a pose feature generation model based on ST-GCN (i.e., spatiotemporal graph convolutional network). The pose feature generation model consists of ST-GCN, a convolutional layer, and three fully connected (FC) layers. The d-dimensional vector of the convolutional layer is used as the pose feature E(x).

[0038] Step 1.3: Use an autoencoder model to learn and process the latent emotion space of the obtained posture feature E(x). The autoencoder model compresses the posture feature into a lower-dimensional emotion representation vector, that is, the character's emotion feature H is obtained. E .

[0039] Specifically, the input of the autoencoder model is a d-dimensional posture feature containing n samples. In the encoder part, the input d-dimensional posture feature E(x)∈R d Embedded into a three-dimensional latent space through a fully connected layer, this means that the original d-dimensional input is compressed into a 3-dimensional potential emotion representation vector; the task of the decoder is to reconstruct the potential emotion representation vector z output by the encoder back to the original posture feature. Assume z∈R 3 , will pass through a fully connected layer of the decoder and be restored to the original dimension d. In order to train the autoencoder model, the difference between the input and the reconstructed output is usually expressed using the mean square error (MSE) as the loss function, which represents the difference between the original input E(x) and the reconstructed output. By minimizing the reconstruction error, the potential emotion representation obtained after compression by the encoder can effectively reconstruct the original input, thereby learning an efficient emotion representation.

[0040] Step 2: Decompose each RGB image frame of the video data into N non-overlapping patches. Use the pre-trained TimeSformer model to extract features from the patches in each RGB image frame of the video. Then, perform a separate time-space attention operation on the features of the extracted RGB image to obtain the RGB feature H that combines the relationship between the time dimension and the spatial dimension. R .

[0041] Specifically, the steps are as follows:

[0042] Step 2.1: Decompose each RGB image frame of the video data into N non-overlapping patches. Use the pre-trained TimeSformer model (i.e., the Time-Space Transformer model, a pure Transformer architecture designed specifically for video understanding tasks) to extract features from the patches in each RGB image frame of the video.

[0043] Specifically, for RGB data X∈R H×W×3×F , represents the F frames of RGB images sampled from the original video, and the size of each frame is expressed as H×W. Each frame is decomposed into N non-overlapping blocks (patches), and the size of each block is P×P, so that these blocks can cover the entire frame. Therefore, the total number of patches is N = HW / P 2; Each patch block will be flattened into a vector, represented as Where p=1, ..., N represents the spatial position of the block in the image, and t=1, ..., F represents the index of the frame.

[0044] The TimeSformer model is used to extract features from patches in video frames (each frame is an RGB image). The TimeSformer model consists of multiple coding blocks L. In each coding block l, the representation of the previous coding block is used to extract features from the patch. The query / key / value vector is calculated for each patch as follows:

[0045]

[0046] Among them, LN represents layer normalization, is the index of multiple attention heads, represents the total number of attention heads; the potential dimension of each attention head is set to represents the query vector, represents the key vector, Represents a vector of values. is generated The linear transformation matrix of is generated The linear transformation matrix of is generated The linear transformation matrix of .

[0047] In step 2.2, a separate time-space attention operation is performed on the features of the RGB image extracted in step 2.1 to obtain RGB features that combine the relationship between the time dimension and the spatial dimension.

[0048] First, temporal attention is calculated by comparing each patch with all patches at the same spatial position in other frames. The calculation formula is as follows:

[0049]

[0050] Second, spatial attention is calculated by comparing each patch with all patches at other spatial locations in the current image frame, using the following formula:

[0051]

[0052] Multiply the temporal attention and spatial attention calculated above by the value vector of the corresponding position and sum them to get the output of each attention head The attention calculation results are combined by weighted sum as follows:

[0053]

[0054] All attention heads The results are concatenated and then transformed by a linear transformation matrix W O , through MLP projection, the final encoding is obtained through residual connection:

[0055]

[0056] Finally, the encoding result is processed again by MLP and layer normalization, and the RGB feature combining the relationship between time dimension and space dimension is obtained through residual connection, which is expressed as H R :

[0057]

[0058] Step 3: Divide the video data into several continuous, non-overlapping small video segments, each containing T frames; for each video segment, use the pre-trained optical flow estimation model (RAFT model) to calculate the optical flow between adjacent frames to generate an optical flow map sequence containing T-1 optical flow maps; each optical flow map contains horizontal and vertical optical flow components, which are represented as a tensor of size H×W×C after downsampling or convolution coding (C is the number of channels, here C=3, corresponding to RGB three channels); then process the encoded optical flow map sequence according to the processing method of step 2 above to obtain the flow feature H that combines the relationship between the time dimension and the spatial dimension. F .

[0059] That is, each optical flow map in the encoded optical flow map sequence is decomposed into N non-overlapping patches, and the pre-trained TimeSformer model is used to extract features from the patches in the optical flow map; then, a separate time-space attention operation is performed on the extracted optical flow map features to obtain the flow feature H that combines the relationship between the time dimension and the spatial dimension. F .

[0060] Step 4: After obtaining the sentiment feature H E , flow characteristics H F and RGB feature H R Afterwards, a cross-modal interaction strategy is used to fuse the flow features H F and RGB feature H R , get the RGB features after flow enhancement

[0061] The specific calculation process is as follows:

[0062]

[0063]

[0064] Among them, M RGB Used to measure the correlation between the current RGB feature and each stream feature. It represents the fused RGB features, which combines the temporal information of the flow features and is helpful for detecting abnormal behavior. T is a row vector used to map input features to scalar attention scores, W1 and W2 are projection matrices, and b1 is a bias term.

[0065]

[0066] Among them, σ represents the activation function, ⊙ represents element-wise multiplication, W3 and b2 are learnable parameters, Represents the RGB feature after flow enhancement, which is used for subsequent anomaly detection score inference.

[0067] Step 5: Get the sentiment feature H E , flow characteristics H F and RGB feature H R Afterwards, a cross-modal interaction strategy is used to fuse the flow features H F and emotional perception features H E , get the emotion-enhanced flow features

[0068] The specific calculation process is as follows:

[0069] M F =w T tanh(W4H F +W5H E +b3)

[0070]

[0071] Among them, M F Used to measure the correlation between stream features and sentiment information at different time points, It represents the flow features that integrate emotional information, emphasizes those flow features that are more relevant to the emotional state, and enhances the abnormality recognition ability of the model. T is a row vector used to map input features to scalar attention scores, W4 and W5 are projection matrices, and b3 is a bias term.

[0072]

[0073] Among them, σ represents the activation function, ⊙ represents element-wise multiplication, W6 and b4 are learnable parameters, Represents the emotion-enhanced stream feature, which is used for subsequent anomaly detection score inference.

[0074] Step 6: Enhance the RGB features based on the obtained flow and emotion-enhanced flow characteristics Generate anomaly scores through the video character abnormal behavior detection model.

[0075] In the video anomaly detection task, each video is regarded as a packet, which consists of T video clips, denoted as And the video-level label Y = {0, 1}. θ represents the video character abnormal behavior detection model (i.e., the model for scoring abnormal behavior of characters in the video). In the model f θ When scoring abnormal behavior of a person, the abnormality score of the positive sample (the sample indicating abnormality) should be higher than the abnormality score of the negative sample (the sample indicating normality). To achieve the above goal, the present invention uses a ranking loss to optimize the model f θ , the loss function of the ranking loss is expressed as follows:

[0076]

[0077] Among them, B a and B n are positive sample set and negative sample set respectively, a i and n i are the individual in the positive sample set and the negative sample set respectively, f θ (a i ) and f θ (n i ) is the abnormality score for the positive sample set and the negative sample set.

[0078] In addition to the above ranking loss, the present invention also uses two other losses to train the model f θ , respectively:

[0079] (1) Binary classification loss (L BCE ), used to train the model to determine whether the video contains anomalies;

[0080] (2) Reconstruction loss (L RE ), which is used to train the model to learn the potential feature representation of the video and help the model better identify abnormal behavior.

[0081] Finally, the sum of the ranking loss, the binary classification loss, and the reconstruction loss is defined as the total loss function, which is expressed as follows:

[0082] L total =L(B a ,B n )+L BCE +L RE

[0083] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for detecting abnormal behavior of people in videos based on emotion enhancement, characterized in that: The following steps are involved: Step 1: Extract the emotional features H of the characters from the video data E ; Step 2: Decompose each RGB image frame of the video data into N non-overlapping patches, and use the pre-trained TimeSformer model to extract features from the patches in each RGB image frame of the video; then perform a separate time-space attention operation on the features of the extracted RGB image to obtain the RGB feature H that combines the relationship between the time dimension and the spatial dimension. R ; Step 3: Divide the video data into several continuous, non-overlapping small video segments, each containing T frames; for each video segment, use the pre-trained optical flow estimation model to calculate the optical flow between adjacent frames to generate an optical flow map sequence containing T-1 optical flow maps; each optical flow map contains horizontal and vertical optical flow components, which are represented as a tensor of size H×W×3 after downsampling or convolutional encoding; then decompose each optical flow map in the encoded optical flow map sequence into N non-overlapping patches, and use the pre-trained TimeSformer model to extract features from the patches in the optical flow map; then perform a separated time-space attention operation on the extracted optical flow map features to obtain the flow feature H that combines the relationship between the time dimension and the spatial dimension. F ; Step 4: Adopt cross-modal interaction strategy to fuse flow features H F and RGB feature H R , get the RGB features after flow enhancement Step 5: Adopt cross-modal interaction strategy to fuse flow features H F and emotional perception features H E , get the emotion-enhanced flow features Step 6: Enhance the RGB features based on the obtained flow and emotion-enhanced flow characteristics Generate anomaly scores through the video character abnormal behavior detection model.

2. The method for detecting abnormal behavior of people in videos based on emotion enhancement according to claim 1, characterized in that: Step 1 is handled as follows: Step 1.1: Use the object detection model to detect the bounding box information of the person in the video frame and identify the position of the person in each frame of the video image; then input the bounding box information extracted by the object detection model into the OpenPose pose recognition model to extract the key point information of the person's pose; Step 1.2: Generate posture features using the posture feature generation model based on the extracted posture key point information. The posture feature generation model consists of ST-GCN, a convolutional layer, and three fully connected layers. Step 1.3: Use an autoencoder model to learn and process the latent emotion space of the obtained posture features. The autoencoder model compresses the posture features into a lower-dimensional emotion representation vector, that is, the character's emotion feature H is obtained. E .

3. The method for detecting abnormal behavior of people in videos based on emotion enhancement according to claim 1, characterized in that: The total loss of the video character abnormal behavior detection model includes: binary classification loss, reconstruction loss and sorting loss.

4. The method for detecting abnormal behavior of people in videos based on emotion enhancement according to claim 3, characterized in that: The loss function of the ranking loss is expressed as follows: Among them, B a and B n are positive sample set and negative sample set respectively, a i and n i are the individual in the positive sample set and the negative sample set respectively, f θ (a i ) and f θ (n i ) is the abnormality score for the positive sample set and the negative sample set.

Images