Abnormal behavior identification method and system based on video analysis

Abstract

The invention relates to the technical field of communication, in particular to an abnormal behavior recognition method and system based on video analysis, and the method specifically comprises the steps: obtaining a to-be-detected video signal as input data, inputting the input data into a pre-trained adaptive multi-modal fusion model, and outputting an abnormal detection result. Identifying whether an abnormal behavior exists in the video according to a detection result; according to the recognition method, the features of the three modes of space, time and space-time are utilized, intelligent fusion is carried out through the self-adaptive feature fusion module, the accuracy and efficiency of abnormal behavior recognition can be effectively improved, and an unexpected technical effect is achieved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an abnormal behavior recognition method and system based on video analysis. Background Art

[0002] As an important part of the comprehensive transportation system, comprehensive transportation hubs have a high passenger throughput and are prone to abnormal behaviors (such as fighting, falling, etc.). However, the traditional hub monitoring system is very time-consuming and labor-intensive due to the large number of monitoring points and the need for manual monitoring. It is difficult to meet the needs of transportation hub security management. Therefore, the use of intelligent video analysis technology to actively detect and alarm abnormal behaviors of passengers in transportation hubs in real time can effectively assist transportation hub security personnel in handling abnormal events and improve the rapid response capabilities of transportation hubs to emergencies.

[0003] The intelligent video analysis technology is used to design the abnormal behavior detection system of the comprehensive transportation hub. Intelligent video analysis equipment is installed at the key entrances and exits of the transportation hub, transfer areas, and public waiting areas, so that the computer can automatically analyze the real-time video image sequence obtained from the monitoring camera and understand the video content in the monitoring scene. Because the system has set certain alarm rules in advance, it can issue a real-time alarm when the system detects abnormal behavior of personnel, ensuring that the hub's operations, security, and emergency management personnel are aware of the area and content of the alarm in a timely manner, so as to take measures to resolve safety hazards and avoid dangerous consequences.

[0004] Currently, the more mainstream abnormal behavior detection method is based on trajectory. This method converts the movement process of the detected target into trajectory information, and determines whether the target has abnormal behavior by analyzing the shape, speed, acceleration, direction of movement and other characteristics of the trajectory.

[0005] In practical applications, traditional abnormal behavior detection technology mainly faces the following shortcomings: (1) Surveillance videos often contain complex background information, such as dust, fog, reflections, etc. Due to these interference factors, it is difficult for the system to accurately analyze the trajectory; (2) When the trajectory is incomplete due to occlusion of the target, the system may misjudge normal behavior as abnormal because it cannot fully capture the movement of the target; (3) Trajectory methods are usually based on simple trajectory patterns and cannot fully model complex behaviors, resulting in insufficient detection accuracy.

[0006] Therefore, it is of great significance to develop a method for identifying abnormal video behavior with high accuracy and efficiency. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for abnormal behavior recognition based on video analysis to solve the problem of low accuracy and efficiency in the existing methods for abnormal behavior recognition in video. The features of three modalities, namely space, time and spatiotemporal, are used and intelligently fused through an adaptive feature fusion module to improve the accuracy and efficiency of abnormal behavior recognition.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A method for identifying abnormal behavior based on video analysis comprises the following steps:

[0010] S1, obtaining a video signal to be tested as input data;

[0011] S2, inputting the input data into the pre-trained adaptive multimodal fusion model;

[0012] S3, output anomaly detection results, and identify whether there is abnormal behavior in the video according to the detection results;

[0013] The adaptive multimodal fusion model includes a spatial feature extractor, a temporal feature extractor, a spatiotemporal feature extractor, a Transformer encoder, an adaptive feature fusion module, and a sequence modeling component; the specific process of the model is as follows:

[0014] S21, input a static image signal to a spatial feature extractor to extract a spatial feature vector; input a continuous video frame sequence to a temporal feature extractor to extract a temporal feature sequence; input a continuous video frame sequence to a spatiotemporal feature extractor to extract a spatiotemporal feature vector;

[0015] S22, use the Transformer encoder to extract spatial features, temporal features and spatiotemporal features from the spatial feature vector, temporal feature sequence and spatiotemporal feature vector respectively;

[0016] S23, inputting the spatial features, the temporal features and the spatiotemporal features into an adaptive feature fusion module to obtain fused features;

[0017] S24, inputting the fused features into the sequence modeling component to obtain the time series prediction result;

[0018] S25. Calculate the anomaly score of each time step based on the time series prediction results;

[0019] S26: classify the abnormal behaviors according to the abnormal scores and output the recognition results.

[0020] The present invention provides an abnormal behavior recognition method based on video analysis, which specifically includes obtaining a video signal to be tested as input data, then inputting the input data into a pre-trained adaptive multimodal fusion model, outputting anomaly detection results, and identifying whether there is abnormal behavior in the video according to the detection results; the recognition method utilizes the features of three modalities, namely space, time and spatiotemporal, and performs intelligent fusion through an adaptive feature fusion module, which can effectively improve the accuracy and efficiency of abnormal behavior recognition and achieve unexpected technical effects.

[0021] Furthermore, in S2, the adaptive multimodal fusion model is trained using the following method:

[0022] A21. Obtain video data for training and testing and perform preprocessing.

[0023] A22, substituting the preprocessed video data for training into the fusion model to be trained for training, and adjusting the model parameters of the fusion model to be trained according to the loss function; substituting the preprocessed video data for testing into the fusion model to be trained to calculate the evaluation index;

[0024] A23, looping through step A22 until the value of the loss function is less than the second threshold or the number of training times reaches the third threshold, selecting the model parameters when the evaluation index is optimal or the model parameters of the last round of training, and completing the training of the adaptive multimodal fusion model;

[0025] The loss function is expressed as: L_total = α*L_class + β*L_fusion, where α and β are hyperparameters, L_total is the total loss value; L_class is the separation loss value; L_fusion is the modal fusion loss value;

[0026] L_class is expressed as: L_class = -\frac{1}{N}\sum{i=1}^{N}\log(p_{i,y_i});

[0027] Where N represents the total number of samples, the i index traverses all samples, y_i is the true category label of sample i, and p_{i,y_i} is the probability that the model predicts sample i to be its true category y_i.

[0028] L_fusion is expressed as: L_fusion = sum{i\neq j}D(F_i,F_j);

[0029] Where D represents the distance function used to measure feature consistency, such as mean square error (MSE) or cosine similarity, F_i and F_j represent feature representations from different modalities, respectively, and i and j are the indices of different modalities.

[0030] Further, in S21, preprocessing includes frame extraction, resizing and normalization.

[0031] Furthermore, in S22, the evaluation indicators include accuracy, recall, precision and F1 score.

[0032] Furthermore, in S21, the spatial feature extractor is based on the ResNet-50 model, including an input layer and an output layer connected in sequence;

[0033] Input layer, adjusts the input static image signal to a uniform pixel size, normalizes it, and scales the pixel value to [0, 1];

[0034] Output layer, remove the fully connected layer of the model, use the output of the last convolutional layer as the feature representation, the dimension of the output feature is [feature map height, feature map width, number of feature channels].

[0035] Furthermore, in S21, the temporal feature extractor includes an optical flow calculation module and a deep CNN module connected in sequence; wherein the optical flow calculation module is used to calculate an optical flow vector field from a continuous video frame sequence; and the deep CNN module is used to calculate the temporal features from the optical flow vector field.

[0036] Furthermore, the specific process of the optical flow calculation module is as follows:

[0037] Video frame sampling: extract two consecutive frames from the video stream, denoted as $F_{t}$ and $F_{t+1}$;

[0038] Inter-frame intensity difference: Calculate the pixel intensity difference between two frames to obtain the difference map $D_{t}$;

[0039] Gradient calculation: Calculate the spatial gradient $\nabla F_{t}$ and the temporal gradient $\partial F_{t} / \partial t$ of $F_{t}$ to estimate the motion of the pixel;

[0040] Optical flow estimation: The TV-L1 optical flow estimation algorithm is used to calculate the displacement vector $(u,v)$ of each pixel in the $x$ and $y$ directions based on intensity difference and gradient information;

[0041] Optical flow vector field generation: The calculated displacement vectors are integrated into the optical flow vector field $\mathbf{V}{t}$, which represents the pixel motion from $F{t}$ to $F_{t+1}$.

[0042] Furthermore, the specific process of the deep CNN module is as follows:

[0043] Convolutional layer configuration: The network is constructed by stacking two convolutional layers. Each convolutional layer uses a set of filters to extract features. The mathematical expression of the convolution operation is:

[0044] [O_{ijk}=\sum_{m}\sum_{n}\sum_{p}I_{(i+m)(j+n)p}\cdot K_{mnpk}+b_k];

[0045] Where (O_{ijk}) represents the value of the (k)th channel at the (i,j)th position of the output feature map, (I) is the input feature map, (K) is the convolution kernel, and (b) is the bias term;

[0046] Activation function: ReLU activation function is applied after each convolutional layer to introduce nonlinearity, defined as (f(x) = \max(0,x));

[0047] Pooling layer: insert the pooling layer to reduce the spatial dimension of the feature map, using maximum pooling, and the pooling operation is expressed as: [P_{ijk}=\max_{(m,n)\in\mathcal{M}{ij}}O{mnk}], where (P_{ijk}) is the value of the (k)th channel at the (i,j)th position of the feature map after pooling, and (\mathcal{M}_{ij}) is the pooling window centered at ((i,j));

[0048] Fully connected layer: A fully connected layer is used at the end of the network to convert the pooled feature map into a feature vector.

[0049] Furthermore, the spatiotemporal feature extractor includes an input layer, a 3D convolutional layer, a residual connection module, and a global average pooling layer connected in sequence;

[0050] Input layer: Input video frames, accepting an input video volume of size (T,H,W,C), where (T) is the number of frames in the time dimension, (H) and (W) are the height and width of the frame respectively, and (C) is the number of color channels;

[0051] 3D convolution layer: Use 3D convolution kernels to process spatial and temporal information simultaneously to capture motion and appearance features in video sequences; the output feature volume size is calculated as: [O = \frac{I-K+2P}{S}+1], where (I) is the input size, (K) is the convolution kernel size, (P) is the padding, (S) is the stride, (O) is the output size, and the number of output channels is set to 64;

[0052] Residual connection module: four groups of residual blocks are configured, namely Conv2_x, Conv3_x, Conv4_x, and Conv5_x;

[0053] Global average pooling layer: A global average pooling layer is used at the end of the network to calculate the average value of each channel and simplify the feature volume of (T, H, W, C) to a feature vector of (1, 1, 1, C).

[0054] Furthermore, the adaptive feature fusion module includes a feature normalization layer, an attention weight learning layer, a fusion strategy module, and an output interface connected in sequence;

[0055] Feature Normalization Layer: For each feature set from the spatial, temporal, and spatiotemporal feature extractors, the mean and variance of all features in the batch are calculated and each feature vector is normalized using the following formula:

[0056] [\hat{x}^{(k)}=\frac{x^{(k)}-\mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2}+

[0057] \epsilon}}], where (x^{(k)}) is the feature vector, (\mu_{\mathcal{B}}) is the mean of the batch (\mathcal{B}), (\sigma_{\mathcal{B}}^{2}) is the variance, and (\epsilon) is a constant;

[0058] Introduce a set of learnable parameters (\gamma) and (\beta) for scaling and translating standardized features: [y^{(k)}=\gamma\hat{x}^{(k)}+\beta], where (y^{(k)}) is the output standardized feature, (\gamma) and (\beta) are the scaling factors and translation parameters of the feature;

[0059] Attention weight learning layer: A fully connected neural network is used to learn feature weights. The input is a feature vector and the output is the weight of each feature. The weights are normalized by the softmax function to ensure that the sum of the weights is 1.

[0060] Fusion strategy module: combines weighted summation and nonlinear activation function to dynamically fuse different features;

[0061] The fusion operation can be expressed as: [F_{fused}=\sigma(w_s\cdot F_s+w_t\cdot F_t+w_{st}\cdot F_{st})], where (F_s), (F_t), and (F_{st}) represent spatial, temporal and spatiotemporal features respectively, (w_s), (w_t), and (w_{st}) are the corresponding attention weights, and (\sigma) is a nonlinear activation function;

[0062] Output interface: The generated fused features (F_{fused}) are output to the subsequent sequence modeling component.

[0063] Furthermore, the sequence modeling component is based on LSTM; each LSTM unit contains an input gate, a forget gate, an output gate, and a unit state to maintain and update the information in the time series; the input gate controls the amount of current input information received, the forget gate controls the amount of past states retained, and the output gate controls the contribution of the current unit state to the output;

[0064] The formulas for the input gate ((i_t)), forget gate ((f_t)), output gate ((o_t)), and unit candidate ((\tilde{C}t)) are as follows:

[0065] [i_t=\sigma(W{xi}x_t+W_{hi}h_{t-1}+W_{ci}C_{t-1}+b_i)];

[0066] [f_t=\sigma(W_{xf}x_t+W_{hf}h_{t-1}+W_{cf}C_{t-1}+b_f)];

[0067] [o_t=\sigma(W_{xo}x_t+W_{ho}h_{t-1}+W_{co}C_{t-1}+b_o)];

[0068] [\tilde{C}t=\tanh(W{xc}x_t+W_{hc}h_{t-1}+b_c)];

[0069] The update formulas for the cell state ((C_t)) and hidden state ((h_t)) are as follows:

[0070] [C_t=f_t\ast C_{t-1}+i_t\ast\tilde{C}_t];

[0071] [h_t=o_t\ast\tanh(C_t)];

[0072] Among them, (x_t) is the input of the current time step, (h_{t-1}) is the hidden state of the previous time step, (C_{t-1}) is the cell state of the previous time step, (W) and (b) are the weight matrix and bias vector, (\sigma) is the sigmoid activation function, and (\ast) represents the Hadamard product.

[0073] Furthermore, the anomaly scoring system is based on the following scoring:

[0074] [S(x)=\text{softmax}(W\cdot F(x)+b)], where (S(x)) is the anomaly score, (W) and (b) are the learned weights and biases, and (h_t) is the output of the sequence modeling component.

[0075] Another object of the present invention is to provide an abnormal behavior recognition system based on video analysis.

[0076] An abnormal behavior recognition system based on video analysis, comprising at least one processor and a memory in communication with the at least one processor; the memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor so that the at least one processor can implement the recognition method described in any one of claims 1 to 9,

[0077] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0078] 1. The present invention provides an abnormal behavior recognition method based on video analysis, which specifically includes obtaining a video signal to be tested as input data, then inputting the input data into a pre-trained adaptive multimodal fusion model, outputting anomaly detection results, and identifying whether there is abnormal behavior in the video according to the detection results; the recognition method utilizes the features of three modalities, namely space, time and spatiotemporal, and performs intelligent fusion through an adaptive feature fusion module, which can effectively improve the accuracy and efficiency of abnormal behavior recognition and achieve unexpected technical effects.

[0079] 2. The present invention is based on an adaptive multimodal fusion network based on deep learning. This framework can effectively integrate information of different modalities, such as the features of space, time and spatiotemporal modalities, to solve the problem of real-time identification of abnormal behaviors in video surveillance.

[0080] 3. Adaptive multimodal fusion network, which can learn and integrate various types of data sources. It usually exhibits better generalization ability and can more effectively handle new scenarios or unseen abnormal behavior types.

[0081] 4. When a single modality is interfered by noise or has poor quality, the multimodal fusion network can still rely on information from other modalities to make accurate judgments, thereby improving the robustness of the system.

[0082] 5. The present invention provides a loss function for an adaptive multimodal network, which integrates multiple objectives, such as classification accuracy and modality fusion quality, to guide the network training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 The present invention is a flowchart of an abnormal behavior recognition method based on video analysis.

[0084] Figure 2 Data processing flow chart of the adaptive multimodal fusion model.

[0085] Figure 3 Data processing flow chart of the temporal feature extractor.

[0086] Figure 4 The data flow diagram for each group of residual blocks.

[0087] Figure 5 Flowchart of the adaptive multimodal fusion network recognition method based on deep learning.

[0088] Figure 6 Building a framework for big data cache processing.

[0089] Figure 7 This is the data flow diagram of the image processing process. DETAILED DESCRIPTION

[0090] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0091] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0092] Example

[0093] like Figure 1 and Figure 2 As shown, a method for identifying abnormal behavior based on video analysis includes the following steps:

[0094] S1, obtaining a video signal to be tested as input data;

[0095] S2, inputting the input data into the pre-trained adaptive multimodal fusion model;

[0096] S3, output anomaly detection results, and identify whether there is abnormal behavior in the video according to the detection results;

[0097] The adaptive multimodal fusion model includes a spatial feature extractor, a temporal feature extractor, a spatiotemporal feature extractor, a Transformer encoder, an adaptive feature fusion module, a sequence modeling component, an anomaly scoring system and a multi-head classifier; the specific process of the model is as follows:

[0098] S21, input a static image signal to a spatial feature extractor to extract a spatial feature vector; input a continuous video frame sequence to a temporal feature extractor to extract a temporal feature sequence; input a continuous video frame sequence to a spatiotemporal feature extractor to extract a spatiotemporal feature vector;

[0099] S22, use the Transformer encoder to extract spatial features, temporal features and spatiotemporal features from the spatial feature vector, temporal feature sequence and spatiotemporal feature vector respectively;

[0100] S23, inputting the spatial features, the temporal features and the spatiotemporal features into an adaptive feature fusion module to obtain fused features;

[0101] S24, inputting the fused features into the sequence modeling component to obtain the time series prediction result;

[0102] S25. Based on the output of the sequence model, the anomaly scoring system calculates the anomaly score for each time step;

[0103] S26. Classify the abnormal behavior using a multi-head classifier and output the recognition result.

[0104] In one or more embodiments, the spatial feature extractor is based on an existing ResNet-50 model, which can effectively extract spatial features from a single video frame to assist in detecting abnormal video behavior. The input and output of the model are as follows:

[0105] Input layer: The input image is resized to a uniform size of 224x224 pixels and normalized. The pixel values ​​are scaled to the range of [0,1] to ensure consistent distribution of image data.

[0106] Output layer: Remove the fully connected layer of the model and use the output of the last convolutional layer as the feature representation to capture the spatial hierarchical structure of the image. The dimension of the output feature is [feature map height, feature map width, number of feature channels].

[0107] Video surveillance systems need to effectively identify and understand dynamic changes between video frames. Existing temporal feature extraction methods usually rely on hand-crafted features or simple differentiation methods, which often perform poorly in complex scenarios.

[0108] In one or more embodiments, a temporal feature extractor using a combination of optical flow and deep convolutional neural network is provided. The temporal feature extractor uses an optical flow algorithm and a deep convolutional network to extract temporal features of a video sequence so as to more accurately analyze and identify abnormal behaviors. The specific process of the temporal feature extractor is as follows: Figure 3 shown.

[0109] Specifically, the optical flow calculation module

[0110] Optical flow is the pattern of visual motion of objects, surfaces, and edges in an image sequence, which shows the change of an object or scene from one position to another.

[0111] In one or more embodiments, an optical flow estimation module can calculate the pixel motion between video frames in real time and accurately, while maintaining edge information and having good robustness to noise. The input of the optical flow calculation module is a continuous sequence of video frames, and the output is an optical flow vector field. The specific workflow is as follows:

[0112] Video frame sampling: Extract two consecutive frames from the video stream, denoted as $F_{t}$ and $F_{t+1}$.

[0113] Inter-frame intensity difference: Calculate the pixel intensity difference between two frames and obtain the difference map $D_{t}$.

[0114] Gradient calculation: Calculate the spatial gradient $\nabla F_{t}$ and the temporal gradient $\partial F_{t} / \partial t$ of $F_{t}$ to estimate the motion of the pixel.

[0115] Optical flow estimation: The TV-L1 optical flow estimation algorithm is used to calculate the displacement vector $(u,v)$ of each pixel in the $x$ and $y$ directions based on intensity difference and gradient information.

[0116] Optical flow vector field generation: The calculated displacement vectors are integrated into the optical flow vector field $\mathbf{V}{t}$, which represents the pixel motion from $F{t}$ to $F_{t+1}$.

[0117] Deep CNN module

[0118] In one or more embodiments, a deep CNN network architecture is specifically used to process the output of the optical flow calculation module and further extract temporal features to support subsequent abnormal behavior detection and analysis. The network structure includes the following components:

[0119] Convolutional layer configuration: The network is constructed by stacking two convolutional layers, each of which uses a set of filters to extract features. The mathematical expression of the convolution operation is:

[0120] [O_{ijk}=\sum_{m}\sum_{n}\sum_{p}I_{(i+m)(j+n)p}\cdot K_{mnpk}+b_k];

[0121] Where (O_{ijk}) represents the value of the (k)th channel at the (i,j)th position of the output feature map, (I) is the input feature map, (K) is the convolution kernel, and (b) is the bias term.

[0122] Activation function: A ReLU activation function is applied after each convolutional layer to introduce non-linearity, defined as (f(x) = \max(0,x)).

[0123] Pooling layer: Insert the pooling layer to reduce the spatial dimension of the feature map. The maximum pooling is adopted. The pooling operation is expressed as: [P_{ijk}=\max_{(m,n)\in\mathcal{M}{ij}}O{mnk}], where (P_{ijk}) is the value of the (k)th channel at the (i,j)th position of the feature map after pooling, and (\mathcal{M}_{ij}) is the pooling window centered at ((i,j)).

[0124] Fully connected layer: A fully connected layer is used at the end of the network to convert the pooled feature map into a feature vector.

[0125] Spatiotemporal feature extractor

[0126] Traditional 2D convolutional neural networks are mainly used to process static images, and they are not enough to capture information in the temporal dimension of video sequences. In view of this limitation, the present invention proposes a spatiotemporal feature extractor based on 3D-ResNet, which can effectively train deep 3D CNNs by introducing a residual learning mechanism while reducing the number of parameters and computational complexity. The spatiotemporal feature extractor of the present invention includes the following components:

[0127] Input layer: Input video frames, accepts an input video volume of size (T,H,W,C), where (T) is the number of frames in the time dimension, (H) and (W) are the height and width of the frame respectively, and (C) is the number of color channels.

[0128] 3D convolution layer: Use 3D convolution kernels to process spatial and temporal information simultaneously to capture motion and appearance features in video sequences. Here, the 3D convolution kernel size is (7,7,7) and the stride is ((1,2,2)) corresponding to ((T,H,W)). The output feature volume size is calculated as follows: [O = \frac{I-K+2P}{S}+1], where (I) is the input size, (K) is the convolution kernel size, (P) is padding, (S) is the stride, (O) is the output size, and the number of output channels is set to 64.

[0129] Residual connection: Four groups of residual block configurations are adopted, namely Conv2_x, Conv3_x, Conv4_x, and Conv5_x. The number of residual blocks contained in each group, that is, the number of 3D convolutional layers contained in each residual block, is shown in Table 1.

[0130] Table 1

[0131] Residual Block Group The number of residual blocks in each group Number of 3D convolutional layers Conv2_x 3 2 Conv3_x 4 3 Conv4_x 6 3 Conv5_x 3 2

[0132] Each 3D convolution layer uses a (3, 3, 3) convolution kernel with a stride of ((1, 1, 1)) and applies 3D batch normalization and ReLU activation functions after each convolution operation. The number of output channels increases from 64 in Conv2_x to 128 in Conv3_x, to 256 in Conv4_x, and finally to 512 in Conv5_x. The residual block group for each group is as follows:

[0133] Global average pooling: A global average pooling layer is used at the end of the network to calculate the average value of each channel and simplify the feature volume of (T, H, W, C) to a feature vector of (1, 1, 1, C). Figure 4 As shown, Figure 4 The data flow diagram for each group of residual blocks.

[0134] Adaptive Fusion Module

[0135] In video analysis tasks, fusing information from different modalities (such as spatial features, temporal features, and spatiotemporal features) is crucial to improving the accuracy of the final decision. Existing feature fusion methods are usually fixed and cannot be dynamically adjusted according to the different characteristics of the data, which limits the adaptability and accuracy of the model when facing diverse scenarios. In order to solve the above problems, an adaptive feature fusion module is provided, which can dynamically adjust the fusion method according to different input features to achieve the optimal combination of multimodal information in video data.

[0136] In one or more embodiments, the provided adaptive feature fusion module includes the following key components:

[0137] Feature normalization layer: For each feature set from the above spatial, temporal, and spatiotemporal feature extractors, the mean and variance of all features in the batch are calculated and each feature vector is normalized using the following formula:

[0138] [\hat{x}^{(k)}=\frac{x^{(k)}-\mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2}+

[0139] \epsilon}}], where (x^{(k)}) is the feature vector, (\mu_{\mathcal{B}}) is the mean of the batch (\mathcal{B}), (\sigma_{\mathcal{B}}^{2}) is the variance, and (\epsilon) is a small constant added for numerical stability. After normalization, a set of learnable parameters (\gamma) and (\beta) are introduced to scale and translate the normalized features: [y^{(k)}=\gamma\hat{x}^{(k)}+\beta], where (y^{(k)}) is the output normalized feature, (\gamma) and (\beta) are the scaling factors and translation parameters of the feature.

[0140] Attention weight learning layer: A fully connected neural network is used to learn feature weights. The input is a feature vector and the output is the weight of each feature. The weights are normalized by the softmax function to ensure that the sum of the weights is 1.

[0141] Fusion strategy: Combine weighted summation and nonlinear activation function to dynamically fuse different features. The fusion operation can be expressed as: [F_{fused}=\sigma(w_s\cdot F_s+w_t\cdot F_t+w_{st}\cdot F_{st})], where (F_s), (F_t), and (F_{st}) represent spatial, temporal and spatiotemporal features respectively, (w_s), (w_t), and (w_{st}) are their corresponding attention weights, and (\sigma) is a nonlinear activation function.

[0142] Output interface: The generated fused features (F_{fused}) are output to the subsequent sequence modeling component.

[0143] Sequence Modeling Components

[0144] When processing time series data such as video sequences, it is crucial to capture the temporal dependencies in the data. The present invention is based on a sequence modeling component based on LSTM, which can effectively capture and model the complex temporal dependencies in video sequences. The sequence modeling component includes the following key technical features:

[0145] LSTM cell configuration:

[0146] Each LSTM unit contains an input gate, a forget gate, an output gate, and a unit state to maintain and update the information in the time series; the input gate controls the amount of current input information received, the forget gate controls the amount of past states retained, and the output gate controls the contribution of the current unit state to the output.

[0147] Status update formula:

[0148] The formulas for the input gate ((i_t)), forget gate ((f_t)), output gate ((o_t)), and unit candidate ((\tilde{C}t)) are as follows:

[0149] [i_t=\sigma(W{xi}x_t+W_{hi}h_{t-1}+W_{ci}C_{t-1}+b_i)];

[0150] [f_t=\sigma(W_{xf}x_t+W_{hf}h_{t-1}+W_{cf}C_{t-1}+b_f)];

[0151] [o_t=\sigma(W_{xo}x_t+W_{ho}h_{t-1}+W_{co}C_{t-1}+b_o)];

[0152] [\tilde{C}t=\tanh(W{xc}x_t+W_{hc}h_{t-1}+b_c)];

[0153] The update formulas for the cell state ((C_t)) and hidden state ((h_t)) are as follows:

[0154] [C_t=f_t\ast C_{t-1}+i_t\ast\tilde{C}_t];

[0155] [h_t=o_t\ast\tanh(C_t)];

[0156] Among them, (x_t) is the input of the current time step, (h_{t-1}) is the hidden state of the previous time step, (C_{t-1}) is the cell state of the previous time step, (W) and (b) are the weight matrix and bias vector, (\sigma) is the sigmoid activation function, and (\ast) represents the Hadamard product.

[0157] Abnormal Scoring System

[0158] The anomaly scoring system is capable of calculating an anomaly score based on the output received from the sequence modeling component, thereby effectively identifying abnormal behaviors in video sequences. The present invention introduces a scoring function as follows: [S(x)=\text{softmax}(W\cdot F(x)+b)], where (S(x)) is the anomaly score, (W) and (b) are the learned weights and biases, and (h_t) is the output of the sequence modeling component.

[0159] Multi-head classifier

[0160] Traditional video surveillance systems often rely on simple threshold judgments or single classification models to identify abnormal events. These methods may be limited in dealing with multiple types of abnormal behaviors because they cannot capture complex data distributions and inherent correlations. The present invention proposes a multi-head classifier that can classify data using scores from an anomaly scoring system, thereby improving the accuracy and efficiency of abnormal event identification.

[0161] Abnormal behaviors in video surveillance systems may include various types, such as people falling, fighting, people staying, etc. A traditional single classifier may not be able to effectively distinguish these different abnormal behaviors, resulting in false positives or false negatives. The present invention provides a design of a multi-head classifier, in which each "head" specializes in the recognition of a specific abnormal behavior in video data.

[0162] Each "head" is defined as an independent classification unit responsible for identifying a specific type of abnormal behavior. For example, "head 1" is responsible for detecting falls, "head 2" is responsible for detecting fights, and "head 3" is responsible for detecting people being stranded.

[0163] Each "head" of the multi-head classifier consists of a series of fully connected layers and activation functions. For each video clip, each classification head in the multi-head classifier makes an independent judgment: [y_{i}=\sigma(W_{i}h+b_{i})], where (y_{i}) is the output of the (i)th classification head, (W_{i}) and (b_{i}) are the weights and biases of the classification head, (h) is the output of the anomaly scoring system, and (\sigma) is the softmax activation function used to calculate the probability of the anomaly category.

[0164] In some or more embodiments, using Figure 5 The flowchart of the adaptive multimodal fusion network recognition method based on deep learning is shown.

[0165] Specifically, the adaptive multimodal fusion model is trained using the following method:

[0166] S21, obtaining video data for training and testing, and performing preprocessing;

[0167] S22, substituting the preprocessed video data for training into the fusion model to be trained for training, and adjusting the model parameters of the fusion model to be trained according to the loss function; substituting the preprocessed video data for testing into the fusion model to be trained to calculate the evaluation index;

[0168] S23, looping through step S22 until the value of the loss function is less than the second threshold or the number of training times reaches the third threshold, selecting the model parameters when the evaluation index is optimal or the model parameters of the last round of training, and completing the training of the adaptive multimodal fusion model.

[0169] Loss Function

[0170] For complex video analysis tasks, the traditional single loss function may not be able to effectively process multimodal data, resulting in poor performance in identifying multiple abnormal behaviors. Therefore, designing a composite loss function that can reflect the characteristics of multimodal data is the key to improving model performance. This paper proposes a composite loss function for a multimodal fusion network, which combines classification loss and modal fusion loss to optimize the detection of abnormal behaviors.

[0171] Classification loss (L_class): The classification loss uses the cross entropy loss function to measure the difference between the class probability distribution predicted by the model and the true distribution. The classification loss is expressed as the average of the negative log-likelihood of the predicted probability calculated for all training samples and categories:

[0172] L_class=-\frac{1}{N}\sum{i=1}^{N}\log(p_{i,y_i})

[0173] N represents the total number of samples, i index traverses all samples, y_i is the true category label of sample i, and p_{i,y_i} is the probability that the model predicts sample i to be its true category y_i.

[0174] Modal fusion loss (L_fusion): The modal fusion loss is expressed as a function that measures the consistency between different modal features, thereby promoting the network to learn more unified and complementary feature representations:

[0175] L_fusion=sum{i\neq j}D(F_i,F_j)

[0176] D represents the distance function used to measure feature consistency, such as mean square error (MSE) or cosine similarity, F_i and F_j represent feature representations from different modalities, respectively, and i and j are the indices of different modalities.

[0177] Total loss function: The total loss is the weighted sum of the classification loss and the modality fusion loss, where the weight coefficient is used to balance the importance of the two losses. The total loss (L_total) can be expressed as L_total = α*L_class + β*L_fusion, where α and β are hyperparameters.

[0178] During the training process, the network parameters are optimized through the back-propagation algorithm, and the gradient is calculated according to the defined loss function.

[0179] Stochastic gradient descent (SGD) is used to update the network parameters.

[0180] Experiment and analysis

[0181] Experimental Dataset,The experiment contains the following datasets:

[0182] Multiple public video datasets, including UCF-Crime, ShanghaiTech, etc., contain samples of various abnormal behaviors.

[0183] Build your own video behavior dataset and collect normal and abnormal video datasets in different scenarios.

[0184] Performance Indicators

[0185] Evaluation indicators include accuracy, recall, precision and F1-Score.

[0186] Accuracy evaluation: reflects the proportion of correct predictions made by the model, and is calculated as follows: [\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}], where (TP) is the number of true positive examples, (TN) is the number of true negative examples, (FP) is the number of false positive examples, and (FN) is the number of false negative examples.

[0187] Recall evaluation: Recall measures the model’s ability to identify positive examples. The calculation formula is: [\text{Recall}=\frac{TP}{TP+FN}]

[0188] Precision evaluation: Precision measures the accuracy of the model's prediction of positive examples. The calculation formula is: [\text{Precision}=

[0189] \frac{TP}{TP+FP}]

[0190] F1 score evaluation: The F1 score is the harmonic mean of recall and precision, and is an indicator for evaluating the robustness of the model. The calculation formula is: [F1\text{-Score}=2\times\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}]

[0191] Experimental Results

[0192] The following table shows the comparison results of using adaptive multimodal network and unimodal network on two different video datasets as shown in Table 2.

[0193] Table 2

[0194] Dataset Model Accuracy Recall Accuracy F1 score Dataset A Unimodal Model 85.2％ 78.3％ 82.1％ 80.2％ Dataset A Multimodal Model 91.8％ 89.9％ 90.4％ 89.8％ Dataset B Unimodal Model 86.2％ 80.0％ 83.7％ 81.8％ Dataset B Multimodal Model 93.7％ 90.9％ 92.2％ 91.5％

[0195] In one or some embodiments, the application scenarios of the identification method are specifically disclosed as follows:

[0196] Install embedded cameras for video behavior analysis at entrances and exits, important internal passages, waiting halls and other areas. Based on the configured monitoring area settings and abnormal behavior alarm settings, the video information can be directly analyzed and processed at the front end to obtain abnormal behavior alarm data for the corresponding entrances and exits, passages and areas. The obtained alarm data is stored in the monitoring database and pushed to the administrator in real time, thereby meeting the administrator's real-time monitoring and management needs. At the same time, the abnormal behavior alarm can also be uploaded to the hub collaborative management and emergency management system through the network for real-time disposal. Figure 6 As shown, the big data cache processing construction framework.

[0197] Data flow diagram, such as Figure 7 As shown:

[0198] Image reading: When collecting video, the system needs to use high-definition surveillance cameras to ensure that the image resolution meets the system reading requirements. The selection of surveillance cameras has a great impact on the detection effect of the intelligent video surveillance system. Surveillance cameras with poor resolution or poor performance will seriously affect the detection effect of the system. Therefore, when selecting surveillance cameras, you should choose surveillance cameras with high resolution, good performance, and high cost performance.

[0199] Object detection: Object detection is an important application in the field of computer vision. Object detection is different from object recognition. Object detection mainly finds objects based on pre-set rules and determines whether a certain area of ​​the video image to be monitored contains the object to be recognized. Object recognition is to test the computer program's ability to recognize objects, and recognition usually occurs in the area of ​​object detection.

[0200] Target recognition: Target detection and target recognition are two consecutive steps in the intelligent monitoring process. The system first detects the target in the video image and then recognizes the detected target. In order to recognize the target, the system needs to extract the features of the target and then recognize the target based on the features.

[0201] Behavior analysis: Behavior analysis is the use of computers to automatically identify abnormal behaviors in surveillance videos. The abnormal behaviors here need to be set in advance according to certain rules, and the setting of rules needs to be combined with the actual business scenario requirements of the hub.

[0202] Alarm display: The behavior analysis module in the system will analyze the video content. When it detects a certain behavior that triggers the abnormal behavior alarm rules that have been defined in advance, the system will issue an alarm. The abnormal behavior detection system based on intelligent video analysis technology mainly detects and identifies targets in the monitoring area, and analyzes in real time whether it matches the preset alarm events. If it matches, an alarm will be issued so that the staff can intervene or deal with it.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying abnormal behavior based on video analysis, characterized in that: The following steps are involved: S1, obtaining a video signal to be tested as input data; S2, inputting the input data into the pre-trained adaptive multimodal fusion model; S3, output anomaly detection results, and identify whether there is abnormal behavior in the video according to the detection results; The adaptive multimodal fusion model includes a spatial feature extractor, a temporal feature extractor, a spatiotemporal feature extractor, a Transformer encoder, an adaptive feature fusion module, and a sequence modeling component; the specific process of the model is as follows: S21, input a static image signal to a spatial feature extractor to extract a spatial feature vector; input a continuous video frame sequence to a temporal feature extractor to extract a temporal feature sequence; input a continuous video frame sequence to a spatiotemporal feature extractor to extract a spatiotemporal feature vector; S22, use the Transformer encoder to extract spatial features, temporal features and spatiotemporal features from the spatial feature vector, temporal feature sequence and spatiotemporal feature vector respectively; S23, inputting the spatial features, the temporal features and the spatiotemporal features into an adaptive feature fusion module to obtain fused features; S24, inputting the fused features into the sequence modeling component to obtain the time series prediction result; S25. Calculate the anomaly score of each time step based on the time series prediction results; S26: classify the abnormal behaviors according to the abnormal scores and output the recognition results.

2. The abnormal behavior recognition method based on video analysis according to claim 1 is characterized in that: In S2, the adaptive multimodal fusion model is trained using the following method: A21. Obtain video data for training and testing and perform preprocessing. A22, substituting the pre-processed video data for training into the fusion model to be trained for training, and adjusting the model parameters of the fusion model to be trained according to the loss function; Substitute the preprocessed video data for testing into the fusion model to be trained to calculate the evaluation index; A23. Repeat step A22 until the value of the loss function is less than the second threshold or the number of training times reaches the third threshold, and select the model parameters when the evaluation index is optimal or the model parameters of the last round of training, thereby completing the training of the adaptive multimodal fusion model.

3. The abnormal behavior identification method based on video analysis according to claim 1 is characterized in that: In S21, the spatial feature extractor is based on the ResNet-50 model, which includes an input layer and an output layer connected in sequence; Input layer, adjusts the input static image signal to a uniform pixel size, normalizes it, and scales the pixel value to [0, 1]; Output layer, remove the fully connected layer of the model, use the output of the last convolutional layer as the feature representation, the dimension of the output feature is [feature map height, feature map width, number of feature channels].

4. The abnormal behavior identification method based on video analysis according to claim 1 is characterized in that: In S21, the temporal feature extractor includes an optical flow calculation module and a deep CNN module connected in sequence; wherein the optical flow calculation module is used to calculate an optical flow vector field from a continuous video frame sequence; and the deep CNN module is used to calculate temporal features from the optical flow vector field.

5. The abnormal behavior recognition method based on video analysis according to claim 4 is characterized in that: The specific process of the deep CNN module is as follows: Convolutional layer configuration: The network is constructed by stacking two convolutional layers, each of which uses a set of filters to extract features; Activation function: ReLU activation function is applied after each convolutional layer to introduce nonlinearity; Pooling layer: insert the pooling layer to reduce the spatial dimension of the feature map; Fully connected layer: A fully connected layer is used at the end of the network to convert the pooled feature map into a feature vector.

6. The abnormal behavior identification method based on video analysis according to claim 1 is characterized in that: The spatiotemporal feature extractor includes an input layer, a 3D convolutional layer, a residual connection module, and a global average pooling layer connected in sequence; Input layer: Input video frames, accepting an input video volume of size (T,H,W,C), where (T) is the number of frames in the time dimension, (H) and (W) are the height and width of the frame respectively, and (C) is the number of color channels; 3D convolution layer: Use 3D convolution kernels to process spatial and temporal information simultaneously to capture motion and appearance features in video sequences; The output feature volume size is calculated as follows: [O = \frac{I-K+2P}{S}+1], where (I) is the input size, (K) is the convolution kernel size, (P) is the padding, (S) is the stride, (O) is the output size, and the number of output channels is set to 64; Residual connection module: four groups of residual blocks are configured, namely Conv2_x, Conv3_x, Conv4_x, and Conv5_x; Global average pooling layer: A global average pooling layer is used at the end of the network to calculate the average value of each channel and simplify the feature volume of (T, H, W, C) to a feature vector of (1, 1, 1, C).

7. The abnormal behavior identification method based on video analysis according to claim 1 is characterized in that: The adaptive feature fusion module includes a feature normalization layer, an attention weight learning layer, a fusion strategy module and an output interface connected in sequence; Feature normalization layer: For each feature set from the spatial, temporal, and spatiotemporal feature extractors, the mean and variance of all features are calculated and each feature vector is normalized; Attention weight learning layer: The neural network of the fully connected layer is used to learn feature weights. The input is the feature vector, and the output is the weight of each feature. The weights are normalized by the softmax function to ensure that the sum of the weights is 1. Fusion strategy module: combines weighted summation and nonlinear activation function to dynamically fuse different features; Output interface: The generated fusion features are output to the subsequent sequence modeling component.

8. The abnormal behavior identification method based on video analysis according to claim 1, characterized in that: The sequence modeling component is generated based on LSTM; each LSTM unit contains an input gate, a forget gate, an output gate, and a unit state to maintain and update the information in the time series; the input gate controls the amount of current input information received, the forget gate controls the amount of past states retained, and the output gate controls the contribution of the current unit state to the output.

9. The abnormal behavior recognition method based on video analysis according to any one of claims 1 to 8, characterized in that: The anomaly scoring system is based on the following scores: [S(x)=\text{softmax}(W\cdot F(x)+b)], where (S(x)) is the anomaly score, (W) and (b) are the learned weights and biases.

10. An abnormal behavior recognition system based on video analysis, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the identification method described in any one of claims 1 to 9.

Images