Crowd abnormal behavior detection method combining multi-scale features and bidirectional time

Abstract

The invention discloses a crowd abnormal behavior detection method combining multi-scale features and bidirectional time, and belongs to the technical field of computer vision. Aiming at the problems that when a single-scale feature is extracted only by using a convolutional neural network in an existing method, extraction of different-scale features is insufficient, and effective extraction of reverse time information is neglected, a global cross-channel feature extraction module is added into an encoder to extract a global context feature, so that the feature extraction efficiency is improved. A multi-scale feature fusion module combined with double-branch operation is added between an encoder and a decoder to extract local space information and global semantic representation of video frames under different scales, the local space information and the global semantic representation are effectively combined, meanwhile, a bidirectional convolution long-short term memory module is added, forward time information and reverse time information are fully utilized to improve detection performance, and the detection efficiency is improved. The method can be widely applied to detection of abnormal behaviors of crowds in videos.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for detecting abnormal behavior of a crowd by combining multi-scale features and bidirectional time. Background Art

[0002] Abnormal behavior detection in crowds is one of the research hotspots in the field of computer vision and is crucial to indoor and outdoor safety. Since the amount of data stored in surveillance videos is large and abnormal behavior is relatively rare, manual screening requires a lot of time and effort. Therefore, real-time and efficient abnormal behavior detection algorithms can greatly improve the intelligence level of the surveillance system, save manpower and material resources, and play an important role in protecting the safety of people's lives and property and maintaining social security and stability. Therefore, the research on video abnormal behavior detection algorithms is urgent and of great significance.

[0003] In recent years, with the emergence of abnormal behavior detection datasets in complex scenarios and the upgrading of computer hardware, many domestic and foreign scholars have conducted relevant research on methods for detecting abnormal crowd behavior that combine spatiotemporal features. The key idea of ​​the prediction-based method is to predict future frames given a video clip. In 2023, Wang et al. proposed STR-VAD, which uses an attention mechanism to focus on the spatiotemporal relationship between objects, and uses a dynamic pattern generator to store the spatiotemporal relationship between objects to improve the model's ability to detect abnormal frames. In 2024, Ning et al. proposed MAMC, which uses a memory-enhanced appearance-action consistency model to comprehensively capture the spatiotemporal information between objects and scenes to detect abnormal behavior in videos. In 2023, Le et al. proposed ASTNet, which uses a dual-branch structure to effectively extract spatiotemporal information, and proposes a residual channel attention module to assign weights to features and reduce redundant information. However, the patent of "A method for detecting abnormal crowd behavior based on autoencoders" and the above-mentioned method ignore the diversity of target sizes in complex scenes. When only a single-scale feature is extracted using a convolutional neural network, it will cause the problem of insufficient extraction of features of different scales. At the same time, the time feature used is unidirectional, ignoring the effective extraction of reverse time information. Therefore, this field still needs the introduction of more ideas to achieve comprehensive and in-depth utilization of spatiotemporal features. Summary of the invention

[0004] In view of the fact that the existing methods only use convolutional neural networks to extract single-scale features, which will result in insufficient extraction of features of different scales and ignore the problem of effective extraction of reverse time information, the present invention provides a method for detecting abnormal behavior of a crowd by combining multi-scale features and bidirectional time; the present invention aims to utilize the multi-scale features of the encoder stage to effectively extract target features of different sizes in complex scenes, maximize the advantages of multi-scale features in representing target information, and effectively utilize bidirectional time information. By introducing a global cross-channel feature extraction module, a global spatiotemporal information enhancement module, a multi-scale feature fusion module and a bidirectional convolutional long short-term memory module, the network can fully extract and utilize the multi-scale local features and global semantic features in the video frames, as well as the bidirectional time information of the video frames, so as to improve the detection capability of the network model and achieve the technical effect of accurate and efficient detection of abnormal behavior.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention proposes a method for detecting abnormal behavior of a crowd by combining multi-scale features and bidirectional time, the method comprising the following steps:

[0007] Step 1: Obtain abnormal behavior detection data and perform cropping, grayscale conversion, and pseudo-anomaly generation operations on it;

[0008] The specific generation process includes:

[0009] 1) Obtain abnormal behavior detection data X N , which is defined as follows:

[0010] X N =(I n ,I n+1 ,...,I n+T-1 ) (1)

[0011] Where, X N is a sequence of consecutive frames from the original training data, I n represents a randomly selected starting frame from the video, and T represents a fixed number of consecutive frames;

[0012] 2) The frame skipping pseudo-anomaly sequence generated by the pseudo-anomaly synthesizer is defined as follows:

[0013] X P =(I n ,I n+s ,...,I n+(T-1)s ) (2)

[0014] Where, X P represents the generated pseudo-anomaly sequence, S represents the number of skipped frames, and the pseudo-anomaly data is only introduced during training.

[0015] Step 2: Construct a crowd abnormal behavior detection network model combining multi-scale features and bidirectional time (MSFFBT);

[0016] The network model includes four parts: encoder, multi-scale feature fusion module, bidirectional convolutional long short-term memory module, and decoder; among which:

[0017] The encoder extracts local spatiotemporal features of the input video frame and performs downsampling;

[0018] The multi-scale feature fusion module extracts local spatial information and global semantic representation of video frames at different scales;

[0019] The bidirectional convolutional long short-term memory module performs forward and reverse time information extraction;

[0020] The decoder is used to restore the input resolution and realize the reconstruction of the input frame.

[0021] Furthermore, the encoder uses STEAL-Net as the backbone network to extract local spatiotemporal features, and adds a global cross-channel feature extraction module after the downsampling operation to obtain global dependencies, thereby reducing the loss of key information caused by downsampling;

[0022] Furthermore, the decoder employs multiple upsampling layers to reconstruct the latent space into an output of the same size as the original frame.

[0023] Furthermore, the global cross-channel feature extraction module is specifically defined as follows: first, 1×1 convolution and softmax function are used to obtain attention weights, and the obtained attention weights are matrix multiplied with the original features to obtain global context features; then the kernel size is selected according to the number of channels to capture cross-channel interactions; finally, the addition operation is used to aggregate the global cross-channel features to the features of each position, and the specific definition is shown in the following formula:

[0024] F GC-CFE =F E-out +sigmod(Conv1(y)) (3)

[0025] y=F E_out ×SoftMax(Conv 1×1 (F E_out )) (4)

[0026] In the formula, F E-out represents the output of each stage of the encoder, F GC-CFE represents the output of the global cross-channel feature extraction module, y represents the acquired attention weight, Conv1 is a 1D convolution, and its kernel size k determines the interaction range. The definition of k is as follows:

[0027]

[0028] Where C represents the number of channels, |t| odd represents the odd number closest to t, and γ and b are constants.

[0029] The multi-scale feature fusion module MSFF adopts a parallel structure and is divided into two branches: a global feature extraction branch and a multi-scale local feature extraction branch;

[0030] The TransConv module in the global feature extraction branch captures spatial and temporal global information, and then feeds the acquired global features into the channel attention mechanism (CA), which weights the features of each channel, highlights important feature channels, and improves the model's ability to express features;

[0031] The multi-scale local feature extraction branch adopts different expansion rates for each input channel to form an atrous spatial convolution module (ASC), which uses receptive fields of different scales to extract multi-scale feature information. The multi-scale local features are input into the spatial attention mechanism (SA) to enhance local details and suppress irrelevant areas.

[0032] Finally, the results generated by each branch will be feature fused and connected to the Inverted Residual Multilayer Perceptron (IRMLP) mechanism, which, to a certain extent, avoids the problems of gradient vanishing, explosion, and network degradation, thereby effectively capturing local and global spatiotemporal feature information at multiple scales. The definitions of CA, SA, and IRMLP are shown in equations (6) to (8):

[0033] CA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x))) (6)

[0034] SA(x)=σ(f 1×1 (Concat[AvgPool(x),MaxPool(x)])) (7)

[0035] IRMLP(x)=f 1×1 (f 3×3 (f 1×1 (x)))+x (8)

[0036] In the formula, σ represents the Sigmoid function, f n×nis a convolution operation with a kernel size of n×n, MLP represents a multi-layer perceptron, AvgPool represents average pooling, MaxPool represents maximum pooling, and x represents the input feature value; the calculation process of the multi-scale feature fusion module MSFF is defined as shown in equations (9) to (11):

[0037] G=CA(R(Input)) (9)

[0038] L=SA(ASC(Input)) (10)

[0039] Output=IRMLP(Concat[G,L]) (11)

[0040] Where R represents the TransConv operation, L represents the output of the multi-scale local feature extraction branch, and G represents the output of the global feature extraction branch.

[0041] The TransConv module combines the convolution and Transformer modules, which can retain the global features to the greatest extent while paying attention to the local detail information. The specific feature extraction process includes: the features output by the encoder are first subjected to two three-dimensional convolutions to extract local features; then they are sent to the Transformer module through the average pooling layer, and the sigmoid function is applied to the calculated weights to promote relevant features and suppress irrelevant features; finally, the features are added to the original features for feature fusion, so that the model can use feature information at different levels, as defined in the following formula:

[0042]

[0043] In the formula, F and F' represent the input and output feature maps respectively. represents element-by-element addition, represents the element-wise product, Y = W2δ(W1F), represents the output of two 3D convolutions, where δ represents the ReLU activation function, W1 and W2 are the weight sets of the two 3D convolutional layers; S(Y) represents the Transformer block, which is defined as follows:

[0044] S(Y)=f(norm(m(norm(Y))+Y))+m(norm(Y))+Y(13)

[0045] In the formula, m represents The multi-head attention layer has 2 heads, f represents the feed-forward neural network layer, and norm represents layer normalization.

[0046] The bidirectional convolutional long short-term memory module uses two ConvLSTMs to process the input data features into two forward and reverse paths to form a BiConvLSTM module. After the features of the multidimensional vector are input, the input data is processed by ConvLSTMs in both forward and reverse directions. The obtained output data depends not only on the previous frame, but also on the upcoming frame in the sequence, which can capture long-term and short-term potential representations. The hidden layer state and the storage unit state are continuously updated through convolution operations in two directions. Finally, the final hidden layer state is determined by the hidden layer states in two directions, which can effectively improve the feature extraction ability of the neural network and overcome the gradient disappearance problem of the neural network. The definition is shown in the following formula:

[0047]

[0048] In the formula, y t represents the final output of the bidirectional spatiotemporal information, t=1,2,···,T-1,T, represents the t-th frame of the video; hf, h b Represent the hidden states of the forward and reverse ConvLSTM units respectively; b is the bias term; tanh represents the hyperbolic tangent function, which can realize the nonlinear combination output of the forward and reverse states; represents the forward weight matrix; represents the inverse weight matrix.

[0049] The ConvLSTM uses convolution operations to convert input to state and state to state. t 、Forget gate t 、Storage unit C t and output gate o t The input gate, output gate, and forget gate act as control gates for accessing, updating, and clearing storage units to obtain the spatiotemporal correlation of the network, as shown in equations (15) to (19):

[0050]

[0051]

[0052] In the formula, h t represents the hidden state; x t Represents input data; W xi , W hi , W ci , W xf , W hf , W cf , W xc , W hc , W xo , W ho , W co represents the weight matrix; bi 、b f 、b c 、b o represents the bias term; Represents the Hadamard operation.

[0053] The overall loss function of the network model includes two parts: minimizing the reconstruction error of normal frames and maximizing the reconstruction error of pseudo-abnormal frames. The calculation process is shown in the following formula:

[0054]

[0055] In the formula, I n+t Indicates the normal frame of input, represents the reconstructed normal frame,||.|| F represents the Frobenius norm, C is the feature dimension, T is the number of frames in the input sequence, H is the height of the video frame, and W is the width of the video frame;

[0056]

[0057] In the formula, I n+ts represents the input pseudo-abnormal frame, It represents the reconstructed pseudo-abnormal frame. The negative sign increases the reconstruction error of the pseudo-abnormal frame and limits the network's ability to reconstruct the abnormal frame.

[0058] The final network training loss L is as follows:

[0059]

[0060] In the formula, X represents the input of the network, L represents the overall loss function of the network model, and L N represents the reconstruction loss of normal frames, L P represents the reconstruction loss of pseudo-anomalous frames.

[0061] Step 3: Input the abnormal behavior detection data obtained in step 1 into the network model obtained in step 2 to perform abnormal behavior detection of crowds combining multi-scale features and bidirectional time.

[0062] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0063] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] The method of the present invention is suitable for abnormal behavior detection in videos. Compared with the patent "A method for detecting abnormal behavior of a crowd based on autoencoder", a multi-scale feature fusion module is added between the encoder and the decoder to extract and utilize the multi-scale local features and global semantic features in the video frames, and a bidirectional convolutional long short-term memory module is added to make full use of the bidirectional time information of the video frames, thereby improving the detection capability of abnormal behavior and improving the overall detection accuracy of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of a method for detecting abnormal behavior of a crowd combining multi-scale features and bidirectional time according to the present invention;

[0067] Figure 2 is a schematic diagram of a global cross-channel feature extraction module of the present invention;

[0068] Figure 3 is a schematic diagram of a multi-scale feature fusion module of the present invention;

[0069] Figure 4 is a schematic diagram of the TransConv module of the present invention;

[0070] Figure 5 Schematic diagram of a bidirectional convolutional long short-term memory module of the present invention;

[0071] Figure 6 Schematic diagram of the ConvLSTM module of the present invention;

[0072] Figure 7 It is a visualization result diagram of ROC curves of different algorithms;

[0073] Figure 8 It is a visualization result diagram of input frame, reconstructed frame and reconstruction error. DETAILED DESCRIPTION

[0074] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0075] A method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time, such as Figure 1 As shown, the following steps are included:

[0076] Step 1: Obtain abnormal behavior detection data, and perform cropping, grayscale conversion, and pseudo-anomaly generation operations on it to adapt to network input;

[0077] Specifically, UCSDPed1, UCSDPed2 and ShanghaiTech are used as data sets for the abnormal behavior detection task of the crowd in the present invention. UCSDPed1 contains 34 training videos and 36 test videos, each video has 200 grayscale frames with a resolution of 238x158. UCSDPed2 contains 16 training videos and 12 test videos, each video has about 170 grayscale frames with a resolution of 360x240. Abnormal events include: the circulation of non-sidewalk objects on the sidewalk (such as trucks, bicycles), abnormal pedestrian movement patterns (such as crossing the lawn, running). ShanghaiTech contains 330 training videos and 107 test videos, and the resolution of each video frame is 856x480. Abnormal events include: climbing over fences, riding bicycles, running, and skateboarding.

[0078] The specific generation process includes:

[0079] 1) Obtain abnormal behavior detection data X N , which is defined as follows:

[0080] X N =(I n ,I n+1 ,...,I n+T-1 ) (1)

[0081] Where, X N is a sequence of consecutive frames from the original training data, I n represents a randomly selected starting frame from the video, and T represents a fixed number of consecutive frames;

[0082] 2) The frame skipping pseudo-anomaly sequence generated by the pseudo-anomaly synthesizer is defined as follows:

[0083] X P =(I n ,I n+s ,...,I n+(T-1)s )(2)

[0084] Where, X Prepresents the generated pseudo-anomaly sequence, adopting the work of Marcella et al. (Astrid M, Zaheer MZ, Lee S I. Synthetic temporal anomaly guided end-to-end video anomaly detection [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 207-214.), s represents the number of skipped frames, which can be randomly selected as 2, 3, 4, 5. Pseudo-anomaly data is only introduced during training.

[0085] Step 2: Construct a crowd abnormal behavior detection network model combining multi-scale features and bidirectional time (MSFFBT);

[0086] The network model includes four parts: encoder, multi-scale feature fusion module, bidirectional convolutional long short-term memory module, and decoder; among which:

[0087] The encoder extracts local spatiotemporal features of the input video frame and performs downsampling;

[0088] The multi-scale feature fusion module extracts local spatial information and global semantic representation of video frames at different scales;

[0089] The bidirectional convolutional long short-term memory module performs forward and reverse time information extraction;

[0090] The decoder is used to restore the input resolution and realize the reconstruction of the input frame.

[0091] Furthermore, the encoder uses STEAL-Net as the backbone network to extract local spatiotemporal features, and adds a global cross-channel feature extraction module after the downsampling operation to obtain global dependencies, thereby reducing the loss of key information caused by downsampling;

[0092] Furthermore, the decoder adopts multiple upsampling layers composed of deconvolution blocks to reconstruct the encoded latent space features into an output of the same size as the original frame, and batch normalization and activation functions are added after each upsampling layer to further enhance the expression ability of the network.

[0093] Furthermore, the global cross-channel feature extraction module (GC-CFE) can strengthen spatial features and establish long-range dependencies between pixels, improving the deficiency of convolutional neural networks that only extract local features of images. At the same time, it can perform cross-channel interaction of features and further obtain the correlation between channels within a certain range, so as to obtain a more comprehensive and accurate feature representation. Figure 2As shown, the specific definition is: first, use 1×1 convolution and softmax function to obtain attention weights, and perform matrix multiplication of the obtained attention weights with the original features to obtain global context features; then select the kernel size according to the number of channels to capture cross-channel interactions; finally, use addition operation to aggregate the global cross-channel features to the features at each position. The specific definition is shown in the following formula:

[0094] F GC-CFE =F E-out +sigmod(Conv1(y)) (3)

[0095] y=F E_out ×SoftMax(Conv 1×1 (F E_out )) (4)

[0096] In the formula, F E-out represents the output of each stage of the encoder, F GC-CFE Represents the output of the global cross-channel feature extraction module, y represents the acquired attention weight, Conv1 is a 1D convolution. In order to make the high-dimensional channel obtain long-distance interaction and the low-dimensional channel obtain relatively short-distance interaction, the kernel size adaptive selector is used to automatically select the kernel size of Conv1. The definition of the kernel size k is shown in the following formula:

[0097]

[0098] In the formula, we adopt the work of Wang et al. (Wang Q, Wu B, Zhu P, et al. ECA-Net: Efficient channel attention for deep convolutional neural networks [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 11534-11542.), | t odd To represent the odd number closest to t, γ is set to 2 and b is set to 1.

[0099] The multi-scale feature fusion module MSFF adopts a parallel structure and is divided into two branches: a global feature extraction branch and a multi-scale local feature extraction branch. Figure 3 As shown;

[0100] The TransConv module in the global feature extraction branch captures spatial and temporal global information, and then feeds the acquired global features into the channel attention mechanism (CA), which weights the features of each channel, highlights important feature channels, and improves the model's ability to express features;

[0101] The multi-scale local feature extraction branch adopts different expansion rates for each input channel to form an atrous spatial convolution module (ASC), which uses receptive fields of different scales to extract multi-scale feature information. The multi-scale local features are input into the spatial attention mechanism (SA) to enhance local details and suppress irrelevant areas;

[0102] Finally, the results generated by each branch will be feature fused and connected to the Inverted Residual Multilayer Perceptron (IRMLP) mechanism, which, to a certain extent, avoids the problems of gradient vanishing, explosion, and network degradation, thereby effectively capturing local and global spatiotemporal feature information at multiple scales. The definitions of CA, SA, and IRMLP are shown in equations (6) to (8):

[0103] CA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x))) (6)

[0104] SA(x)=σ(f 1×1 (Concat[AvgPool(x),MaxPool(x)])) (7)

[0105] IRMLP(x)=f 1×1 (f 3×3 (f 1×1 (x)))+x (8)

[0106] In the formula, σ represents the Sigmoid function, f n×n is a convolution operation with a kernel size of n×n, MLP represents a multi-layer perceptron, AvgPool represents average pooling, MaxPool represents maximum pooling, and x represents the input feature value; the calculation process of the multi-scale feature fusion module MSFF is defined as shown in equations (9) to (11):

[0107] G=CA(R(Input)) (9)

[0108] L=SA(ASC(Input)) (10)

[0109] Output=IRMLP(Concat[G,L]) (11)

[0110] Where R represents the TransConv operation, L represents the output of the multi-scale local feature extraction branch, and G represents the output of the global feature extraction branch.

[0111] The TransConv module combines the convolution and Transformer modules, which can retain the global features to the greatest extent while paying attention to the local details. Figure 4 As shown in the figure, the specific feature extraction process includes: the features output by the encoder are first subjected to two three-dimensional convolutions to extract local features; then they are sent to the Transformer module through the average pooling layer, and the sigmoid function is applied to the calculated weights to promote relevant features and suppress irrelevant features; finally, the features are added to the original features for feature fusion, so that the model can use feature information at different levels, as defined in the following formula:

[0112]

[0113] In the formula, F and F' represent the input and output feature maps respectively. represents element-by-element addition, represents the element-wise product, Y = W2δ(W1F), represents the output of two 3D convolutions, where δ represents the ReLU activation function, W1 and W2 are the weight sets of the two 3D convolutional layers; S(Y) represents the Transformer block, which is defined as follows:

[0114] S(Y)=f(norm(m(norm(Y))+Y))+m(norm(Y))+Y (13)

[0115] In the formula, m represents The multi-head attention layer has 1 head, f represents the feedforward neural network layer, and norm represents layer normalization. First, Y is normalized, concatenated with itself after passing through the multi-head attention layer, and then the concatenated features are normalized again, passed through the feedforward neural network, and concatenated with itself again to obtain the final result.

[0116] The bidirectional convolutional long short-term memory module (BiConvLSTM) uses two ConvLSTMs to process the input data features into two forward and reverse paths to form a BiConvLSTM module. Figure 5As shown in the figure, after the features of the multidimensional vector are input, the input data is processed by ConvLSTM in both the forward and reverse directions. The output data obtained depends not only on the previous frame, but also on the upcoming frame in the sequence, which can capture long-term and short-term potential representations. The hidden layer state and the storage unit state are continuously updated through convolution operations in both directions. Finally, the final hidden layer state is determined by the hidden layer state in both directions, which can effectively improve the feature extraction ability of the neural network and overcome the gradient disappearance problem of the neural network. The definition is shown in the following formula:

[0117]

[0118] In the formula, y t represents the final output of the bidirectional spatiotemporal information, t=1,2,···,T-1,T, represents the t-th frame of the video; hf, h b Represent the hidden states of the forward and reverse ConvLSTM units respectively; b is the bias term; tanh represents the hyperbolic tangent function, which can realize the nonlinear combination output of the forward and reverse states; represents the forward weight matrix; represents the inverse weight matrix.

[0119] The ConvLSTM uses convolution operations to convert input to state and state to state, such as Figure 6 As shown, by the input gate i t 、Forget gate t 、Storage unit C t and output gate o t The input gate, output gate, and forget gate act as control gates for accessing, updating, and clearing storage units to obtain the spatiotemporal correlation of the network, as shown in equations (15) to (19):

[0120]

[0121] In the formula, h t represents the hidden state; x t Represents input data; W xi , W hi , W ci , W xf , W hf , W cf , W xc , W hc , W xo , W ho , W co represents the weight matrix; b i 、b f 、b c 、b o represents the bias term; Represents the Hadamard operation.

[0122] The overall loss function of the network model includes two parts: minimizing the reconstruction error of normal frames and maximizing the reconstruction error of pseudo-abnormal frames. The calculation process is shown in the following formula:

[0123]

[0124] In the formula, I n+t Indicates the normal frame of input, represents the reconstructed normal frame,||.|| F represents the Frobenius norm, C is the feature dimension, T is the number of frames in the input sequence, H is the height of the video frame, and W is the width of the video frame;

[0125]

[0126] In the formula, I n+ts represents the input pseudo-abnormal frame, It represents the reconstructed pseudo-abnormal frame. The negative sign increases the reconstruction error of the pseudo-abnormal frame and limits the network's ability to reconstruct the abnormal frame.

[0127] The final network training loss L is as follows:

[0128]

[0129] In the formula, X represents the input of the network, L represents the overall loss function of the network model, and L N represents the reconstruction loss of normal frames, L P represents the reconstruction loss of pseudo-anomalous frames.

[0130] Step 3: Input the abnormal behavior detection data obtained in step 1 into the network model obtained in step 2 to perform abnormal behavior detection of crowds combining multi-scale features and bidirectional time.

[0131] Specifically, the video frames are preprocessed, including uniformly cropping to a size of 256x256, normalizing pixel values, and converting to grayscale images; the Adam optimizer with an initial learning rate of 1e-4 is used, and the learning rate is adjusted in a downward trend according to the StepLR method. The batch size is set to 4, the training epoch is 60, and the learning rate decreases by 50% after every 15 epochs of training. The results are obtained through multiple tests, as shown in Table 1. Figure 7 and Figure 8 .

[0132] Table 1 Comparison of AUC results of different methods on public datasets (unit: %)

[0133]

[0134] Table 1 lists the comparison results of the frame-level AUC of the method of the present invention with some classic methods on the public data sets UCSD Ped1, UCSD Ped2 and ShanghaiTech. As can be seen from Table 1, the MSFFBT algorithm achieves an AUC of 98.4% on the UCSD Ped2 data set, 86.8% on the UCSD Ped1 data set, and 78.21% on the ShanghaiTech data set, which is in a leading position. Among them, MPN and STEAL-Net use retrained models. From the comparison results, it can be seen that the MSFFBT algorithm adds a multi-scale feature fusion module and a bidirectional convolutional long short-term memory module on the basis of GST-STEAL-Net to extract local features, global semantic representations and bidirectional temporal features at different scales for anomaly detection, which can effectively improve the AUC index, indicating that it is very important to make full use of spatiotemporal information when detecting abnormal behavior. The GC-CFE module, MSFF module and BiConvLSTM module of the MSFFBT algorithm fully extract the spatiotemporal information features and effectively avoid the acquisition of redundant information, playing a key role in abnormal behavior detection.

[0135] Figure 7 The ROC performance comparison curves of the proposed method, MPN algorithm and STEAL-Net algorithm are shown in Figure 1. It can be observed from the figure that the proposed method has a larger TPR, a smaller FPR, and a ROC curve closer to the (0,1) point. The curve position is higher than that of the other two comparison methods, reflecting good performance and effectiveness. In general, the proposed algorithm has good performance on the three public data sets and is very competitive.

[0136] Figure 8 The method of the present invention was demonstrated in UCSD Ped2 ( Figure 8 (a), 8(b)), UCSD Ped1( Figure 8 (c), 8(d)) and ShanghaiTech( Figure 8 (e), 8(f)) Visualization results of the reconstructed frames and the error map between the reconstructed frames and the real frames on the test dataset. The abnormal events in the frame are marked by rectangular boxes. Figure 8 (a), 8(c) and 8(e), we can see that the proposed method has a good reconstruction effect on normal frames. It can also be seen from the error graph that the error between the reconstructed frame and the real frame is very small. Figure 8 As can be seen from (b), 8(d) and 8(f), the abnormal behavior area in the reconstructed frame appears blurred, which is significantly different from the real frame, indicating that the anomaly has been detected; the abnormal behavior area in the reconstructed error map is brighter and the error is obvious, which intuitively shows that the proposed algorithm has a good performance in detecting abnormal behavior.

[0137] It is intuitively demonstrated that the method of the present invention has good performance in detecting abnormal behaviors.

[0138] It should be noted that the embodiments of the present invention may be provided as methods, systems and / or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0140] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0141] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0142] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time, characterized in that: The method comprises the following steps: Step 1: Obtain abnormal behavior detection data and perform cropping, grayscale conversion, and pseudo-anomaly generation operations on it; Step 2: Construct a crowd abnormal behavior detection network model that combines multi-scale features and bidirectional time; Step 3: Input the abnormal behavior detection data obtained in step 1 into the network model obtained in step 2 to perform abnormal behavior detection of crowds combining multi-scale features and bidirectional time.

2. According to claim 1, a method for detecting abnormal crowd behavior combining multi-scale features and bidirectional time, characterized in that: The step 1 obtains abnormal behavior detection data and performs cropping, grayscale conversion, and pseudo-anomaly generation operations on it. The specific generation process includes: 1) Obtain abnormal behavior detection data X N , which is defined as follows: X N =(I n ,I n+1 ,...,I n+T-1 ) (1) Where, X N is a sequence of consecutive frames from the original training data, I n represents a randomly selected starting frame from the video, and T represents a fixed number of consecutive frames; 2) The frame skipping pseudo-anomaly sequence generated by the pseudo-anomaly synthesizer is defined as follows: X P =(I n ,I n+s ,...,I n+(T-1)s ) (2) Where, X P represents the generated pseudo-anomaly sequence, S represents the number of skipped frames, and the pseudo-anomaly data is only introduced during training.

3. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 1, characterized in that: Step 2 constructs a crowd abnormal behavior detection network model combining multi-scale features and bidirectional time. The network model includes four parts: encoder, multi-scale feature fusion module, bidirectional convolutional long short-term memory module, and decoder; wherein: The encoder extracts local spatiotemporal features of the input video frame and performs downsampling; The multi-scale feature fusion module extracts local spatial information and global semantic representation of video frames at different scales; The bidirectional convolutional long short-term memory module performs forward and reverse time information extraction; The decoder is used to restore the input resolution and realize the reconstruction of the input frame.

4. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 3 is characterized in that: The encoder uses STEAL-Net as the backbone network to extract local spatiotemporal features, and adds a global cross-channel feature extraction module after the downsampling operation to obtain global dependencies and reduce the loss of key information caused by downsampling; the decoder uses multiple upsampling layers to reconstruct the latent space into an output of the same size as the original frame.

5. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 4 is characterized in that: The specific definition of the global cross-channel feature extraction module is: first, a 1×1 convolution and a softmax function are used to obtain attention weights, and a matrix multiplication is performed on the obtained attention weights and the original features to obtain global context features; Then the kernel size is selected according to the number of channels to capture cross-channel interactions; finally, the addition operation is used to aggregate the global cross-channel features to the features at each position, as defined in the following formula: F GC-CFE =F E-out +sigmod(Conv1(y)) (3) y=F E_out ×SoftMax(Conv 1×1 (F E_out )) (4) In the formula, F E-out represents the output of each stage of the encoder, F GC-CFE represents the output of the global cross-channel feature extraction module, y represents the acquired attention weight, Conv1 is a 1D convolution, and its kernel size k determines the interaction range. The definition of k is as follows: Where C represents the number of channels, |t| odd represents the odd number closest to t, and γ and b are constants.

6. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 3, characterized in that: The multi-scale feature fusion module adopts a parallel structure and is divided into two branches: a global feature extraction branch and a multi-scale local feature extraction branch; The TransConv module in the global feature extraction branch captures spatial and temporal global information, and then feeds the acquired global features into the channel attention mechanism CA; The multi-scale local feature extraction branch adopts different expansion rates for each input channel to form a dilated spatial convolution module ASC; the multi-scale local features are input into the spatial attention mechanism SA to enhance local details and suppress irrelevant areas; Finally, the results generated by each branch will be feature fused and connected to the inverted residual multilayer perceptron mechanism IRMLP. The definitions of CA, SA, and IRMLP are shown in equations (6) to (8): CA(x)=σ(MLP(AvgPool(x))+MLP(MaxPool(x))) (6) SA(x)=σ(f 1×1 (Concat[AvgPool(x),MaxPool(x)])) (7) IRMLP(x)=f 1×1 (f 3×3 (f 1×1 (x)))+x (8) In the formula, σ represents the Sigmoid function, f n×n It is a convolution operation with a kernel size of n×n, MLP represents a multi-layer perceptron, AvgPool represents average pooling, MaxPool represents maximum pooling, and x represents the input feature value; The calculation process of the multi-scale feature fusion module MSFF is defined as shown in equations (9) to (11): G=CA(R(Input)) (9) L=SA(ASC(Input)) (10) Output=IRMLP(Concat[G,L]) (11) Where R represents the TransConv operation, L represents the output of the multi-scale local feature extraction branch, and G represents the output of the global feature extraction branch.

7. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 6, characterized in that: The TransConv module combines the convolution and Transformer modules. The specific feature extraction process includes: the features output by the encoder are first subjected to two three-dimensional convolutions to extract local features; then they are sent to the Transformer module through the average pooling layer, and the sigmoid function is applied to the calculated weights to promote relevant features and suppress irrelevant features; finally, the features are added to the original features for feature fusion, which is defined as follows: In the formula, F and F' represent the input and output feature maps respectively. represents element-by-element addition, represents the element-wise product, Y = W2δ(W1F), represents the output of two 3D convolutions, where δ represents the ReLU activation function, W1 and W2 are the weight sets of the two 3D convolutional layers; S(Y) represents the Transformer block, which is defined as follows: S(Y)=f(norm(m(norm(Y))+Y))+m(norm(Y))+Y (13) In the formula, m represents The multi-head attention layer has 2 heads, f represents the feed-forward neural network layer, and norm represents layer normalization.

8. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 3 is characterized in that: The bidirectional convolutional long short-term memory module uses two ConvLSTMs to process the input data features into two forward and reverse paths to form a BiConvLSTM module. After the features of the multidimensional vector are input, the input data is processed by the ConvLSTM in the forward and reverse directions. The hidden layer state and the storage unit state are continuously updated through convolution operations in both directions. Finally, the final hidden layer state is determined by the hidden layer states in the two directions, which is defined as shown in the following formula: In the formula, y t represents the final output of the bidirectional spatiotemporal information, t=1,2,···,T-1,T, represents the t-th frame of the video; hf, h b denote the hidden states of the forward and reverse ConvLSTM units, respectively; b is the bias term; tanh denotes the hyperbolic tangent function, represents the forward weight matrix; represents the inverse weight matrix.

9. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 8, characterized in that: The ConvLSTM uses convolution operations to convert input to state and state to state. t 、Forget gate t 、Storage unit C t and output gate o t The input gate, output gate, and forget gate act as control gates for accessing, updating, and clearing storage units to obtain the spatiotemporal correlation of the network, as shown in equations (15) to (19): In the formula, h t represents the hidden state; x t Represents input data; W xi , W hi , W ci , W xf , W hf , W cf , W xc , W hc , W xo , W ho , W co represents the weight matrix; b i 、b f 、b c 、b o represents the bias term; Represents the Hadamard operation.

10. The method for detecting abnormal crowd behavior by combining multi-scale features and bidirectional time according to claim 3, characterized in that: The overall loss function of the network model includes two parts: minimizing the reconstruction error of normal frames and maximizing the reconstruction error of pseudo-abnormal frames. The calculation process is shown in the following formula: In the formula, I n+t Indicates the normal frame of input, represents the reconstructed normal frame,||.|| F represents the Frobenius norm, C is the feature dimension, T is the number of frames in the input sequence, H is the height of the video frame, and W is the width of the video frame; In the formula, I n+ts represents the input pseudo-abnormal frame, It represents the reconstructed pseudo-abnormal frame. The negative sign increases the reconstruction error of the pseudo-abnormal frame and limits the network's ability to reconstruct the abnormal frame. The final network training loss L is as follows: In the formula, X represents the input of the network, L represents the overall loss function of the network model, and L N represents the reconstruction loss of normal frames, L P represents the reconstruction loss of pseudo-anomalous frames.

Images