A weakly supervised video anomaly detection method based on self-guided encoder

Abstract

The present invention proposes a weakly supervised video anomaly detection method based on a self-guided encoder, which specifically includes the following steps: processing the original video data; inputting the video data into a backbone network for feature calculation, and inputting it into a self-guided feature encoder for training. The two branches of the encoder will respectively output the frame-level anomaly score of the video; the frame-level anomaly score of the self-guided branch is fed into a t-distribution factor analysis module for fitting to obtain a frame-level pseudo-label; using pseudo-labels to supervise the training of the self-guided feature encoder again, continuously updating the pseudo-labels, and the trained encoder attention branch outputs the predicted frame-level anomaly score; and performing anomaly judgment according to a set anomaly threshold. The present invention utilizes the self-guided feature encoder to capture the dependencies between different abnormal events while automatically locating the most abnormal part of the video frame, thereby achieving the purpose of improving anomaly detection accuracy and reducing false alarm rate.

Description

Technical Field

[0001] The present invention belongs to the field of video anomaly detection, and in particular relates to a weakly supervised video anomaly detection method based on a self-guided encoder. Background Art

[0002] Surveillance cameras and other security equipment are rapidly becoming ubiquitous in both urban and rural areas. Effectively processing and analyzing the massive amounts of video data generated by these devices is a pressing challenge in the security field. Video anomaly detection uses artificial intelligence algorithms to analyze video data, automatically locate the specific locations of unusual events in various scenarios, and issue timely warnings. This research holds significant theoretical and practical value for preventing and resolving public safety incidents.

[0003] Video anomaly detection methods based on supervised learning require labels for each training example in the training set, and for video data, each video must be annotated. Obviously, labeling massive amounts of surveillance video is extremely challenging and difficult to implement. Therefore, many studies adopt an unsupervised paradigm, or one-class classification. This paradigm requires only normal training examples during training. The goal of unsupervised anomaly detection is to learn a representation of "normal." During testing, abnormal examples are used for testing, and test examples that deviate from normal are detected as anomalies. However, because the definition of abnormality is sometimes context-dependent, a normal event in one setting may appear abnormal in another. For example, running is normal in a playground, but abnormal in a shopping mall or other indoor setting. Because unsupervised methods cannot include all normal events in the training set, normal events that did not appear in the training set may be detected as anomalies during testing, resulting in reduced anomaly detection accuracy. The weakly supervised video anomaly detection method uses video-level pseudo-labels for training. This label only indicates whether there is an anomaly in the current video, and does not annotate a specific video frame. With minimal human effort, it greatly improves the overall accuracy of the anomaly detection network model. Summary of the Invention

[0004] The present invention provides a weakly supervised video anomaly detection method based on a self-guided encoder, which can capture the dependencies between different abnormal events while automatically locating the most abnormal part of the video frame, thereby achieving the purpose of improving anomaly detection accuracy and reducing the false alarm rate.

[0005] To achieve the above object, the present invention uses the following technical solutions:

[0006] A weakly supervised video anomaly detection method based on a self-guided encoder includes the following steps:

[0007] Step 1: Process the original video data;

[0008] Step 2: The video data is fed into the backbone network for feature calculation and then fed into the self-guided feature encoder for training. The anomaly classification heads of the two branches of the encoder will output the frame-level anomaly scores of the video respectively.

[0009] Step 3: The frame-level anomaly scores of the self-guided branch are fed into the factor analysis module for fitting to obtain frame-level pseudo labels.

[0010] Step 4: Use pseudo-label supervision to train the self-guided feature encoder again. During the training process, the pseudo-label is continuously updated until the model performance reaches the best. The attention branch of the encoder outputs the final frame-level anomaly score.

[0011] Step 5: Anomaly judgment is performed based on the set anomaly threshold. Frames with anomaly scores greater than the threshold are considered abnormal frames, otherwise they are considered normal frames.

[0012] Furthermore, step 1 includes the following steps for the video data: performing frame extraction on the original unprocessed surveillance video, extracting 16 frames per second, and treating it as a video segment; each video will be divided into several video segments.

[0013] Furthermore, the structure of the self-guided encoding signature encoder in step 2 is:

[0014] The self-guided encoder proposed in this paper consists of two branches: the attention mechanism branch and the self-guided channel attention branch;

[0015] Specifically, the attention branch proposed in the present invention is based on the feature map M output by the penultimate layer of the backbone network. b-4 And the feature map M output by the last layer b-5 As input features. Feature map M b-4 The intermediate layer features are obtained by encoding through the encoding unit F1 The feature map is then passed through the encoding unit F2 to generate the attention feature map A:

[0016] A=F2(F1(M b-4 ))

[0017] Then the attention mechanism is used to transform the feature map M b-5 Element-wise multiplication with the attention feature map A plus M b-5 Get the weighted feature map M A :

[0018]

[0019] Finally, the feature map M A Perform global average pooling operation and input to the abnormal classification head hc , classification head h c Output the anomaly score calculated by the attention branch;

[0020] Specifically, the encoding unit F1 consists of a 3D convolution with a kernel size of 1×1×1 and a GELU activation function, and the encoding unit F2 consists of a 3D convolution with a kernel size of 3×3×3 and a Sigmoid activation function. c It is essentially a fully connected layer.

[0021] Specifically, the self-guided channel attention branch proposed in this paper consists of a dilated convolution module, a SE module, and a classification head h g This branch only uses the feature map M output by the penultimate layer of the backbone network b-4 As input, the encoding unit F1 is shared with the attention branch, and the intermediate layer features The resulting image is fed into the dilated convolution module to obtain spatiotemporal features at different scales, and its output features are fed into the SE module. The SE module extracts channel attention features from the feature map and learns the dependencies between different anomalies to assist in learning the anomaly feature map.

[0022] The dilated convolution module is composed of two parallel 3D convolution layers DC3d1 and DC3d2 with different convolution kernels and dilation rates. The output features are added to obtain the multi-scale fusion feature map M. d :

[0023]

[0024] Specifically, the SE module is implemented as follows: the input feature map of size H×W×C is first compressed sq , that is, after global pooling, the feature map is compressed to a 1×1×C vector. Then the vector is stimulated by the operation F ex ,The excitation operation uses a fully connected neural network to perform a nonlinear transformation on the ,feature vector to obtain the weights of different channels.

[0025] Specifically, to reduce the amount of computation, the compressed vector is sent to a fully connected layer to reduce its number of channels. This layer uses the ReLu activation function to ensure the nonlinearity of the output features. Another fully connected layer is then used to restore the number of channels in the feature map to the same number of channels as the input feature map, C. A sigmoid activation function is then used to derive the attention feature weights corresponding to each channel. Finally, the channel attention weights are multiplied by the original feature map to achieve channel-wise weight redistribution.

[0026] The feature map output by the SE module is converted into a guided feature map M through the encoding unit F3: M=F3(se(M d ))

[0027] Then through the classification head h g And the softmax operation outputs the anomaly score predicted by the self-guided channel attention branch.

[0028] Specifically, the loss function of the encoder is:

[0029] L tatal =L c +L g

[0030] Among them L c and L g are the loss functions of the attention mechanism branch and the self-guided channel attention branch respectively. Specifically, L c and L g The specific form is:

[0031]

[0032]

[0033] Among them, λ1 and λ2 are hyperparameters. and are the anomaly scores predicted by the attention branch and the self-guided channel branch, respectively. Among them, λ1 and λ2 are hyperparameters. and are the anomaly scores predicted by the attention branch and the self-guided channel branch, L WBCE and topk are the weighted cross entropy loss function and top-k ranking loss, respectively.

[0034] Specifically, the weighted cross entropy loss function is calculated as:

[0035]

[0036] Specifically, the top-k ranking loss is calculated as:

[0037]

[0038] Among them, parameter k represents the number of video clips selected in each video, Ω k is a subset of the video with size k.

[0039] Specifically, the factor analysis module in step 3 refits the frame-level anomaly score of the self-guided branch by the following formula:

[0040]

[0041] Among them, μ k , ψ k ,Σ kis the mean, factor and diagonal covariance matrix of the kth Gaussian component, λ k is the coefficient of each Gaussian component, v k These parameters can be solved using the expectation-maximization algorithm.

[0042] In summary, compared with existing weakly supervised video anomaly detection techniques, this paper offers the following advantages: It proposes a dual-branch self-guided attention feature encoder that effectively captures inter-anomaly dependencies. While processing specific anomaly representations, it automatically focuses on the most anomalous parts of a video frame, thereby assisting in learning anomaly features. Using this encoder for anomaly detection effectively improves detection accuracy and achieves a very low false alarm rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The algorithm flow chart of the present invention is

[0044] Figure 2 Schematic diagram of the network structure of the present invention

[0045] Figure 3 Schematic diagram of the SE module network structure DETAILED DESCRIPTION

[0046] In order to describe the embodiments of the present invention more clearly and completely, the following Figure 1-3 It should be noted that the embodiments of the present invention are only a part thereof and do not represent all embodiments. Therefore, according to the embodiments of the present invention, ordinary technicians in this field can obtain other embodiments, and these embodiments also fall within the scope of protection of the present invention.

[0047] Attachment Figure 1 This is a flowchart of an anomaly detection algorithm according to an embodiment of the present invention. Specifically, a weakly supervised video anomaly detection method based on a self-guided encoder includes the following steps:

[0048] Step 1: Process the original video data;

[0049] Step 2: The video data is fed into the backbone network for feature calculation and then fed into the self-guided feature encoder for training. The anomaly classification heads of the two branches of the encoder will output the frame-level anomaly scores of the video respectively.

[0050] Step 3: The frame-level anomaly scores of the self-guided branch are fed into the factor analysis module for fitting to obtain frame-level pseudo labels.

[0051] Step 4: Use pseudo-label supervision to train the self-guided feature encoder again. During the training process, the pseudo-label is continuously updated until the model performance reaches the best. The attention branch of the encoder outputs the final frame-level anomaly score.

[0052] Step 5: Anomaly judgment is performed based on the set anomaly threshold. Frames with anomaly scores greater than the threshold are judged as abnormal frames, otherwise they are normal frames.

[0053] Furthermore, step 1 includes the following steps for the video data: performing frame extraction on the original unprocessed surveillance video, extracting 16 frames per second, and treating it as a video segment; each video will be divided into several video segments.

[0054] Furthermore, the structure of the self-guided signature encoder in step 2 is as shown in the attached Figure 2 As shown:

[0055] The self-guided encoder constructed in the embodiment of the present invention consists of two branches: an attention mechanism branch and a self-guided channel attention branch;

[0056] Specifically, the embodiment of the present invention adopts the I3D network as the video feature encoder, and the attention branch uses the feature map M output by the penultimate layer of the backbone network. b-4 And the feature map M output by the last layer b-5 As input features. Feature map M b-4 The intermediate layer features are obtained by encoding through the encoding unit F1 The feature map is then passed through the encoding unit F2 to generate the attention feature map A:

[0057] A=F2(F1(M b-4 ))

[0058] in It contains 2n channels, each channel corresponds to a type of abnormal event. This embodiment uses subsequent modules to learn the dependency between different abnormal events to assist in learning the abnormal feature map.

[0059] Furthermore, the attention branch uses the attention mechanism to convert the feature map M b-5 Element-wise multiplication with the attention feature map A plus M b-5 Get the weighted feature map M A :

[0060]

[0061] In the embodiment of the present invention, by A Perform global average pooling operation and input to the abnormal classification head h c , classification head h cIt will output the anomaly score calculated by the attention branch.

[0062] Specifically, in the embodiment of the present invention, the encoding unit F1 is composed of a three-dimensional convolution with a convolution kernel size of 1×1×1 and a GELU activation function, and the encoding unit F2 is composed of a three-dimensional convolution with a convolution kernel size of 3×3×3 and a Sigmoid activation function. c It is essentially a fully connected layer.

[0063] Specifically, the self-guided channel attention branch constructed in this embodiment of the present invention is composed of a dilated convolution module, a SE module, and a classification head h g This branch only uses the feature map M output by the penultimate layer of the backbone network b-4 As input, the encoding unit F1 is shared with the attention branch, and the intermediate layer features The resulting image is fed into the dilated convolution module to obtain spatiotemporal features at different scales, and its output features are fed into the SE module. The SE module extracts channel attention features from the feature map and learns the dependencies between different anomalies to assist in learning the anomaly feature map.

[0064] The dilated convolution module is composed of two parallel 3D convolution layers DC3d1 and DC3d2 with different convolution kernels and dilation rates. The convolution kernel sizes of the two 3D dilated convolution layers are 3×3×3 and 5×5×5, and the dilation rates are 2 and 4, respectively. The output features are added to obtain the multi-scale fusion feature map M. d :

[0065]

[0066] Specifically. Figure 3 As shown, the specific implementation of the SE module in the embodiment of the present invention is as follows: the feature map of the input size H×W×C is first compressed F sq , that is, after global pooling, the feature map is compressed to a 1×1×C vector. Then the vector is stimulated by the operation F ex ,The excitation operation uses a fully connected neural network to perform a nonlinear transformation on the ,feature vector to obtain the weights of different channels.

[0067] Specifically, to reduce the amount of computation, the compressed vector is sent to a fully connected layer to reduce its number of channels. This layer uses the ReLu activation function to ensure the nonlinearity of the output features. Another fully connected layer is then used to restore the number of channels in the feature map to the same number of channels as the input feature map, C. A sigmoid activation function is then used to derive the attention feature weights corresponding to each channel. Finally, the channel attention weights are multiplied by the original feature map to achieve channel-wise weight redistribution.

[0068] The feature map output by the SE module is converted into a guided feature map M through the encoding unit F3: M=F3(se(M d ))

[0069] Then through the classification head h g And the softmax operation outputs the anomaly score predicted by the self-guided channel attention branch.

[0070] Specifically, in the embodiment of the present invention, the classification head h g It consists of a global average pooling operation.

[0071] Specifically, the loss function used in training the self-guided feature encoder in the embodiment of the present invention is:

[0072] L tatal =L c +L g

[0073] Among them L c and L g are the loss functions of the attention mechanism branch and the self-guided channel attention branch respectively. Specifically, L c and L g The specific form is:

[0074]

[0075]

[0076] Wherein, λ1 and λ2 are hyperparameters. In the implementation of the present invention, their corresponding values are set to 1 and 1.3 respectively. and are the anomaly scores predicted by the attention branch and the self-guided channel branch, L WBCE and topk are the weighted cross entropy loss function and top-k ranking loss, respectively.

[0077] Specifically, the weighted cross entropy loss function is calculated as:

[0078]

[0079] Specifically, the top-k ranking loss is calculated as:

[0080]

[0081] Among them, parameter k represents the number of video clips selected in each video, Ω k is a subset of the video with a size of k. In the embodiment of the present invention, the parameter k is set to 7.

[0082] Specifically, in this embodiment, the anomaly score output by the self-guided channel attention branch is input into the factor analysis module to generate a pseudo label. The pseudo label is generated by fitting the following formula:

[0083]

[0084] Among them, μ k , ψ k ,Σ k is the mean, factor and diagonal covariance matrix of the kth Gaussian component, λ k is the coefficient of each Gaussian component, v k These parameters can be solved using the expectation-maximization algorithm.

Claims

1. A weakly supervised video anomaly detection method based on a self-guided encoder, characterized in that The following steps are involved: Step 1: Process the original video data. Extract 16 frames per second from the original unprocessed surveillance video and use them as a video segment. Each video will be processed into a different number of continuous video segments according to the video length. Step 2: Input the video data into the pre-trained backbone network I3D network for feature calculation, and input its features into the self-guided feature encoder for training. The encoder's attention branch and the self-guided channel attention branch will output the frame-level anomaly score of the video respectively; Step 3: The frame-level anomaly scores of the self-guided branch are fed into the t-distribution factor analysis module for fitting to obtain frame-level pseudo labels. Step 4: Use pseudo-label supervision to train the self-guided feature encoder again. The pseudo-label is continuously updated during the training process. The attention branch of the trained encoder outputs the predicted frame-level anomaly score. Step 5: Anomaly judgment is performed based on the set anomaly threshold. Frames with anomaly scores greater than the threshold are considered abnormal frames, otherwise they are considered normal frames. In step 2, the steps to build the self-guided channel attention branch are: The self-guided channel attention branch consists of a dilated convolution module, a SE module, and a classification head. composition; This branch only uses the feature map output by the penultimate layer of the backbone network As input, the encoding unit is shared with the attention branch , intermediate layer features It will be sent to the dilated convolution module to obtain spatiotemporal features of different scales. Its output features will be input to the SE module. The SE module extracts the channel attention features of the feature map and learns the dependencies between different anomalies to assist in the learning of the anomaly feature map. The dilated convolution module consists of two 3D convolution layers with different convolution kernels and dilation rates. and The output features are added together to obtain a multi-scale fusion feature map. : 。 2. A weakly supervised video anomaly detection method based on a self-guided encoder according to claim 1, characterized in that In step 2, the steps to build the attention branch are: The attention branch uses the feature map output by the penultimate layer of the backbone network I3D And the feature map of the penultimate layer output As input features, feature maps Encoding by coding unit Get the intermediate layer features , the feature map is then passed through the encoding unit Generate attention feature map : Then the attention mechanism is used to transform the feature map and attention feature map Perform element-wise multiplication and add Get the weighted feature map : Finally, the feature map Perform global average pooling operation and input to the anomaly classification head , classification head Output the anomaly score calculated by the attention branch.

3. The weakly supervised video anomaly detection method based on a self-guided encoder according to claim 1 is characterized in that In step 1, the loss function of the self-guided encoder is: The loss function of the self-guided encoder is: in and are the loss functions of the attention mechanism branch and the self-guided channel attention branch respectively. Specifically, and The specific form is: in, , is a hyperparameter, and are the anomaly scores predicted by the attention branch and the self-guided channel branch respectively. , is a hyperparameter, and are the anomaly scores predicted by the attention branch and the self-guided channel branch, respectively. and They are weighted cross entropy loss function and top-k ranking loss respectively; Specifically, the weighted cross entropy loss function is calculated as: Specifically, the top-k ranking loss is calculated as follows: Among them, the parameters Indicates the number of video clips selected in each video, The video size is A subset of .

4. The weakly supervised video anomaly detection method based on a self-guided encoder according to claim 3 is characterized in that In step 3, the t-distribution factor analysis module generates pseudo labels using the following formula: in, , , is the mean, factor and diagonal covariance matrix of the kth Gaussian component, k is a variable with a value range of 1-7, is the coefficient of each Gaussian component, These parameters can be solved using the expectation-maximization algorithm.

5. A device for weakly supervised video anomaly detection method based on the self-guided encoder according to claim 1, characterized in that: include: A video data processing module is used to extract frames from unprocessed video data and process the video data into continuous video segments; An attention feature module that enables the network to automatically focus on the most abnormal parts of the network while learning specific abnormal representations; The self-guided channel attention module uses dilated convolutions with different dilation rates and SE modules to obtain the dependencies between different anomalies to assist in learning anomaly feature maps. The t-distributed factor analysis module learns the anomaly scores output by the self-guided feature module and uses the learned mean, factors, and diagonal covariance matrix parameters to generate reliable pseudo-label supervision for training the self-guided encoder.

Images