A traffic abnormal event detection method based on multiple example learning

Abstract

The application provides a traffic anomaly event detection method based on multiple example learning, constructs a 3D convolution network based on MFNet as a feature extraction network and a scoring network composed of a full connection layer and a dropout layer; acquires a real-time video stream, decodes the real-time video stream to obtain a video frame image; extracts a feature once for the video frame image in groups of 16 frames, then combines all feature vectors of a video into a 32*768 feature vector and inputs the feature vector into the feature extraction network for feature extraction; scores the video clip by using the scoring network to obtain a scoring result of the video stream frame sequence; draws an anomaly scoring curve according to the scoring result of the video stream frame sequence, determines a position interval of anomaly scoring rising and falling, and outputs a time period corresponding to an abnormal video clip, thereby realizing extraction and classification of video image fine-grained features while reducing the calculation amount of a space-time network, and improving the anomaly event detection accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of video abnormal events, and in particular to a method for detecting abnormal traffic events based on multi-instance learning. Background Art

[0002] With the advancement of computer science and technology, the use of technologies such as image processing, computer vision, and deep learning can overcome the limitations of traditional video surveillance systems, enabling intelligent video analysis, proactive detection of abnormal events, and real-time early warning. This has significant value in intelligent video monitoring and analysis for urban transportation. In urban traffic scenarios, illegal parking, slow driving, and driving against traffic can easily cause traffic accidents and lead to road congestion. Therefore, the ability to promptly detect abnormal events and take swift and effective measures to address and resolve them is crucial to ensuring urban traffic safety.

[0003] Current traffic anomaly detection technology is still in its infancy. Most anomaly detection is done through manual intervention, which is not only labor-intensive but also has poor real-time performance. Video perception-based anomaly detection methods are mainly divided into three categories: semi-supervised video anomaly detection, unsupervised video anomaly detection, and weakly supervised video anomaly detection. Semi-supervised methods often require a certain amount of training data to learn the types of anomaly events. However, most anomaly events are scarce and difficult to collect, so this method has applicability issues. Although unsupervised methods do not require sample data, they cannot distinguish the feature expressions of abnormal video sequences, resulting in low accuracy in actual detection. Weakly supervised methods aim to overcome the problem of semi-supervised video anomaly detection ignoring prior information about known anomalies. They are closer to the setting of anomaly detection in real life, but there are currently few available algorithms, leaving much room for research and exploration. Summary of the Invention

[0004] The present invention provides a method for detecting abnormal traffic events based on multi-instance learning. The method adopts a multi-fiber network structure based on image semantic segmentation, splits a complex network into a superposition of lightweight sub-networks, and introduces a multiplexer module by utilizing the information flow between fibers. This method realizes the extraction and classification of fine-grained features of video images while reducing the computational complexity of the spatiotemporal network. A multi-instance learning method is designed in the scoring stage of abnormal event detection to iteratively optimize the classification results.

[0005] The present invention provides a method for detecting abnormal traffic events based on multi-instance learning, which is characterized by comprising:

[0006] Construct a feature extraction network and a scoring network, wherein the feature extraction network is a 3D convolutional network based on MFNet, and the scoring network is composed of a fully connected layer and a dropout layer;

[0007] Obtaining a real-time video stream, and decoding the real-time video stream to obtain a video frame image;

[0008] Extract features from the video frame images in groups of 16 frames, then combine all feature vectors of a video into a 32×768 feature vector and input it into the feature extraction network for feature extraction;

[0009] Use the scoring network to score the video clips and obtain the scoring results of the video stream frame sequence;

[0010] Draw an anomaly score curve based on the scoring results of the video stream frame sequence, determine the position intervals where the anomaly score rises and falls, and output the time period corresponding to the abnormal video clip.

[0011] In an optional embodiment, the feature extraction network adopts a ResNet-50 network and replaces the residual block inside the network with a multi-fiber module unit. The multi-fiber module unit is designed as follows:

[0012] The complex residual parameter unit is sliced ​​and converted into N separable parallel channels. The network width of the unit module input remains unchanged, and the number of connections is reduced to 1 / N of the original. The formula is as follows:

[0013] Connections=N*(M in / N*M mid / N+M mid / N*M out / N)

[0014] =(M in *M mid +M mid *M out ) / N

[0015] Among them, M in Indicates the number of input channels, M id Indicates the number of channels in the middle layer of the residual unit, M out Indicates the number of output channels;

[0016] Before performing convolution operations on each channel, a multiplexer module is uniformly connected to the network. The multiplexer module amplifies and redirects the multi-channel feature information. The multiplexer module first performs channel dimensionality reduction using a 1*1 convolution kernel, and then performs a 1*1 convolution kernel dimensionality increase operation to restore the number of input channels.

[0017] In an optional embodiment, combining all feature vectors of a video into a 32×768 feature vector includes:

[0018] If the video has less than 32 eigenvectors, adjacent eigenvectors are used for padding;

[0019] If the video has more than 32 feature vectors, 32 feature vectors are taken out by uniformly spaced sampling.

[0020] In an optional embodiment, the scoring network adopts a 3-layer fully connected neural network, and a dropout layer is added inside the network. The training process is as follows:

[0021] Define positive and negative packages. Divide each training video into 32 equal segments. If all 32 segments show normal events, the video constitutes a negative package. If one or more abnormal events are present in the 32 segments, the video is defined as a positive package. For negative packages, no additional temporal and spatial annotation information is required to determine the specific location of the abnormal event.

[0022] The scoring network is used to input the video feature segments of the positive and negative sample packages. The scoring network scores each feature segment and selects the feature segment with the highest score from the positive sample package as a potential abnormal sample. Similarly, the feature segment with the highest score from the negative sample package is selected as a non-abnormal sample. The scores of these two samples are used to train the model parameters.

[0023] Hinge-loss is used as the loss function in the scoring network to maximize the score gap between positive and negative samples. The formula is as follows:

[0024]

[0025] Here, f() represents the scoring output of the network. Two constraints are added to the loss function. The first is a smoothness constraint, which calculates the square of the difference between two adjacent feature fragments and ensures that the difference is small enough to achieve a smooth transition of feature values. The second condition is a sparse constraint, which is added to constrain the overall score separately. The formula is as follows:

[0026]

[0027] Finally, add the regularization term to get the final objective function expression, the formula is as follows:

[0028] L(W)=l(β a ,β n )+λ3||W|| F .

[0029] The application provides a traffic abnormal event detection method based on multiple example learning.

[0030] 1. Define the video slice rule and design the MFNet network, introduce the fibers multiplexer structure to realize information transmission between different fibers, reduce the space-time network calculation amount and complete the video feature extraction.

[0031] 2. Use the weakly supervised multiple example learning method in the scoring network to maximize the score difference between positive examples and negative examples, and improve the abnormal event detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0033] Figure 1 It is a schematic diagram for the process of converting from the ResNet network to the multi-fiber network structure.

[0034] Figure 2 It is a schematic diagram for image frame feature extraction based on the MFNet network.

[0035] Figure 3 It is a schematic diagram for positive example bag and negative example bag division.

[0036] Figure 4 It is a flowchart of the scoring network abnormal event detection. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0038] For feature learning of video images, the most common method is to use 3D convolutional neural networks for feature extraction and classification. Unlike 2D convolutional neural networks, 3D convolutional neural networks use 3D convolution kernels that span both time and space dimensions to implement convolution operations in the network. Although 3D convolutional neural networks have high accuracy in image feature extraction, they also bring huge convolution operation overhead. The input and output tensor dimensions of 3D networks also increase exponentially compared to 2D networks. In order to minimize the computational complexity of the spatiotemporal network while maintaining classification accuracy and ensure that the inference speed is close to that of the 2D version of the corresponding network, a Multi-Fiber network was proposed. It uses a sparse connection structure to split the complex network model into an integration of multiple lightweight networks, and uses a multiplexer module to achieve information fusion between different fibers.

[0039] Due to the complex and diverse characteristics of abnormal events, it is difficult to use only data from normal events for anomaly detection. Therefore, this application uses two types of data to train the model. In order to avoid a large amount of labeling work for training video data, a multi-instance learning network structure is used to score the samples. This method only requires a small amount of data labeling work to ensure the accuracy of event classification and the accuracy of scoring.

[0040] The present disclosure provides a method for detecting abnormal traffic events based on multi-instance learning, including:

[0041] Constructing a feature extraction network and a scoring network; wherein the feature extraction network is a 3D convolutional network based on MFNet, and the scoring network is composed of a fully connected layer and a dropout layer;

[0042] The feature extraction network uses a ResNet-50 network and replaces the residual blocks within the network with multi-fiber modules. This module reduces the number of connected layers within the network and effectively improves model efficiency. It also exhibits good compatibility and adaptability with both 2D and 3D convolutional neural networks. We first embed the multi-fiber module into a 2D convolutional neural network to demonstrate its effectiveness and evaluate its accuracy in image recognition tasks. Next, we apply this module to a 3D convolutional neural network and discuss its specific design.

[0043] The design of the multi-fiber module unit is based on the improvement and optimization of the highly modular residual unit, and is easy to train and deploy the model. The traditional residual unit uses two convolutional layers to learn features. This connection method is clear and easy to understand, but the computational overhead is very high. Figure 1 -(a), the number of convolutional connections of the structural unit is explained by the formula, let M in Indicates the number of input channels, M mid Indicates the number of channels in the middle layer of the residual unit, M outrepresents the number of output channels, so the total number of connections between the two layers can be calculated by formula (1):

[0044] Connections=M in *M mid +M mid *M out Formula (1)

[0045] The above formula is a simplified version of the formula, ignoring the computational overhead of constants such as the input feature dimension and the convolution kernel dimension, and only considering the variable computational overhead caused by the change of the connection relationship. Formula (1) shows that the number of connections is quadratically related to the network width, so increasing the network width of the structural unit by a factor of k will result in k 2 In order to reduce the number of network connections and the overall computational overhead, the complex residual parameter unit is sliced ​​and converted into N separable parallel channels, which we call module fiberization. After fiberization, each channel in the residual module is independent, such as Figure 1 -(c). Through this method, the network width of the unit module input remains unchanged, and the number of connections is reduced to 1 / N of the original, as shown in formula (2):

[0046] Connections=N*(M in / N*M mid / N+M mid / N*M out / N)

[0047] =(M in *M mid +M mid *M out ) / N formula (2)

[0048] At the same time, in order to realize the fusion and sharing of feature information between different channels, a multiplexer module is uniformly connected before the convolution operation on each channel, which plays the role of routing distribution. The multiplexer module amplifies and redirects the multi-channel feature information. The multiplexer module first performs channel dimensionality reduction through a 1*1 convolution kernel. The channel dimensionality reduction operation can effectively realize the information interaction between different channels and make up for the information loss caused by slicing. Then, a 1*1 convolution kernel dimensionality increase operation is performed to restore the number of input channels.

[0049] Obtaining a real-time video stream, and decoding the real-time video stream to obtain a video frame image;

[0050] The video frame images are grouped into 16 frames to extract features once, and then all feature vectors of a video are combined into a 32×768 feature vector and input into the feature extraction network for feature extraction. If the video has less than 32 feature vectors, adjacent feature vectors are used for filling; if the video has more than 32 feature vectors, 32 feature vectors are taken out by uniformly spaced sampling. The overall process is as follows Figure 2 shown.

[0051] Use the scoring network to score the video clips and obtain the scoring results of the video stream frame sequence;

[0052] In the scoring network, a weak supervision method based on multi-instance learning is used to classify and score abnormal events. First, positive and negative packages are defined, such as Figure 3 As shown in the figure, each training video is divided into 32 video clips of equal number. If all 32 video clips are normal events, the video constitutes a negative example package. If there are one or more abnormal events in the 32 video clips, the video is defined as a positive example package. For the negative example package, no additional time and space annotation information is required to determine the specific location of the abnormal event.

[0053] Next, we use a 3-layer fully connected neural network to build a scoring network, such as Figure 4 As shown in the figure, each video clip is scored. To avoid overfitting of the network structure, a dropout layer is added to the network to randomly stop some feature detectors to improve the generalization ability of the scoring network. The network model inputs positive and negative video feature clips. The scoring network scores each feature clip and selects the feature clip with the highest score from the positive sample as a potential abnormal sample. Similarly, the feature clip with the highest score from the negative sample is selected as a non-abnormal sample. The scores of these two samples are used to train the model parameters.

[0054] Hinge-loss is used as the loss function in the scoring network to maximize the score gap between positive and negative samples, as shown in formula (3):

[0055]

[0056] Among them, f() represents the scoring output of the network. It can be seen that when the abnormal event is scored as 1 and the normal event is scored as 0, the loss function is 0, reaching the minimum value. Otherwise, the loss value is greater than 0, so the loss function can maximize the distance space between positive and negative samples. At the same time, in order to improve the accuracy of abnormal event detection, two constraints are added to the loss function. The first is the smoothness constraint. Since the feature segments in each positive or negative sample are from the same video, the scores between the feature segments should change smoothly. The square of the difference between two adjacent feature segments is calculated and ensured to be small enough to achieve a smooth transition of the feature value. The second condition is the sparse constraint. Since there are often only a few abnormal samples in the positive sample, and most of them are normal samples, the smoothness constraint may cause all samples in the positive sample to tend to get higher scores. The loss is added to constrain its overall score separately, as shown in formula (4):

[0057]

[0058] Finally, the regularization term is added to obtain the final objective function expression, as shown in formula (5):

[0059] L(W)=l(β a ,β n )+λ3||W|| F Formula (5)

[0060] Draw an anomaly score curve based on the scoring results of the video stream frame sequence, determine the position intervals where the anomaly score rises and falls, and output the time period corresponding to the abnormal video clip.

Claims

1. A traffic anomaly event detection method based on multi-instance learning, characterized in that: include: Construct a feature extraction network and a scoring network, wherein the feature extraction network is a 3D convolutional network based on MFNet, and the scoring network is composed of a fully connected layer and a dropout layer; Obtaining a real-time video stream, and decoding the real-time video stream to obtain a video frame image; Extract features from the video frame images in groups of 16 frames, then combine all feature vectors of a video into a 32×768 feature vector and input it into the feature extraction network for feature extraction; Use the scoring network to score the video clips and obtain the scoring results of the video stream frame sequence; Draw an anomaly score curve based on the scoring results of the video stream frame sequence, determine the position intervals where the anomaly score rises and falls, and output the time period corresponding to the abnormal video segment; The scoring network uses a 3-layer fully connected neural network and adds a dropout layer inside the network. The training process is as follows: Define positive and negative packages. Divide each training video into 32 equal segments. If all 32 segments show normal events, the video constitutes a negative package. If one or more abnormal events are present in the 32 segments, the video is defined as a positive package. For negative packages, no additional temporal and spatial annotation information is required to determine the specific location of the abnormal event. The scoring network is used to input the video feature segments of the positive and negative sample packages. The scoring network scores each feature segment and selects the feature segment with the highest score from the positive sample package as a potential abnormal sample. Similarly, the feature segment with the highest score from the negative sample package is selected as a non-abnormal sample. The scores of these two samples are used to train the model parameters. Hinge-loss is used as the loss function in the scoring network to maximize the score gap between positive and negative samples. The formula is as follows: Here, f() represents the scoring output of the network. Two constraints are added to the loss function. The first is a smoothness constraint, which calculates the square of the difference between two adjacent feature fragments and ensures that the difference is small enough to achieve a smooth transition of feature values. The second condition is a sparse constraint, which is added to constrain the overall score separately. The formula is as follows: Finally, add the regularization term to get the final objective function expression, the formula is as follows: 。 2. The method for detecting abnormal traffic events based on multi-instance learning according to claim 1, characterized in that: The feature extraction network adopts the ResNet-50 network and replaces the residual block inside the network with a multi-fiber module unit. The multi-fiber module unit is designed as follows: The complex residual parameter unit is sliced ​​and converted into N separable parallel channels. The network width of the unit module input remains unchanged, and the number of connections is reduced to 1 / N of the original. The formula is as follows: in, Indicates the number of input channels, represents the number of channels in the middle layer of the residual unit, Indicates the number of output channels; Before performing convolution operations on each channel, a multiplexer module is uniformly connected to the network. The multiplexer module amplifies and redirects the multi-channel feature information. The multiplexer module first performs channel dimensionality reduction using a 1*1 convolution kernel, and then performs a 1*1 convolution kernel dimensionality increase operation to restore the number of input channels.

3. The method for detecting abnormal traffic events based on multi-instance learning according to claim 1, characterized in that: The method of combining all feature vectors of a video into a 32×768 feature vector includes: If the video has less than 32 eigenvectors, adjacent eigenvectors are used for padding; If the video has more than 32 feature vectors, 32 feature vectors are taken out by uniformly spaced sampling.

Images