Violence video detection model based on three-dimensional convolutional neural network

Abstract

The invention discloses a violent video detection model based on a three-dimensional convolutional neural network, and the model comprises a data obtaining and processing module which is used for obtaining and processing video data, the processed video data is a five-dimensional array, and the dimensions comprise the batch, the number of video frames, the height, the width and the number of channels; the 3D convolution layer is used for applying a three-dimensional kernel on a cube formed by adjacent frames of the stacked video data to extract features of the video data from the adjacent frames, and the features comprise spatio-temporal features; the 3D maximum pooling layer is used for reducing space and time dimensions of feature mapping; the batch normalization layer is used for accelerating model training and improving model stability; the Flatten layer is used for converting the multi-dimensional feature mapping into a one-dimensional vector in the form of a square matrix; and the Dense layer is used for classifying the one-dimensional vectors output by the Flatten module to obtain a final classification category, the classification category comprises violence and non-violence, and through detection, the total precision calculated by the model can reach 80.58%.

Description

Technical Field

[0001] The present invention relates to the technical field of violent video detection, and in particular to a violent video detection model based on a three-dimensional convolutional neural network. Background Art

[0002] In recent years, the increase in public violence incidents around the world has triggered widespread concern about the safety supervision of public places. Although the monitoring of public places can be improved by increasing the number of surveillance cameras, manual monitoring is still required to deal with abnormal situations in a timely manner. However, the manual decision-making process is usually slow and biased, and it is impossible to monitor multiple cameras at the same time. This limitation inhibits the ability to monitor simultaneously in a multi-camera environment.

[0003] In situations with real-time requirements, such as violence prevention, the limitations of using manual supervision are demonstrated, so more reliable and effective alternatives to monitoring systems are needed. To this end, researchers have turned to computer systems for solutions, with the most attention paid to machine learning technology. Machine learning extracts information from data by training algorithms, enabling the system to predict and judge similar problems. Applying machine learning models to different fields and industries significantly reduces dependence on manual labor and improves efficiency, performance, and effectiveness.

[0004] Over the years, the field of machine learning has made great progress, especially with the advent of deep learning, which has been particularly effective in pattern recognition tasks. Violence detection can be described as a pattern recognition problem, in which machines can be taught to recognize patterns in videos to detect the presence of violent behavior. This allows machines to quickly and impartially identify the presence of violent behavior and immediately issue an alarm so that necessary measures can be taken in time to prevent the continuation of violent incidents. In the long run, the implementation of automated violence detection systems will significantly reduce costs. These views support the consensus that machine learning models are more efficient than humans in many tasks, especially in detecting violence in public places.

[0005] The existing methods for detecting violent videos can be divided into two types. One is the manual feature method. Early studies used visual or audio features to detect flames, blood, skin, and blood, such as using Gaussian mixture models and hidden Markov models to detect gunshots and explosions. Later, the bag-of-words method used for images was applied to video recognition and was frequently used in video classification tasks. Based on the descriptor derived from the optical flow histogram of the automatic access control point, a method for verifying the identity of people and detecting abnormal human behavior was developed. There is also a combination of the BoW framework and STIP and MoSIF descriptors for violence detection, and the Hockey Fights Dataset was created, which is a commonly used dataset in violence detection tasks. The change of the flow vector size over time is considered, and the violent flow descriptor is used to represent these statistics, and then the violent flow descriptor is classified using a linear support vector machine. The above are some methods for using manual features for detecting violent behavior in videos. Although some progress has been made, there are still many shortcomings. The following points out some of the main defects: 1. High computational complexity: Some early manual feature methods usually require a large amount of feature information to be extracted for each video frame. These methods have large computational complexity and slow processing speed, making it difficult to meet real-time requirements. For example: Bag-of-Words and Histograms of Optical Flow.

[0006] 2. Feature selection and design rely on experience: Most of these methods rely on professional experience to design and select features. For example, Violent Flows and Orientation Histogram of Optical Flow rely on carefully designed feature descriptors. This dependence limits the generalization ability of the model.

[0007] 3. Poor adaptability: Manual feature methods have poor adaptability to changing scenes and different environmental conditions. For example, when the background or lighting conditions of the dataset change significantly, the performance of these methods may drop significantly.

[0008] 4. Sensitivity to noise: Many manual feature methods are very sensitive to noise and interference in the video, which may lead to inaccurate feature extraction and thus affect the final classification results.

[0009] Another method is the currently popular deep learning method, in which convolutional neural networks are often used to extract features. Before the emergence of three-dimensional convolutional neural networks, because temporal features need to be processed in video recognition, the more common method is to use CNN models to extract spatial features and use long short-term memory networks for classification. For three-dimensional convolutional neural networks, the spatial and temporal features of video data can be directly extracted to classify videos. Deep learning methods have made great progress in the detection of violent videos, but there are still some problems, such as: 1. Large model size and number of parameters: When it comes to models with more complex network architectures, a large amount of computing resources are usually required due to the large number of model parameters. Training these models requires high-performance GPUs and large-scale datasets, which may result in long training costs and inference times. For example, when using pre-trained CNN models and ConvLSTM for feature extraction and time series modeling, a large amount of computing needs to be processed.

[0010] 2. Model overfitting problem: Although deep learning models are generally powerful in expression, they are also prone to overfitting, especially when the amount of data is insufficient or the data quality is not high. Even with the use of regularization methods, the performance of the model on the training set may be too optimistic, while the performance on the test set is not satisfactory. For example, when using 3DCNN combined with SVM, overfitting may occur.

[0011] 3. Lack of explainability: The "black box" nature of deep learning models makes it difficult to explain their decision-making process, especially in complex neural networks, where the internal operation of the model is often difficult to explain and understand. This is a significant flaw for tasks that require high explainability. Summary of the invention

[0012] The purpose of the present invention is to provide a violence video detection model based on a three-dimensional convolutional neural network to solve the problems raised in the above background technology.

[0013] To achieve the above object, the present invention provides the following technical solution: a violence video detection model based on a three-dimensional convolutional neural network, comprising: Data acquisition and processing module: used to acquire and process video data. The processed video data is a five-dimensional array, and the dimensions include batch, video frame number, height, width and number of channels; at least one 3D convolutional layer for applying a three-dimensional kernel on a cube consisting of adjacent frames of stacked video data to extract features of the video data from the adjacent frames, including spatiotemporal features; At least one 3D max pooling layer to reduce the spatial and temporal dimensions of the feature map; At least one batch normalization layer to speed up model training and improve model stability; At least one Flatten layer, which is used to convert the multi-dimensional feature map into a one-dimensional vector in the form of a square matrix; At least one Dense layer is used to classify the one-dimensional vector output by the Flatten module to obtain the final classification category, which includes violent and non-violent categories.

[0014] Preferably, it includes two 3D convolution layers, two 3D maximum pooling layers and two batch normalization layers. The video data is processed by a 3D convolution layer, a 3D maximum pooling layer and a batch normalization layer, and then circulated through a 3D convolution layer, a 3D maximum pooling layer and a batch normalization layer before being output to the Flatten layer.

[0015] Preferably, the 3D convolutional layer is configured with 8 filters, the kernel size is (3,7,7), effective padding is used so that there is no zero-filled boundary around the image, the stride is set to 1 by default, and the activation function uses the rectified linear unit ReLU.

[0016] Preferably, the 3D maximum pooling layer adopts a 3D Max-Pooling layer, which divides the 3D video data input into cubic pooling areas, calculates the maximum value of each area and performs nonlinear downsampling.

[0017] Preferably, the 3D Max-Pooling layer kernel size is (2,2,2) and padding is enabled.

[0018] Preferably, the process of calculating the value of the position of the feature map (x, y, z) by three-dimensional convolution is: ,in, is the hyperbolic tangent function, which is used to activate neurons. is the bias of the feature map at the current position, is the weight of the convolution kernel, are the sizes of the convolution kernel in height, width, and time dimensions, respectively.

[0019] Preferably, the data set used for model training is a large data set obtained by merging Hockey Fights, Movies, RWF-2000 and Real Life Violence Situations, the large data set includes 50% violent videos and 50% non-violent videos, and the large data set is divided into a training set, a test set and a validation set, corresponding to 60%, 20% and 20% respectively.

[0020] Preferably, the model uses accuracy as an evaluation indicator, where the accuracy is the ratio of the number of samples correctly predicted by the model to the total number of samples.

[0021] Preferably, the model is validated using accuracy, precision, recall and f1 score as evaluation indicators.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. This model can reduce model size and number of parameters: This model aims to improve computational efficiency by reducing the number of parameters, thereby achieving faster training speed and reducing memory requirements, which makes the model more suitable for running in resource-constrained environments, such as mobile devices and edge computing devices; 2. Enhance model interpretability: Reducing the number of parameters not only helps improve computational efficiency, but also reduces overfitting and enhances model interpretability. Fewer parameters help improve model transparency in practical applications, especially in tasks that require high interpretability, such as security monitoring and abnormal behavior detection. 3. Transfer learning and deployment: Models with fewer parameters are particularly suitable for transfer learning and deployment on edge devices. This design helps improve energy efficiency, enables the model to run efficiently on edge devices, and adapts to various practical application scenarios; 4. Model performance optimization: While maintaining a balance between model complexity and performance, it avoids the loss of model representation capability that may be caused by excessive parameter reduction. By reasonably reducing parameters, it ensures that the model can effectively perform brute force detection without sacrificing the model's representation capability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a module schematic diagram of a violent video detection model based on a three-dimensional convolutional neural network proposed in the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The present invention provides a violence video detection model based on a three-dimensional convolutional neural network. In order to design an effective violence detection model, violence must first be clearly defined. In the present invention, violence is defined as a voluntary physical behavior performed by one or more individuals with the intention of harming one or more people. Although the form and degree of violence may vary, the present invention focuses on the detection of violent behavior based on this definition.

[0026] In recent years, 2D CNN has performed well in the field of static image recognition. This success is because 2D CNN is specially designed for spatial information in image processing. Video itself has strong spatiotemporal information, and 2D CNN lacks attention to spatiotemporal information. This limitation causes it to lose motion information when processing videos, so it is not the best choice for processing video data. In contrast, 3D convolutional neural network combines the time dimension when calculating, which can effectively capture the spatiotemporal information in the video and make better use of the motion information of the difference between consecutive frames. Therefore, 3D CNN is more suitable for video recognition tasks.

[0027] like Figure 1 As shown in the figure, the input data of this violent video detection model based on 3D convolutional neural network is the pre-processed video data, which is a five-dimensional array with dimensions [batch_size, number_of_frames, height, width, channels], which are batch, number of video frames, height, width and number of channels respectively. The model structure is composed of 3D CNN, 3DMax-Pooling, Flatten layer, Dense layer and normalization layer, as shown in the figure. Figure 1 As shown in the figure, the data is input into the model after preprocessing, first passes through a 3D CNN layer, then enters the 3D Max-Pooling, and then the output is handed over to the normalization layer, thereby improving the stability and efficiency of model training. After that, this process is repeated once to obtain the output result, and the result is flattened to obtain the classification result. The design of the 3D Max-Pooling operation with two cycles has several advantages, including: 1. 3D Max-Pooling extracts features by pooling data in space, time or depth. In three-dimensional data, two cycles can extract richer features in different dimensions. 2. In 3D convolutional networks, data is usually highly dimensional. By performing pooling in multiple dimensions, especially the two-loop design, the spatial dimension of the data can be effectively reduced; 3. Through multiple pooling, the network's robustness to local noise and irrelevant changes is enhanced.

[0028] This model uses video as input data instead of static images. Video is a series of continuous image frames, which produces the effect of continuous motion. Although images only have spatial information, the motion features between adjacent frames in videos introduce a temporal dimension. Due to this additional temporal dimension, two-dimensional convolution is less efficient for feature extraction because a lot of temporal information is lost. To solve this problem, three-dimensional convolution is often used to extract features in video analysis tasks. Three-dimensional convolution operates in a similar way to two-dimensional convolution, except that the kernel slides in three dimensions instead of two. Three-dimensional convolution is implemented by applying a three-dimensional kernel on a cube composed of stacked adjacent frames. Therefore, motion information can be obtained from adjacent frames. By using three-dimensional convolution and three-dimensional pooling, the temporal information of the input video can be well preserved. The operations of two-dimensional convolution and three-dimensional convolution are given by the following formulas, respectively, , where K is the convolution kernel, A is the convolution matrix, M represents the number of rows, N represents the number of columns, and B is the result matrix. The time dimension T is added to the two-dimensional convolution operation.

[0029] The specific formula for the three-dimensional convolution feature calculation process is as follows: , Calculate the value of the feature map (x, y, z) position for the three-dimensional convolution. is the hyperbolic tangent function, which is used to activate neurons. is the bias of the feature map at the current position, is the weight of the convolution kernel, are the sizes of the convolution kernel in height, width, and time dimensions, respectively. With this construction, the feature maps in a convolutional layer are connected to multiple consecutive frames of the previous layer, thereby capturing motion information.

[0030] In this model, the 3D convolutional layer is configured with 8 filters, a kernel size of (3,7,7), and uses effective padding, which means that there is no border with zero padding added around the image. The stride is set to 1 by default, which represents the step size of each kernel movement, and the activation function applied is the rectified linear unit (ReLU). The purpose of the activation function in a neural network is to convert the weighted sum of a node into the activation of the node. Commonly used activation functions are ReLU, hyperbolic tangent, sigmoid, etc. ReLU is a piecewise function that retains the value of positive numbers and changes all negative numbers to zero.

[0031] In most CNN architectures, each convolutional layer is usually followed by a pooling layer. The pooling operation is similar to convolution, grouping pixels in the input image and filtering them into a subset. Pooling layers reduce the size of the convolutional feature map by reducing the connections between layers. There are multiple types of pooling, and the type used in this model is 3D Max-Pooling. The 3D Max-Pooling layer performs non-linear downsampling by dividing the 3D input into cubic pooling regions and then calculating the maximum value of each region. Although downsampling may be misunderstood as data loss, it is a key step to reduce overfitting and speed up model training by removing redundant data. In the proposed model, the 3D Max-Pooling layer kernel size is (2,2,2) and padding is effective.

[0032] Normalization is often used to standardize raw data, thereby reducing the range of the data. This process improves the learning rate and convergence speed of the model and prevents the model from diverging, thus facilitating training. Batch Normalization is a special type of normalization that differs from standard normalization in that it is applied between model layers along the training mini-batches. It can be added to the model as a layer. Batch Normalization allows the use of higher learning rates, is less careful about initialization, and in some cases, eliminates the need for Dropout. Considering the use of Batch Normalization in the proposed model, the incorporation of Dropout is considered unnecessary.

[0033] The flatten layer converts the pooled feature maps (initially in the form of square matrices) into a one-dimensional column vector. While the purpose of this layer is straightforward, it is an essential step in neural networks because neural networks accept inputs in the form of one-dimensional linear vectors. Once the data is flattened, it is suitable for input to the fully connected layers of the network.

[0034] The Dense layer consists of neurons, each of which receives input from all neurons in the previous layer, so it is also called a fully connected layer. The output of the convolutional layer is flattened in the previous flattening layer to facilitate input to the Dense layer. After extracting the features of the previous layer, the Dense layer performs classification based on the output of the convolutional layer. Considering that the proposed model has two output classes, fight and non-fight, the shape of the final Dense layer is set to 2.

[0035] The violence detection datasets used in the experiment are Hockey Fights, Movies, RWF-2000, and Real Lifeviolence Situations (RLVS). In the experiment, all four datasets are merged into a large dataset, consisting of a total of 5200 videos, of which 2600 are labeled as violent and 2600 are labeled as non-violent. The merged dataset is then divided into training, test, and validation sets, corresponding to 60%, 20%, and 20%, respectively. The percentage split corresponds to 3120 videos for training, 1040 for validation, and 1040 for the test set. Assume that the model has two output classes, half of each group is violent videos and the other half is non-violent videos.

[0036] Five experiments were conducted on the combined dataset and the performance of the proposed model was evaluated using metrics such as accuracy, precision, recall, and f1 score. These metrics were calculated based on the confusion matrix created from the test set of each experiment. The metrics are calculated using the following formula: , where TP, TN, FP, and FN are true positive, true negative, false positive, and false negative, respectively. Their corresponding elements in the confusion matrix are the upper left corner of TP, the upper right corner of FN, the lower left corner of FP, and the lower right corner of TN. The accuracy, loss, precision, recall, and f1 score values ​​of each experiment are shown in the following table: . The average value of each evaluation metric of the five experiments is calculated and considered to be the overall performance of the network. The overall accuracy, loss, precision, recall and f1 score values ​​are shown in the following table: . The model computed an overall accuracy of 80.58%, which is in line with expectations. It is worth noting that the network was trained using 3120 videos out of the 5200 videos in the combined dataset, many of which were of low resolution and some of which had out-of-focus camera motion. Furthermore, it should be acknowledged that accuracy was sacrificed to some extent in order to reduce the number of network parameters so that it can run effectively on mobile systems, which is the main goal of this work.

[0037] Real-time violence detection poses a considerable challenge. In order to seamlessly integrate the violence detection framework into real-life surveillance systems and ensure that it does not ignore anomalies or trigger false alarms, the model must be trained on a large number of videos. Training the model on a larger dataset introduces different situations and contexts, thereby improving accuracy. However, processing large amounts of volumetric data significantly increases the runtime, memory usage, and storage requirements of the network. Considering the available resources and data, as well as the computational efficiency of the proposed model, a satisfactory trade-off between accuracy and model size is achieved. The main goal of this model was to design a lightweight CNN with a fairly small size for violence detection, which was achieved.

[0038] It should be noted that the accuracy and performance of the model can be improved through hyperparameter tuning, weight initialization, and exploring effective combinations with other ML algorithms. These improvements are intended to be considered in future work and to expand the definition of violence beyond physical fights to ensure comprehensive coverage of all types of violence and threats to society.

[0039] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A violent video detection model based on a three-dimensional convolutional neural network, characterized by: include: Data acquisition and processing module: used to acquire and process video data. The processed video data is a five-dimensional array, and the dimensions include batch, video frame number, height, width and number of channels; at least one 3D convolutional layer for applying a three-dimensional kernel on a cube consisting of adjacent frames of stacked video data to extract features of the video data from the adjacent frames, including spatiotemporal features; At least one 3D max pooling layer to reduce the spatial and temporal dimensions of the feature map; At least one batch normalization layer to speed up model training and improve model stability; At least one Flatten layer, which is used to convert the multi-dimensional feature map into a one-dimensional vector in the form of a square matrix; At least one Dense layer is used to classify the one-dimensional vector output by the Flatten module to obtain the final classification category, which includes violent and non-violent categories.

2. The violent video detection model based on a three-dimensional convolutional neural network according to claim 1, characterized in that: It includes two 3D convolution layers, two 3D maximum pooling layers and two batch normalization layers. The video data is processed by a 3D convolution layer, a 3D maximum pooling layer and a batch normalization layer, and then circulated through a 3D convolution layer, a 3D maximum pooling layer and a batch normalization layer before being output to the Flatten layer.

3. A violent video detection model based on a three-dimensional convolutional neural network according to claim 1 or 2, characterized in that: The 3D convolutional layer is configured with 8 filters, the kernel size is (3,7,7), effective padding is used so that there is no zero-padding boundary around the image, the stride is set to 1 by default, and the activation function uses the rectified linear unit ReLU.

4. The violent video detection model based on a three-dimensional convolutional neural network according to claim 1, characterized in that: The 3D Max-Pooling layer is used. The 3D Max-Pooling layer divides the 3D video data input into cubic pooling regions and calculates the maximum value of each region for nonlinear downsampling.

5. The violent video detection model based on a three-dimensional convolutional neural network according to claim 4, characterized in that: The 3D Max-Pooling layer kernel size is (2,2,2) and padding is enabled.

6. The violent video detection model based on a three-dimensional convolutional neural network according to claim 5, characterized in that: The process of calculating the value of the position of the feature map (x, y, z) by three-dimensional convolution is: ,in, is the hyperbolic tangent function, which is used to activate neurons. is the bias of the feature map at the current position, is the weight of the convolution kernel, are the sizes of the convolution kernel in height, width, and time dimensions, respectively.

7. The violent video detection model based on a three-dimensional convolutional neural network according to claim 1, characterized in that: The dataset used for model training is a large dataset obtained by merging Hockey Fights, Movies, RWF-2000, and Real Life violence Situations. The large dataset includes 50% violent videos and 50% non-violent videos. The large dataset is divided into training set, test set, and validation set, corresponding to 60%, 20%, and 20% respectively.

8. The violent video detection model based on a three-dimensional convolutional neural network according to claim 7, characterized in that: The model was validated using accuracy, precision, recall and f1 score as evaluation indicators.

Images