Dual-flow gated violence detection method and system based on dilated 3D convolutional network and transformer

Abstract

The application is based on a dual-flow gate violence detection method and system of an expanded 3D convolutional network and a Transformer, and realizes classification through the following steps; S1: obtaining a video stream, segmenting the video stream into frames, and calculating the dense optical flow between the continuous frames of the video stream; S2: segmenting the video stream segment through a sliding window, and suppressing non-motion elements in the RGB mode by using the optical flow mode; S3: performing spatiotemporal modeling on the dense optical flow and the SRGB mode by using a 3D convolutional neural network, and performing feature fusion to create a feature map; S4: performing time modeling on the spatiotemporal feature sequence of the video stream segment by using a Transformer encoder, and generating a time feature map; S5: performing average pooling by using a 1D convolutional neural network, and performing classification by using a multilayer perceptron, and classifying the video as violence or non-violence. The application solves the deficiency of the violence detection model in processing spatiotemporal features, improves the accuracy and efficiency of violence detection, and especially has an application effect in a dynamic video monitoring scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent video surveillance technology, specifically to a dual-stream gating brute-force detection method and system based on dilated 3D convolutional networks and Transformers. Background Technology

[0002] Accurate violence detection is crucial for intelligent surveillance systems, enabling proactive security prevention, timely response to emergencies, and support for public safety management decisions. With increasing public safety demands and the growing volume of surveillance video data, real-time detection of violent behavior faces significant challenges, primarily stemming from the inherent spatiotemporal dependencies and dynamic nature of video. Traditional single-modal input methods often struggle to effectively capture key motion features in video.

[0003] In recent years, advancements in deep learning have driven the application of neural network architectures in modeling spatial and temporal dependencies. Two-dimensional convolutional neural networks (2DCNNs) and three-dimensional convolutional neural networks (3DCNNs) have been widely used in video content understanding. However, single-modal models struggle to capture spatial and temporal features in videos and, due to their large number of parameters and high training costs, are difficult to effectively model complex motion dependencies in videos. Existing brute-force detection models have limitations in handling the spatiotemporal dependencies of video data. Traditional methods often rely on fixed video frames, making it difficult to adapt to dynamic changes in videos. Although some advanced models, such as Transformer-based models, perform well in modeling long-range dependencies, their high computational complexity limits their application in real-time detection. Therefore, a new model is needed to improve the accuracy of brute-force detection while reducing computational complexity.

[0004] To address the spatiotemporal dependency problem, a two-stream network architecture was introduced to effectively model the relational structure in videos. FGITNet combines a dilated 3D convolutional network (I3D) and a Transformer encoder architecture to handle spatiotemporal data. Furthermore, the model's adaptability is further enhanced by dynamically updating the spatial relationships between frames. However, these methods typically rely on predefined or static video frames, limiting their adaptability to constantly changing patterns of violent behavior.

[0005] Transformer-based models have garnered significant attention due to their ability to effectively capture long-term dependencies using self-attention mechanisms. FGITNet employs a multi-head attention mechanism to model spatiotemporal correlations. While these models perform well on complex video data, their high computational complexity makes them less suitable for real-time detection applications. Although the FGITNet method aims to reduce this complexity, achieving a balance between efficiency and accuracy remains challenging.

[0006] State-space models have been re-examined in sequence modeling because they can efficiently capture long-term dependencies. This application introduces a method that combines state-space models with neural networks through FGITNet, which can effectively model complex temporal dynamics without the computational overhead of traditional recursive units. Summary of the Invention

[0007] This invention relates to a two-stream gated brute-force detection method based on dilated 3D convolutional networks and Transformers, comprising the following steps:

[0008] S1: Acquire the video stream, divide the video stream into frames, and calculate the dense optical flow between consecutive frames of the video stream;

[0009] S2: Video stream segments are divided by a sliding window, and non-moving elements in the RGB mode are suppressed using optical flow modes;

[0010] S3: Spatiotemporal modeling of dense optical flow and sRGB modes is performed using a 3D convolutional neural network, and feature fusion is carried out to create a feature map;

[0011] S4: The temporal feature sequence of the video stream segment is modeled using a Transformer encoder to generate a temporal feature map;

[0012] S5: Average pooling is performed using a 1D convolutional neural network, and classification is performed using a multilayer perceptron to classify the video as violent or non-violent.

[0013] Preferably, in step S1, consecutive frames of the video stream are normalized using Z-score. After processing, the dense optical flow between consecutive frames of the video stream is calculated using the Farneback method, with the formula: DOF t =Farneback(RGB) t RGB t+1 ), where RGB t For continuous frames at time t, RGB t+1 For consecutive frames at time t+1, DOF t Let t be the dense optical flow; by iterating the above formula over each frame of a continuous video stream, n-1 dense optical flow sequences are obtained, where n represents the length of the video.

[0014] Preferably, in step S2, the dense optical flow is divided into m segments using a sliding window method, and the suppression effect is enhanced by accumulating the optical flow vector amplitude of all frames, as shown in the formula: Where |DOF t | represents the DOF frame at time t ∈ [i, j], where i and j are indices indicating the start and end frames within the k-th DOF segment. A kTo accumulate all frames; the optical flow vector amplitude of all frames is obtained by normalizing the accumulated optical flow amplitude using a minimum-maximum normalized (MMN) method, scaling it to a value between 0 and 1, and using the normalized accumulated value as a scaling factor to perform element-wise multiplication adjustment of the RGB segments, suppressing non-moving elements. The formula is as follows: in For the i-th frame of the k-th RGB segment, For the suppressed RGB frame, Element-wise multiplication across all spatial dimensions and color channels yields the suppressed sRGB fragment.

[0015] Preferably, in step S3, spatiotemporal features are extracted from the SRGB mode and dense optical flow mode using a pre-trained I3D model to obtain segment spatiotemporal features. The segment spatiotemporal features are then fused by element-wise multiplication to generate a comprehensive segment spatiotemporal feature map. The segment spatiotemporal feature maps are connected along the time dimension to form two spatiotemporal feature sequences of shape l×1024, where l is the sum of all segment output logical values ​​determined by the segment size and number of segments.

[0016] Preferably, the Transformer encoder in S4 receives the output X of the segment spatiotemporal modeling; the output X is mapped to a query Q, a key K, and a value V, respectively, with the following mapping relationship: Q = XW Q K = XW K V = XW V Among them, W Q W K W V These are parameter matrices used to map the input X to a specific space of queries, keys, and values; the Transformer encoder is used to perform temporal modeling on the spatiotemporal feature sequence of each video segment, generating a temporal feature map.

[0017] Preferably, the selection of video segments depicting violent activities is performed using a multi-head self-attention mechanism via a Transformer encoder; the multi-head self-attention mechanism transforms the input X using the formula: MultiHead(Q, K, V) = {Concat(head)1, ..., head} h W o Where h represents the number of attention heads, W o The parameter matrix of the multi-head self-attention mechanism, where It is a linear transformation of the attention heads of the query Q, key K, and value V; the multi-head self-attention mechanism transforms the input query Q, key K, and value V using the following formula: Where d k is the dimension of each key, and softmax is the activation function.

[0018] Preferably, the multi-head self-attention mechanism of the Transformer encoder normalizes the output X and passes it through a residual connection, with the formula: Z = LayerNorm(X + MultiHead(Q, K, V)). Each position in Z is obtained and processed independently by a feedforward network FFN, with the formula: FFN(z) = max(0, zW1 + b1)W2 + b2, where W1 and W2 are the weight parameters of the feedforward network, and b1 and b2 are the bias coefficients. The output of the feedforward network FFN is normalized and added back to Z through another residual connection, with the formula: Y = LayerNorm(Z + FFN(Z)), where Y is the output value.

[0019] Preferably, in step S5, average pooling is performed using a 1D convolutional neural network. The input data is fed into the 1D convolutional neural network architecture in the time dimension, and sliding window operations are performed on the time series. For each data point within the window, the average value is calculated to realize the downsampling process of the time features. The dimension of the input data is compressed to output a tensor.

[0020] Preferably, after the 1D convolutional neural network average pooling is completed, the data enters a multilayer perceptron for classification. The output tensor obtained after average pooling is flattened into a vector, which is then fed into the 1D convolutional neural network as the input to the multilayer perceptron. The multilayer perceptron consists of multiple fully connected layers. The input vector and the neurons of the multilayer perceptron undergo a linear transformation through a weight matrix, followed by a nonlinear transformation through an activation function. The transformed result is then passed to the next layer, repeating the linear and nonlinear transformation processes. After multilayer processing, the data reaches the output layer, which outputs a classification of non-violent or violent behavior in the video. The video is classified as either violent or non-violent based on the activation values ​​of two neurons in the output layer.

[0021] A dual-stream gated brute-force detection system based on dilated 3D convolutional networks and Transformers includes a dual-stream gated network FGITNet model; the dual-stream gated network FGITNet model includes a data preprocessing module, a background suppression module, a fragment spatiotemporal modeling module, a fragment temporal modeling module, and a classification module;

[0022] The data preprocessing module standardizes the input surveillance video, adjusts the resolution pixels of the image, normalizes the image data through Z-score to make it conform to the standard format, and extracts a fixed number of frames through uniform sampling to generate standardized video frame data.

[0023] The background suppression module suppresses non-moving elements in the video, extracts segments from RGB frames and dense optical flow frames using a sliding window method, calculates the amplitude of motion information in dense optical flow segments, adjusts the optical flow amplitude to a value between 0 and 1 through minimum-maximum normalization, and adjusts RGB segments by element-wise multiplication.

[0024] The segment spatiotemporal modeling module uses a pre-trained I3D model to extract spatiotemporal features from the suppressed RGB mode and dense optical flow mode. It independently models spatiotemporal features on each segment of length clip_size, and fuses the two types of modal features by element-wise multiplication to form a comprehensive spatiotemporal feature map of the segment.

[0025] The segment time modeling module adopts a Transformer encoder structure and introduces a multi-head self-attention mechanism. It takes the spatiotemporal feature sequence output by the segment spatiotemporal modeling module as input, and through linear transformation, multi-head self-attention calculation, residual connection and feedforward network processing operations, the FGITNet model selectively observes segments of violent activity and generates a time feature map.

[0026] The classification module downsamples the temporal features using 1D average pooling, compresses the dimension of the input data, outputs a tensor, then flattens it into a vector, processes it through a multilayer perceptron containing multiple fully connected layers, and outputs the classification result of each video segment as either non-violent or violent behavior.

[0027] The video frame data processed by the data preprocessing module enters the background suppression module. The data processed by the background suppression module is then sent to the segment spatiotemporal modeling module. The spatiotemporal feature sequence generated by the segment spatiotemporal modeling module is passed as input to the segment temporal modeling module. The temporal feature map output by the segment temporal modeling module is then sent to the classification module. After processing by the classification module, the video classification result is output.

[0028] Compared with the prior art, the technical solution of this application has the following technical effects:

[0029] This invention innovatively solves the shortcomings of traditional brute-force detection models in handling spatiotemporal features by combining dilated 3D convolutional networks with Transformer encoder architecture, significantly improving the accuracy and efficiency of brute-force detection, especially in dynamic video surveillance scenarios.

[0030] The FGITNet model proposed in this invention has the ability to process video structural data and capture complex relationships in the spatial and temporal dimensions. The FGITNet model not only uses optical flow as a unique feature of each frame, but also uses RGB data as multiple features. It captures video relationships through the spatiotemporal FGITNet layer, and then the classification layer processes them to make the final prediction. By combining the dilated 3D convolutional network and the Transformer encoder architecture, it innovatively solves the shortcomings of traditional brute-force detection models in handling spatiotemporal dependencies.

[0031] This invention improves the model's feature representation ability through multi-dimensional feature learning in the data preprocessing module, enabling it to deeply capture the rich features of the input video and enhance the model's learning ability. Through the proposed background suppression method, RGB frames are effectively filtered by optical flow mode to suppress non-moving elements and highlight moving objects in the video, thereby improving the model's ability to capture motion features while maintaining low computational overhead.

[0032] The spatiotemporal modeling module of this invention optimizes video sequence analysis capabilities. Through dilated 3D convolutional networks and Transformer encoder mechanisms, the model can more accurately capture the dynamic dependencies between time points and spatial locations. The FGITNet layer structure of this invention supports flexible adjustment, and users can adjust the depth of the model according to specific task requirements to adapt to video data detection tasks of different complexities, thus exhibiting good adaptability.

[0033] This invention demonstrates excellent computational efficiency. By integrating Transformer, it provides an optimized computational method, significantly reducing computational complexity and making the model more suitable for real-time detection. At the same time, it can dynamically adapt to changes in video surveillance, thereby improving detection accuracy.

[0034] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings.

[0035] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0037] Figure 1 A schematic diagram of the FGITNet dual-stream gated network model of this invention;

[0038] Figure 2 Flowchart of the dual-stream gating violent detection method of the present invention;

[0039] Figure 3 A schematic diagram of the spatiotemporal modeling process of this invention;

[0040] Figure 4 A schematic diagram of the FGITNet classification stage of this invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the following description, specific details such as specific configurations and components are provided merely to help fully understand the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. In addition, for clarity and brevity, descriptions of known functions and structures are omitted in the embodiments.

[0042] It should be understood that the phrase "an embodiment" or "this embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "an embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0043] Furthermore, reference numerals and / or letters may be repeated in different examples within this application. Such repetition is for the purpose of simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0044] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" in this article describes another type of relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the related objects before and after it are in an "or" relationship.

[0045] In this article, the term "at least one" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, "at least one of A and B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0046] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion.

[0047] Example 1

[0048] This embodiment mainly describes a two-stream gated brute-force detection method based on dilated 3D convolutional networks and Transformers, R as... Figure 2 As shown, it includes the following steps:

[0049] S1: Acquire the video stream, divide the video stream into frames, and calculate the dense optical flow between consecutive frames of the video stream;

[0050] S2: Video stream segments are divided by a sliding window, and non-moving elements in the RGB mode are suppressed using optical flow modes;

[0051] S3: Spatiotemporal modeling of dense optical flow and sRGB modes is performed using a 3D convolutional neural network, and feature fusion is carried out to create a feature map;

[0052] S4: The temporal feature sequence of the video stream segment is modeled using a Transformer encoder to generate a temporal feature map;

[0053] S5: Average pooling is performed using a 1D convolutional neural network, and classification is performed using a multilayer perceptron to classify the video as violent or non-violent.

[0054] Furthermore, in S1, consecutive frames of the video stream are normalized using Z-score. After processing, the dense optical flow between consecutive frames of the video stream is calculated using the Farneback method, with the formula: DOF t=Farneback(RGB) t RGB t+1 ), where RGB t For continuous frames at time t, RGB t+1 For consecutive frames at time t+1, DOF t Let t be the dense optical flow; by iterating the above formula over each frame of a continuous video stream, n-1 dense optical flow sequences are obtained, where n represents the length of the video.

[0055] Furthermore, as shown in Table 1, in S2, the dense optical flow is divided into m segments using a sliding window method, and the suppression effect is enhanced by accumulating the optical flow vector amplitude of all frames. The formula is: Where |DOF t | represents the DOF frame at time t ∈ [i, j], where i and j are indices indicating the start and end frames within the k-th DOF segment. A k To accumulate all frames; the accumulated optical flow vector amplitude of all frames is obtained by normalizing the accumulated optical flow amplitude using a minimum-maximum normalization (MMN) method, scaling it to a value between 0 and 1. The normalized accumulated value is used as a scaling factor for element-wise multiplication adjustment of the RGB segments, suppressing non-moving elements. The formula is as follows: in For the i-th frame of the k-th RGB segment, For the suppressed RGB frame, Element-wise multiplication across all spatial dimensions and color channels yields the suppressed sRGB fragment.

[0056] Table 1 Background Suppression

[0057]

[0058]

[0059] Furthermore, such as Figure 3 As shown, in S3, the pre-trained I3D model extracts spatiotemporal features from the SRGB mode and dense optical flow mode to obtain segment spatiotemporal features. The segment spatiotemporal features are fused by element-wise multiplication to generate a comprehensive segment spatiotemporal feature map. The segment spatiotemporal feature maps are connected along the time dimension to form two spatiotemporal feature sequences of shape l×1024, where l is the sum of all segment output logical values ​​determined by the segment size and the number of segments.

[0060] Furthermore, in S4, the Transformer encoder receives the output X from the spatiotemporal modeling of the segment. X is then transformed into a query Q, a key K, and a value V, using the formula: Q = XW Q K = XW K V = XW V WQ W K W V It is a parameter matrix that maps the input X to the respective spaces of query, key, and value, performs temporal modeling on the spatiotemporal feature sequence of each segment, and generates a temporal feature map through the Transformer encoder.

[0061] Furthermore, a multi-head self-attention mechanism using a Transformer encoder is employed to select video segments depicting violent activities. This multi-head self-attention mechanism transforms the input X using the formula: MultiHead(Q, K, V) = {Concat(head)1, ...,head} h W O Where h represents the number of attention heads, W O The parameter matrix of the multi-head self-attention mechanism, where It is a linear transformation of the attention heads of the query Q, key K, and value V; the multi-head self-attention mechanism transforms the input query Q, key K, and value V, as shown in the formula: Where d k is the dimension of each key, and softmax is the activation function.

[0062] Furthermore, the multi-head self-attention mechanism of the Transformer encoder normalizes the output X and passes it through a residual connection, with the formula: Z = LayerNorm(X + MultiHead(Q, K, V)). Each position in Z is processed independently through the feedforward network FFN, with the formula: FFN(z) = max(0, zW1 + b1)W2 + b2, where W1 and W2 are the weight parameters of the feedforward network, and b1 and b2 are the bias coefficients. The output of the feedforward network FFN is normalized and added back to Z through another residual connection, with the formula: Y = LayerNorm(Z + FFN(Z)), where Y is the output value.

[0063] Furthermore, in S5, average pooling is performed through a 1D convolutional neural network. The input data is fed into the 1D convolutional neural network architecture in the time dimension, and a sliding window operation is performed on the time series. For the data points in each window, the average value is calculated to realize the downsampling process of the time features; the dimension of the input data is compressed and a tensor is output.

[0064] Furthermore, after the 1D convolutional neural network completes average pooling, the data enters a multilayer perceptron for classification. The output tensor obtained after average pooling is flattened into a vector, which is then fed into the 1D convolutional neural network as input to the multilayer perceptron. The multilayer perceptron consists of multiple fully connected layers. The input vector and the neurons of the multilayer perceptron undergo a linear transformation through a weight matrix, followed by a nonlinear transformation through an activation function. The transformed result is then passed to the next layer, repeating the linear and nonlinear transformation processes. After multilayer processing, the data reaches the output layer, which outputs the classification of non-violent or violent behavior in the video. The activation values ​​of two neurons in the output layer determine whether the video is classified as violent or non-violent.

[0065] This embodiment captures complex spatial and temporal relationships by processing video structural data, using optical flow as a unique feature for each frame, and also using RGB data as multiple features. It captures video relationships through a spatiotemporal FGITNet layer, which is then processed by a classification layer to make a final prediction. By combining a dilated 3D convolutional network with a Transformer encoder architecture, it innovatively solves the shortcomings of traditional brute-force detection models in handling spatiotemporal dependencies.

[0066] Example 2

[0067] This embodiment describes in detail a two-stream gated brute-force detection system based on dilated 3D convolutional networks and Transformers, with specific connection methods as follows: Figure 1 As shown, the dual-stream gated brute-force detection system includes a dual-stream gated network FGITNet model; the dual-stream gated network FGITNet model includes a data preprocessing module, a background suppression module, a fragment spatiotemporal modeling module, a fragment temporal modeling module, and a classification module;

[0068] The data preprocessing module standardizes the input surveillance video, adjusts the resolution pixels of the images, such as standardizing the resolution of each image to 224×224 pixels, normalizes the image data to conform to the standard format through Z-score normalization, and extracts a fixed number of frames through uniform sampling to generate standardized video frame data.

[0069] The background suppression module suppresses non-moving elements in the video, extracts segments from RGB frames and dense optical flow frames using a sliding window method, calculates the amplitude of motion information in dense optical flow segments, adjusts the optical flow amplitude to a value between 0 and 1 through minimum-maximum normalization, and adjusts RGB segments by element-wise multiplication.

[0070] The background suppression module utilizes optical flow modes to suppress non-moving elements, denoted as follows: The sRGBclips module retrieves optical flow amplitudes from DOFclips for each time step, normalizes and broadcasts these amplitudes to form motion suppression representations of the video sequence, and the background suppression module effectively encodes dynamic patterns in the video, enhancing the model's ability to capture repetitive motions in the video.

[0071] The fragment spatiotemporal modeling module uses a pre-trained I3D model to extract spatiotemporal features from suppressed RGB modes and dense optical flow modes. It independently models spatiotemporal features on each fragment of length clip_size, and fuses the two types of modal features by element-wise multiplication to form a comprehensive spatiotemporal feature map of the fragment.

[0072] The segment spatiotemporal modeling module uses I3D to perform spatiotemporal modeling of DOF and RGB modes for each segment in the video, where DOF stands for dense optical flow; and combines the spatiotemporal features of DOF and RGB in each segment to perform feature fusion and create l feature maps.

[0073] This invention utilizes the I3D model as the backbone network to extract spatiotemporal features from both sRGB and DOF modalities. The sRGB modality consists of three channels, while the DOF modality includes two channels in the vertical and horizontal directions, respectively processed using the I3D model. The FGITNet model is configured with the corresponding number of input channels: 3 for sRGB and 2 for DOF, and the output layer dimension is set to 1024. Independent spatiotemporal modeling is performed on each video clip of length `clip_size` to construct a spatiotemporal feature sequence for each clip. By applying the I3D model to these clips separately, the features of each clip are captured. These features are fused using element-wise multiplication to generate a comprehensive spatiotemporal feature map of the clips. The spatiotemporal feature maps of each clip are then connected along the time dimension to form a spatiotemporal feature sequence of shape l×1024, where r is determined by the clip size and number of clips. This sequence integrates the spatiotemporal features of each clip's sRGB and DOF modalities. Each logit in the sequence serves as an input token for subsequent models to perform inter-segment temporal modeling, providing support for the analysis of spatiotemporal features.

[0074] The fragment temporal modeling module adopts a Transformer encoder structure and introduces a multi-head self-attention mechanism. It takes the spatiotemporal feature sequence output by the fragment spatiotemporal modeling module as input and performs linear transformation, multi-head self-attention calculation, residual connection and feedforward network processing operations to enable the FGITNet model to selectively observe fragments of violent activity and generate temporal feature maps.

[0075] The classification module downsamples the temporal features using 1D average pooling, compresses the dimension of the input data, outputs a tensor, then flattens it into a vector, processes it through a multilayer perceptron containing multiple fully connected layers, and outputs the classification result of each video segment as either non-violent or violent behavior.

[0076] like Figure 4 As shown, the input data in the classification module is downsampled by time features and then 1D average pooling (AvgPooling) is performed in the time dimension to reduce dimensionality and retain time information. The output of the pooling layer is a tensor of shape 8×1024, which is then flattened and normalized into a vector of size 8192. The vector is processed by an MLP containing fully connected layers of dimensions 2048, 512, and finally, corresponding to the classification output of non-violent or violent behavior in the video.

[0077] The video frame data processed by the data preprocessing module enters the background suppression module. The data processed by the background suppression module is then sent to the segment spatiotemporal modeling module. The spatiotemporal feature sequence generated by the segment spatiotemporal modeling module is passed as input to the segment temporal modeling module. The temporal feature map output by the segment temporal modeling module is then sent to the classification module. After processing by the classification module, the video classification result is output.

[0078] This embodiment describes in detail the dual-stream gated brute-force detection system. The video sequence analysis capability of the segment spatiotemporal modeling module is optimized. Through the dilated 3D convolutional network and Transformer encoder mechanism, it can more accurately capture the dynamic dependencies between time points and spatial locations. The depth of the model can be adjusted according to specific task requirements to adapt to video data detection tasks of different complexities, and it has good adaptability.

[0079] Example 3

[0080] This embodiment, based on Embodiment 1, describes in detail the technical effects of this application, specifically including:

[0081] This application uses three main metrics to evaluate the effectiveness of our proposed model: accuracy, precision, and recall. Accuracy quantifies the proportion of correctly predicted instances, including both true positives and true negatives in all predictions. Precision is a measure of the accuracy of positive predictions; it represents the proportion of predicted positive instances that are true positives, thus minimizing unnecessary alerts and maintaining trust in system notifications. Recall determines the model's ability to identify all relevant instances, representing the proportion of actual positive instances that are correctly predicted as positive. This is crucial for detecting as many genuine violent incidents as possible, thereby enhancing public safety and reducing the risk of missed incidents. Our model is comprehensively evaluated using these metrics, and accuracy is used for comparison with other models.

[0082] The RWF-2000 dataset, the largest and most extensive benchmark dataset for the VD task, is used, containing 2000 real-world surveillance videos from YouTube. These videos are divided into two groups: 1000 labeled "fighting" and another 1000 labeled "non-fighting." For comparison, the Hockey dataset, which covers 1000 videos of hockey games captured by the National Hockey League (NHL), is used. Similar to RWF-2000, the Hockey dataset is also divided into two groups: 500 labeled "fighting" and another 500 labeled "non-fighting." Both datasets show an equal distribution between violent and non-violent videos, demonstrating a good property of no class imbalance, which is advantageous for binary classification tasks. See Table 2 for details on these datasets;

[0083] Table 2 Details of the datasets used in the invention

[0084]

[0085] Experimental results:

[0086] The performance of FGITNet was evaluated on the RWF-2000 and Hockey datasets to assess its ability to detect violence in real-world fight videos. In the experiments, we used 80% of each dataset for training and validation, reserving the remaining 20% ​​for testing. Before training, all videos were preprocessed to ensure consistency in frame resolution and length, specifically: each frame was resized to 224×224 pixels, Z-score normalized, and uniformly sampled across a fixed number of frames to ensure consistency across datasets. The FGITNet model was compared with existing models, as shown in Table 3.

[0087] Table 3 Comparison of binary classification accuracy (%) on the benchmark dataset

[0088]

[0089] As can be seen from Table 3, the technical solution of this application has a high accuracy rate;

[0090] By comparing the model's evaluation on the benchmark datasets in Table 4, we can further see that the accuracy of this application achieved the highest accuracy of 93.75% and 99.50% on the RWF-2000 and Hockey datasets, respectively, as shown in Table 4.

[0091] Table 4. Model evaluation on benchmark datasets

[0092]

[0093] To further evaluate the effectiveness of the background suppression mechanism in FGITNet, an ablation study was conducted on two variants of the model on the RWF-2000 dataset. The baseline model was FGITNet without the background suppression module, while the experimental model included the aforementioned background suppression mechanism. The background suppression mechanism improves the model's performance by reducing non-motion elements and emphasizing motion-related features. Specifically, it calculates the cumulative optical flow amplitude for each segment k, uses a minimum-maximum normalized MMN, and adjusts the corresponding RGB segments through element-wise multiplication.

[0094] By integrating this mechanism, the model effectively suppresses static background regions, allowing it to focus on dynamic regions that are more indicative of violent activity. This targeted focus enhances motion-related feature extraction, thereby improving classification performance, as shown in Table 5.

[0095] Table 5 Ablation Study of FGITNet on the RWF-2000 Dataset

[0096]

[0097] Table 5 shows a comparative analysis of the baseline model and the experimental model in terms of various performance indicators such as accuracy, precision, and recall. The experimental model shows significant improvements over the baseline model in all indicators, highlighting the effectiveness of the background suppression mechanism.

[0098] This embodiment validates the effectiveness of the background suppression mechanism through significant improvements observed in the experimental model. By focusing on motion center features and reducing background interference, the model achieves superior classification results on the RWF-2000 dataset. The background suppression mechanism plays a crucial role in improving FGITNet performance, enabling the model to focus on important motion features while mitigating the influence of non-informative background elements, thereby improving the accuracy and reliability of brute force detection tasks.

[0099] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any changes, modifications, substitutions, integrations, and parameter changes made to these embodiments within the spirit and principles of the present invention, without departing from the principles and spirit of the present invention, through conventional substitutions or to achieve the same function, fall within the scope of protection of the present invention.

Claims

1. A dual-flow gated violence detection method based on dilated 3D convolution network and Transformer, characterized in that, Includes the following steps: S1: Acquire the video stream, divide the video stream into frames, and calculate the dense optical flow between consecutive frames of the video stream; S2: Video stream segments are divided by a sliding window, and non-moving elements in the RGB mode are suppressed using optical flow modes; S3: Spatiotemporal modeling of dense optical flow and sRGB modes is performed using a 3D convolutional neural network, and feature fusion is carried out to create a feature map; S4: The temporal feature sequence of the video stream segment is modeled using a Transformer encoder to generate a temporal feature map; S5: Average pooling is performed using a 1D convolutional neural network, and the video is classified as violent or non-violent using a multilayer perceptron. In S1, consecutive frames of the video stream are normalized using Z-score. After processing, the dense optical flow between consecutive frames of the video stream is calculated using the Farneback method, with the following formula: ,in for Temporal consecutive frames, for Temporal consecutive frames, for Temporally dense optical flow; obtained by iterating the above formula over each frame of a continuous video stream. A dense optical flow sequence, in which Indicates the length of the video; In step S2, the dense optical flow is divided into segments using a sliding window method. The suppression effect is enhanced by accumulating the optical flow vector amplitude of all frames for each segment, using the following formula: ,in For time of frame, and For index, indicating the first indivual The start and end frames within a clip, To accumulate all frames; the optical flow vector magnitude of all accumulated frames is obtained through min-max normalization. The accumulated optical flow amplitude is normalized and scaled to a value between 0 and 1. The normalized accumulated value is then used as a scaling factor for element-wise multiplication adjustment. Fragments, suppressing non-moving elements, the formula is: ,in For the first indivual The first segment frame, For the suppressed frame, Element-wise multiplication across all spatial dimensions and color channels yields the suppressed [color]. Fragment; The S3 uses a pre-trained I3D model from Spatiotemporal features are extracted from modalities and dense optical flow modes to obtain fragment spatiotemporal features. These fragment spatiotemporal features are then fused through element-wise multiplication to generate a comprehensive fragment spatiotemporal feature map. The fragment spatiotemporal feature maps are then connected along the time dimension to form two shapes. The spatiotemporal feature sequence, where Output the sum of logical values ​​for all segments determined by the segment size and number of segments; The Transformer encoder in S4 receives the output of the spatio-temporal modeling of the segment ; the output X is respectively mapped as query Q, key K and value V, and the mapping relationship is as follows: , wherein, are respectively parameter matrices for mapping the input X to a specific space of query, key and value; the time feature map is generated by using the Transformer encoder to perform time modeling on the spatio-temporal feature sequence of each video segment.

2. The dual-flow gated violence detection method based on dilated 3D convolutional network and Transformer according to claim 1, wherein, The selection of video segments depicting violent activity is achieved through a multi-head self-attention mechanism using a Transformer encoder; the multi-head self-attention mechanism uses the input... The conversion is performed using the following formula: ,in Indicates the number of attention heads. The parameter matrix of the multi-head self-attention mechanism, where , It is a query ,key Sum The attention head linear transformation; the multi-head self-attention mechanism uses the input query... ,key Sum The conversion is performed using the following formula: ,in It is the dimension of each key. This is the activation function.

3. The two-stream gated brute-force detection method based on dilated 3D convolutional networks and Transformers according to claim 2, characterized in that, The multi-head self-attention mechanism of the Transformer encoder outputs... Normalization is performed and propagated through residual connections, using the following formula: , The output of the multi-head self-attention mechanism, after normalization, is the intermediate representation, representing an intermediate state in the Transformer encoder, carrying the current layer's global feature information about the input X; This yields... Each position in the process is processed independently through a feedforward network (FFN), as shown in the formula: ,in , These are the weight parameters of the feedforward network. , The bias coefficient is used; the output of the feedforward network FFN is normalized and then added back through another residual connection. The formula is: ,in As the output value, it combines Global features were extracted and the input was preserved through residual connections. Partial information.

4. The dual-flow gated violence detection method based on dilated 3D convolutional network and Transformer according to claim 1, characterized in that, In S5, average pooling is performed using a 1D convolutional neural network. The input data is fed into the 1D convolutional neural network architecture in the time dimension, and a sliding window operation is performed on the time series. For each data point in the window, the average value is calculated to realize the downsampling process of the time features. The dimension of the input data is compressed and a tensor is output.

5. The dual-flow gated violence detection method based on dilated 3D convolutional network and Transformer according to claim 4, characterized in that, After average pooling in a 1D convolutional neural network, the data enters a multilayer perceptron for classification. The output tensor obtained after average pooling is flattened into a vector, which is then fed into the 1D convolutional neural network as input to the multilayer perceptron. The multilayer perceptron consists of multiple fully connected layers. The input vector and the neurons of the multilayer perceptron undergo a linear transformation through a weight matrix, followed by a nonlinear transformation through an activation function. The transformed result is then passed to the next layer, repeating the linear and nonlinear transformation processes. After multilayer processing, the data reaches the output layer, which outputs a classification of non-violent or violent behavior in the video. The activation values ​​of two neurons in the output layer determine whether the video is classified as violent or non-violent.

6. A dual-stream gated brute-force detection system based on dilated 3D convolutional networks and Transformers, applicable to any one of claims 1-5, characterized in that, The system includes a two-stream gated network (FGITNet) model; the FGITNet model includes a data preprocessing module, a background suppression module, a segment spatiotemporal modeling module, a segment temporal modeling module, and a classification module. The data preprocessing module standardizes the input surveillance video, adjusts the resolution pixels of the image, normalizes the image data through Z-score to make it conform to the standard format, and extracts a fixed number of frames through uniform sampling to generate standardized video frame data. The background suppression module suppresses non-moving elements in the video, extracts segments from RGB frames and dense optical flow frames using a sliding window method, calculates the amplitude of motion information in dense optical flow segments, adjusts the optical flow amplitude to a value between 0 and 1 through minimum-maximum normalization, and adjusts RGB segments by element-wise multiplication. The segment spatiotemporal modeling module uses a pre-trained I3D model to extract spatiotemporal features from the suppressed RGB mode and dense optical flow mode. It independently models spatiotemporal features on each segment of length clip_size, and fuses the two types of modal features by element-wise multiplication to form a comprehensive spatiotemporal feature map of the segment. The segment time modeling module adopts a Transformer encoder structure and introduces a multi-head self-attention mechanism. It takes the spatiotemporal feature sequence output by the segment spatiotemporal modeling module as input, and through linear transformation, multi-head self-attention calculation, residual connection and feedforward network processing operations, the FGITNet model selectively observes segments of violent activity and generates a time feature map. The classification module downsamples the temporal features using 1D average pooling, compresses the dimension of the input data, outputs a tensor, then flattens it into a vector, processes it through a multilayer perceptron containing multiple fully connected layers, and outputs the classification result of each video segment as either non-violent or violent behavior. The video frame data processed by the data preprocessing module enters the background suppression module. The data processed by the background suppression module is then sent to the segment spatiotemporal modeling module. The spatiotemporal feature sequence generated by the segment spatiotemporal modeling module is passed as input to the segment temporal modeling module. The temporal feature map output by the segment temporal modeling module is then sent to the classification module. After processing by the classification module, the video classification result is output.

Images