Violent emergency classification method, system and device based on multi-modal information fusion and medium

Abstract

The invention discloses a violent emergency recognition and classification method, system and device based on multi-modal information fusion. The method comprises the steps that firstly, audio and video data from multiple monitoring sources are acquired and preprocessed, and the data sources comprise video monitoring equipment and audio acquisition equipment; then, I3D and VGGish models are used for respectively extracting spatial-temporal features and acoustic features of the video data and the audio data; then, cross-modal fusion is carried out on the video features and the audio features through a multi-modal feature fusion module of a self-attention mechanism, and deep association between multi-modal information is captured; and learning and weighted fusion are carried out on the fused features through a time sequence convolutional network model, so that the representation capability of the features is further improved. And finally, classifying the fused feature vectors by adopting a classification model (GAT-C) based on a graph neural network, and accurately identifying the type of the violent emergency. According to the multi-modal fusion technology, the recognition precision of violent events is greatly improved, and good robustness and classification capability are shown especially in complex and diverse violent scenes. The method not only can effectively improve the event response efficiency and guarantee the public safety, but also has relatively high adaptability and popularization value.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and emergency classification, and provides an emergency classification method, system and device based on multimodal feature fusion. Technical Field

[0003] The present invention relates to the fields of video surveillance and public safety technology, and in particular to a method, system, device and storage medium for classifying violent incidents based on multimodal information fusion. Background Art

[0004] With the acceleration of urbanization, public safety issues are becoming increasingly prominent, and the frequency of violent incidents is gradually increasing. Traditional video surveillance systems rely primarily on manual monitoring, which is inefficient and prone to missed detections. In recent years, deep learning-based video analysis technology has been widely used in violent incident detection. However, due to the complexity and diversity of violent incidents, single-modality data (such as video) often fails to provide sufficient information, resulting in low detection accuracy and generalization. Multimodal fusion technology effectively improves detection accuracy and generalization by combining data from multiple modalities, such as video and audio. However, existing multimodal fusion methods are prone to information loss during data processing and fail to fully utilize the advantages of each modality. Furthermore, violent incidents exhibit large intra-class variability and small inter-class variability, resulting in low classification accuracy. To address these issues, this paper proposes a violent incident classification method, system, and device based on multimodal information fusion. This method effectively fuses multimodal features through a graph attention network (GAT-C), thereby improving the accuracy and robustness of violent incident classification. Summary of the Invention

[0005] To address the aforementioned shortcomings of the prior art, the present invention aims to provide a method, apparatus, device, and medium for classifying violent incidents based on multimodal information fusion. This approach enables intelligent detection and classification of violent incidents in public spaces, improving detection accuracy and efficiency, enabling timely identification and classification of various types of violent incidents, safeguarding public safety, and supporting relevant decision-making. By combining multimodal information such as video surveillance and audio data, the present invention effectively enhances the accuracy and robustness of event classification, overcomes the detection limitations of single-modal data, and improves the system's adaptability to complex and diverse violent incidents.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0007] A method for classifying violent incidents based on multimodal information fusion, the method comprising:

[0008] (1) Obtain video surveillance data and synchronous audio data in the XD-violence public dataset, and divide the collected audio and video data into four categories: physical conflict, weapon attack, group riot, and non-violent event according to the "GA / T1583-2019 Technical Specification for Public Security Video Surveillance System", construct a violent incident sample dataset represented by multi-modal data and preprocess it;

[0009] (2) Use I3D model to extract spatio-temporal features of video data. I3D model is based on 3D convolution operation, which can extract spatial and temporal features from video and capture dynamic change information in video. Use VGGish model to extract acoustic features of audio data. Audio data extracts frequency, amplitude, waveform and other information through convolutional neural network, which can effectively capture key signals in sound (such as impact sound, shouting sound, etc.);

[0010] (3) Construct a multi-modal feature fusion module based on self-attention mechanism, which fuses video and audio features through self-attention mechanism. Self-attention mechanism can weight different modalities and dynamically adjust the importance of video and audio features to ensure that important information gets more attention. This mechanism calculates the similarity between video and audio features to get the weighted fused feature representation;

[0011] (4) Based on the early fusion features obtained in step (3), use time convolution network (TCN) model to weight and fuse the fused feature vectors. Time convolution network can model the time sequence features of video and audio, capture the dynamic changes of violent events in time sequence, and get the late fusion feature vector. This vector can accurately represent the spatio-temporal features of violent events during the occurrence process;

[0012] (5) Use a classification model based on graph neural network (GAT-C) to learn and classify the late fusion feature vectors. Graph neural network establishes a graph structure between samples, considers the relationship of multi-modal data as nodes and edges of the graph, and realizes information sharing and propagation through graph aggregation process. Finally, the classification results of violent incidents are output, including physical conflict, weapon attack, group riot, non-violent event, etc.

[0013] According to the preferred embodiment of the present invention, in step (1), a data set is constructed, and video surveillance data and synchronized audio data of violent incidents are obtained from the XD-violence public data set; in terms of data preprocessing, two modal data containing video and corresponding audio are retained, and the sample ratio of video to audio data is 1:1. Key frame extraction is performed on the video data, 16 frames are uniformly sampled for each video segment, and the frame size is adjusted to 224×224; the audio data is framed, each audio segment is divided into 1-second segments, and Mel spectrum features are extracted. Blurred or severely jittered frames in the video and silent or noisy segments in the audio are removed to ensure data quality. According to the "GA / T 1583-2019 Technical Specifications for Public Security Video Surveillance Systems", the collected audio and video data are categorized and labeled, and violent incidents are divided into four categories (physical conflict, armed attack, group riot, and non-violent incidents) based on the nature and behavioral characteristics of the incidents.

[0014] According to the preferred embodiment of the present invention, in step (2), the multi-modal data is preprocessed, and the implementation process is as follows: the video feature extractor adopts the I3D model, inputs the video key frame sequence (16 frames, each frame size is 224×224), and extracts spatiotemporal features through 3D convolution and pooling operations. First, the video frame sequence is input into the Inflated 3D convolutional network to extract spatial and temporal information respectively; secondly, the dimension is gradually reduced through multi-layer 3D convolution and maximum pooling, and finally the video feature vector is output, and the feature vector dimension is 1024. The audio feature extractor adopts the VGGish model, inputs the Mel spectrum map of the audio (96 Mel bands, each segment is 1 second), and extracts acoustic features through 2D convolution and fully connected layers. First, the Mel spectrum map is input into the VGGish network, and local features are extracted through multiple convolution blocks (each block contains convolution, ReLU and maximum pooling); secondly, the features are further compressed through the fully connected layer, and finally the audio feature vector is output, and the feature vector dimension is 128.

[0015] Preferably, according to the present invention, the specific implementation process of step (3) is as follows: early fusion of different modal feature vectors is performed, and a multimodal feature fusion module is constructed to fuse the extracted video feature vectors and audio feature vectors. The multimodal feature fusion module uses a self-attention mechanism, takes video features as queries (Q), audio features as keys (K) and values ​​(V), calculates the similarity weights of video and audio features, and performs weighted fusion on audio features; conversely, takes audio features as queries (Q), video features as keys (K) and values ​​(V), calculates the similarity weights of audio and video features, and performs weighted fusion on video features. The fusion results of the two directions are added together to fully capture the spatiotemporal alignment relationship between audio and video modalities, and obtain an early feature fusion result with a feature vector dimension of 1024.

[0016] Preferably, according to the present invention, the specific implementation process of step (4) is as follows: for the three types of feature vectors extracted in steps 2 and 3 - video features (1024 dimensions), audio features (128 dimensions) and early fusion features (1024 dimensions), three sets of temporal convolutional networks (TCN) with consistent structures are respectively constructed for modeling. Each TCN consists of 4 consecutive temporal convolution blocks, each of which includes a one-dimensional dilated convolution (the convolution kernel size is 3, and the dilation rates are 1, 2, 4, and 8, respectively), weight normalization and ReLU activation function, and a residual connection is used to ensure stability and information retention. During the convolution stacking process, the channel dimension is gradually compressed, mapped from the original dimension to a unified 256 dimension, and the output is a T×256 temporal feature representation. Subsequently, the TCN outputs of the three modalities are globally average pooled to obtain three 1×256 vectors. These are then concatenated and fed into a fully connected layer to generate three normalized weights (using Softmax activation) for weighted fusion of the temporal features of the three modalities. Finally, the three sets of T×256 features are weighted and summed along the time dimension to obtain the fused multimodal feature representation with a dimension of T×256, which serves as the final fusion output.

[0017] According to the preferred embodiment of the present invention, the specific implementation process of step (5) is as follows: a classifier based on a graph attention network (GAT-C) is constructed, which consists of two graph attention layers and a fully connected output layer. The input layer receives a 384-dimensional fused feature vector. The first graph attention layer contains four attention heads, each of which outputs 64-dimensional features, which are spliced ​​to form a 256-dimensional intermediate representation; the second graph attention layer uses an average aggregation method to output 128-dimensional refined features. The final fully connected output layer contains four neurons, corresponding to four categories of violent incidents (physical conflict, armed assault, group riot, and non-violent incidents). The GELU activation function is used for nonlinear transformation, and the category probability distribution is calculated by the softmax function. In particular, we added residual connections and layer normalization between the graph attention layers, and used label smoothing technology (smoothing factor = 0.1) during training to improve the generalization ability of the model, ultimately achieving end-to-end violent incident classification prediction.

[0018] A device for classifying violent emergency events based on multimodal information fusion, used to implement the method for classifying violent emergency events based on multimodal information fusion described in any one of claims 1 to 5, characterized in that the device includes: a data acquisition and processing module, a feature extraction module, an event classification module, and a result display module.

[0019] The data acquisition and processing module is used to obtain audio and video data of violent incidents through video surveillance equipment, audio acquisition equipment, etc., and preprocess this data, including removing low-quality data and noise interference; the feature extraction module is used to use the I3D model and the VGGish model to extract the spatiotemporal features of video modal data and the acoustic features of audio modal data respectively; the event classification module is used to call the violent emergency classification method based on multimodal information fusion, learn and classify the fused features, and identify the specific types of violent emergencies (such as physical conflicts, armed attacks, mass riots, non-violent incidents, etc.); the result display module is used to display the classification results and provide visual analysis for relevant personnel to use in decision-making.

[0020] A terminal device, characterized in that the terminal device includes a memory, a processor, and a violent emergency event classification program based on multimodal information fusion stored in the memory and executable on the processor, wherein when the processor executes the violent emergency event classification method based on multimodal information fusion, the steps of the violent emergency event classification method based on multimodal information fusion as described in any one of claims 1 to 5 are implemented. The implementation process is as follows:

[0021] The terminal device executes a violent emergency classification method program based on multimodal information fusion through a built-in processor. After obtaining video surveillance and audio data, it uses I3D and VGGish models to extract features of video and audio data, fuses features of different modalities through a self-attention mechanism, and then performs late fusion through a temporal convolutional network. Finally, it performs classification and prediction through a graph neural network classifier, identifies the type of violent incident, and displays the results on the device interface.

[0022] A computer storage medium, characterized in that a program for classifying a violent emergency event based on multimodal information fusion is stored on the computer storage medium, and when the program for classifying a violent emergency event based on multimodal information fusion is executed by a processor, the steps of the method for classifying a violent emergency event based on multimodal information fusion according to any one of claims 1 to 5 are implemented.

[0023] Beneficial effects of the present invention:

[0024] 1. The violent incident identification method proposed in this paper comprehensively considers image and audio information, constructs a unified multimodal feature representation framework, and can more comprehensively and accurately describe the semantic and visual characteristics of violent incidents, significantly improving the robustness of identification.

[0025] 2. This paper designs a deep model that integrates a convolutional neural network (CNN) and a multi-layer Transformer structure. During the image feature extraction stage, it retains the local details of violent behavior in the image. At the same time, it uses the Transformer structure to deeply model the audio semantic information, enabling the system to still have good generalization capabilities when facing complex and diverse violent scenes.

[0026] 3. This paper proposes a multimodal attention mechanism for establishing an alignment between images and audio. It can automatically focus on key areas and keywords describing violent behavior, improving the model's sensitivity to key information. Compared with traditional recognition methods based on only a single modality, it performs better in terms of recognition accuracy and scenario adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of the steps of the violent incident classification method based on multimodal information fusion of the present invention.

[0028] Figure 2 This is a structural diagram of the I3D model of the present invention

[0029] Figure 3 This is the structural diagram of the VGGish model of the present invention DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only some preliminary embodiments of the present invention, rather than all embodiments.

[0031] It should be understood that the step numbers used herein are only for the convenience of description and are not intended to limit the order in which the steps are executed.

[0032] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0033] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] Example 1

[0035] A violent incident classification method based on multimodal information fusion, such as Figure 1 As shown, the following steps are included:

[0036] (1) Obtain video surveillance data and synchronized audio data from the XD-violence public dataset. According to the GA / T1583-2019 Technical Specifications for Public Security Video Surveillance Systems, the collected audio and video data are divided into four categories: physical conflict, armed assault, group riots, and non-violent incidents. A sample dataset of violent incidents represented by multimodal data is constructed and preprocessed.

[0037] (2) Use the I3D model to extract spatiotemporal features from video data. The I3D model is based on 3D convolution operations and can extract features in both spatial and temporal dimensions from videos, capturing dynamic changes in the video. Use the VGGish model to extract acoustic features from audio data. The convolutional neural network extracts frequency, amplitude, waveform, and other information from audio data, effectively capturing key signals in the sound (such as impact sounds, shouting, etc.);

[0038] (3) Construct a multimodal feature fusion module based on the self-attention mechanism to fuse video and audio features through the self-attention mechanism. The self-attention mechanism can weight different modalities and dynamically adjust the importance of video and audio features to ensure that important information receives more attention. The mechanism calculates the similarity between video and audio features to obtain a weighted fusion feature representation;

[0039] (4) Based on the early fusion features obtained in step (3), the fused feature vectors are weighted fused using a temporal convolutional network (TCN) model. The TCN model can model the temporal features of video and audio, capture the dynamic changes of violent events in the time series, and obtain a late fusion feature vector. This vector can accurately represent the spatiotemporal characteristics of the violent event process;

[0040] (5) A classification model based on a graph neural network (GAT-C) is used to learn and classify the feature vectors after late fusion. The graph neural network establishes a graph structure between samples, treats the relationship between multimodal data as nodes and edges of the graph, and realizes information sharing and dissemination through the aggregation process of the graph. Finally, the classification results of violent emergencies are output, which are specifically classified into physical conflicts, armed attacks, mass riots, non-violent incidents, etc.

[0041] Example 2

[0042] The intelligent inspection method for gas storage based on multimodal data fusion described in Example 1 is different in that:

[0043] In step (1), the pre-processing of the obtained video data and audio data means that the audio and video data obtained from the XD-violence public data set is divided into four categories of physical conflict, weapon attack, group riot and non-violent event according to the "GA / T 1583-2019 Technical Specification for Public Security Video Monitoring System", and data cleaning, de-duplication and missing value supplement are performed. Video decoding is performed on the video data, and feature extraction is performed on the audio data to ensure the synchronization of video and audio data. After data preprocessing, the video and audio data are subjected to data normalization and time alignment operation. For noisy multi-modal data, a joint variational model is used to denoise the data to reduce the influence of noise on data quality, as shown in formula (1):

[0044]

[0045] where u m (x) represents the input data from modality m, ω m is a non-negative weight satisfying

[0046] Embodiment 3

[0047] The violence emergency classification method based on multi-modal information fusion according to any one of embodiments 1 or 2 is different in that:

[0048] The specific implementation process of step (3) is as follows:

[0049] The text module, as shown in Figure 2 , can be represented as formula (2):

[0050] F T =f T (X T ; θ T ) (2)

[0051] where F T represents the output of the text module (text feature representation), X T represents the text input, and θ T represents the parameters of the text module.

[0052] The text module is composed of a feature extraction module and a sequence attention module. The main component of the feature extraction module is a fine-tuned BERT model for extracting intermediate feature representation. The sequence attention module is used to generate enhanced context information text feature representation. Finally, a fully connected layer is added to constrain the length of the feature sequence to adjust the text feature representation.

[0053] The visual module, as shown in Figure 1 , can be represented as formula (3):

[0054] F V =f V (X V θ V ) (3)

[0055] Among them, F V represents the output of the visual module (image feature representation), X V is the video input, θ V Represents the parameters of the vision module.

[0056] Audio modules such as Figure 1 As shown, it can be expressed as formula (4):

[0057] F A =f A (T;θ A ) (4)

[0058] Among them, F A represents the output of the audio module (audio feature representation), T represents the audio data, θ A Represents the parameters of the vision module.

[0059] Audio feature extraction uses methods such as short-time Fourier transform (STFT) and Mel-frequency cepstral coefficients (MFCC) to obtain characteristic information such as audio frequency and amplitude. A gating unit (sigmoid activation function layer) is used to learn the importance weight of each modal feature.

[0060] Example 4

[0061] The difference between the method for classifying violent incidents based on multimodal information fusion according to embodiment 1 or 2 is that:

[0062] In step (3), a multimodal feature fusion module based on the self-attention mechanism is constructed to fuse the video and audio features through the self-attention mechanism. The self-attention mechanism can perform weighting between different modalities and dynamically adjust the importance of video and audio features to ensure that important information receives more attention. Specifically, the mechanism calculates the similarity between video and audio features, generates attention weights, and applies these weights to the video and audio features to obtain a weighted fused feature representation.

[0063] The process can be expressed as formula (5):

[0064]

[0065] Among them, F fusion is the feature representation after weighted fusion, F m represents the features from mode m, ω m is the weight of mode m, satisfying

[0066] The self-attention mechanism can dynamically adjust the weights according to the importance of each modality, ensuring that the key information of video and audio features is fully focused during the fusion process.

[0067] Example 5

[0068] The intelligent inspection method for gas storage based on multimodal data fusion according to embodiment 1, 2, 3 or 4 is different in that:

[0069] The specific implementation process of step (4) is as follows:

[0070] The text features extracted in step (3) are represented as F T , image feature representation F V and audio feature representation F A As input, the multimodal fusion module can be expressed as formula (6):

[0071] F R =f F (F V ,F T ,F A θ F ) (6)

[0072] Among them, F R represents the output of the multimodal fusion module, θ F are the parameters of the fusion module.

[0073] To address the heterogeneity and dependencies between modal data, the Transformer model is used to map features from different modalities to the same data domain, resulting in better fusion effects. The Transformer can reduce the differences between modalities and capture the deep dependencies between them.

[0074] The attention mechanism is used to implement the dependency between text, visual, and audio features. For example, the intermediate matrix is ​​generated using weights, and then the visual feature representation with attention to the text content is obtained by calculating the attention coefficient. The calculation process is shown in formula (7):

[0075]

[0076] Similarly, we can also obtain text feature representations with visual content attention. Afterwards, we can use the Transformer model to achieve cross-modal feature fusion.

[0077] Example 6

[0078] The method for classifying violent incidents based on multimodal information fusion according to embodiment 1, 2, 3, 4 or 5 is different in that:

[0079] The specific implementation process of step (5) is as follows:

[0080] A classification model based on a graph neural network (GAT-C) is used to learn and classify the late-stage fused feature vectors. Graph neural networks establish a graph structure between samples, treating the relationships between multimodal data as nodes and edges of the graph, and achieve information sharing and dissemination through the graph aggregation process. Specifically, graph neural networks process the feature information of each node in the graph and its connection relationship with other nodes to achieve deep fusion and dependency learning of multimodal data. The GAT-C model can capture potential dependencies in complex multimodal data and use the graph structure to weight each feature vector, thereby improving the accuracy and robustness of violent incident classification.

[0081] The process can be expressed as formula (8):

[0082]

[0083] Among them, F GAT-C represents the output of the graph neural network (the fused feature vector), is the set of neighbor nodes of node v, α ij is the importance weight of neighbor node j to node i calculated by the attention mechanism, F j is the characteristic of neighbor node j.

[0084] Through the aggregation process of graph neural networks, information between nodes can be effectively transmitted, and ultimately the classification results of violent emergencies can be obtained.

[0085] Example 7

[0086] A violent emergency event classification device based on multimodal information fusion, comprising: a data acquisition and processing module, a feature extraction module, an event classification module, and a result display module;

[0087] The data acquisition and processing module is used to obtain audio and video data of violent incidents through video surveillance equipment, audio acquisition equipment, etc., and perform preprocessing. The feature extraction module is used to use the I3D and VGGish models to extract features of video and audio modal data respectively. The event classification module is used to call the violent incident classification method based on multimodal information fusion, learn and classify the fused features, and identify the type of violent incident. The result display module is used to display the classification results and provide visual analysis.

[0088] Example 8

[0089] A terminal device, characterized in that the terminal device includes a memory, a processor, and a violent emergency event classification program based on multimodal information fusion stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the violent emergency event classification method based on multimodal information fusion as described in any one of claims 1 to 5 are implemented.

[0090] Example 9

[0091] A computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for classifying violent incidents based on multimodal information fusion described in any one of Examples 1-6.

Claims

1. A violent incident classification method based on multimodal information fusion, characterized in that: The method comprises: (1) Obtain video surveillance data and synchronized audio data from the XD-violence public dataset. According to the GA / T1583-2019 Technical Specifications for Public Security Video Surveillance Systems, the collected audio and video data are divided into four categories: physical conflict, armed assault, group riots, and non-violent incidents. A sample dataset of violent incidents represented by multimodal data is constructed and preprocessed. (2) Use the I3D model to extract spatiotemporal features from video data. The I3D model is based on 3D convolution operations and can extract features in both spatial and temporal dimensions from videos, capturing dynamic changes in the video. Use the VGGish model to extract acoustic features from audio data. The convolutional neural network extracts frequency, amplitude, waveform and other information from audio data, effectively capturing key signals in the sound (such as impact sounds, shouting, etc.). (3) Construct a multimodal feature fusion module based on the self-attention mechanism to fuse video and audio features through the self-attention mechanism. The self-attention mechanism can weight different modalities and dynamically adjust the importance of video and audio features to ensure that important information receives more attention. The mechanism calculates the similarity between video and audio features to obtain the weighted fusion feature representation. (4) Based on the early fusion features obtained in step (3), the fused feature vectors are weighted fused using a temporal convolutional network (TCN) model. The TCN model can model the temporal features of video and audio, capture the dynamic changes of violent events in the time series, and obtain a late fusion feature vector that can accurately represent the spatiotemporal characteristics of the violent events. (5) A classification model based on a graph neural network (GAT-C) is used to learn and classify the feature vectors after late fusion; the graph neural network establishes a graph structure between samples, regards the relationship between multimodal data as nodes and edges of the graph, realizes information sharing and dissemination through the aggregation process of the graph, and finally outputs the classification results of violent emergencies, which are specifically classified into physical conflicts, armed attacks, mass riots, non-violent incidents, etc.

2. The method for classifying violent incidents based on multimodal information fusion according to claim 1, characterized in that: In step (1), constructing the relevant data set specifically refers to: A dataset was constructed, and audio and video surveillance data, including video stream data and synchronized audio data, was obtained from the XD-violence public dataset. In terms of data preprocessing, multimodal data with synchronized video and audio was retained, with a sample ratio of 1:1 between video and audio data. Low-quality video frames and noisy audio were removed, and format conversion, frame extraction, and key object detection were performed on the video data. The audio data was also subjected to noise reduction and enhancement. The collected audio and video data were categorized according to the "GA / T 1583-2019 Technical Specifications for Public Security Video Surveillance Systems." Violent incidents were divided into four categories (physical conflict, armed attack, mass riot, and non-violent incidents) based on their behavioral characteristics, degree of harm, and manifestations.

3. The method for classifying violent incidents based on multimodal information fusion according to claim 1, characterized in that: In step (2), multiple modal data are preprocessed, and the implementation process is as follows: The I3D model extracts spatiotemporal features from video modality data. First, the video is segmented into time segments and each frame is preprocessed (e.g., resized to 224×224 pixels). The frame is then fed into the I3D model, which simultaneously extracts spatial and temporal features through 3D convolution operations. The audio data is used to extract acoustic features through the VGGish model; The audio is first converted into a Mel-spectrogram and preprocessed, and then input into the VGGish model. The convolutional layer extracts acoustic features such as frequency and amplitude. Finally, the video and audio features extracted by I3D and VGGish, respectively, are used for subsequent processing.

4. The method for classifying violent incidents based on multimodal information fusion according to claim 1, characterized in that: The specific implementation process of step (3) is as follows: Early fusion of feature vectors of different modalities is performed to construct a multimodal feature fusion module, and the extracted video feature vectors and audio feature vectors are fused. The multimodal feature fusion module uses a self-attention mechanism. The query comes from one modality, while the key and value come from another modality. That is, if Q comes from the video, K and V come from the additional audio, then the attention value calculated using Q and K can be used as a similarity measure between the video and audio, and then the audio features are weighted, and vice versa; cross-fusion operations are performed on the video features and audio features respectively, and the fusion results obtained by the two self-attention multimodal feature fusion modules are weightedly combined to fully capture the spatiotemporal correlation between the audiovisual modalities, thereby obtaining the early feature fusion result.

5. The method for classifying violent incidents based on multimodal information fusion according to claim 1, characterized in that: In step (4), the temporal convolutional network model performs temporal modeling of features through multiple convolutional layers to capture the dynamic changes of violent events. The implementation process is as follows: For the video feature vectors, audio feature vectors, and feature vectors after early fusion obtained in steps 2 and 3, a temporal convolutional network model is constructed, which has 1 input layer, 1 output layer, and 3 hidden layers; the 3 hidden layers contain 512, 256, and 128 neurons respectively, and the output layer contains 256 neurons; at the same time, LeakyReLU is used as the activation function after each hidden layer, and the network is used to learn the three feature vectors separately to obtain feature vectors with a dimension of 256. The three are then weightedly fused to obtain the late fusion result, that is, the final fusion result.

6. The method for classifying violent incidents based on multimodal information fusion according to claim 1, characterized in that: In step (5), the temporal convolutional network model performs temporal modeling of features through multiple convolutional layers to capture the dynamic changes of violent events. The implementation process is as follows: The classifier is built based on the graph attention network (GAT-C), in which the input layer contains 256 neurons and the output layer contains 4 neurons. GELU is used as the activation function of the graph attention layer to obtain a feature vector of dimension 4. Finally, a softmax layer is used to perform classification prediction on the obtained 4-dimensional feature vector to obtain the final classification result.

7. A device for classifying violent incidents based on multimodal information fusion, for implementing the method for classifying violent incidents based on multimodal information fusion according to any one of claims 1 to 5, characterized in that: The device includes: a data acquisition and processing module, a feature extraction module, an event classification module, and a result display module; The data acquisition and processing module is used to obtain audio and video data of violent incidents through video surveillance equipment, audio acquisition equipment, etc., and preprocess this data, including removing low-quality data and noise interference; the feature extraction module is used to use the I3D model and the VGGish model to extract the spatiotemporal features of video modal data and the acoustic features of audio modal data respectively; the event classification module is used to call the violent emergency classification method based on multimodal information fusion, learn and classify the fused features, and identify the specific types of violent emergencies (such as physical conflicts, armed attacks, mass riots, non-violent incidents, etc.); the result display module is used to display the classification results and provide visual analysis for relevant personnel to use in decision-making.

8. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a violent emergency event classification program based on multimodal information fusion stored in the memory and executable on the processor. When the processor executes the violent emergency event classification method based on multimodal information fusion, the steps of the violent emergency event classification method based on multimodal information fusion according to any one of claims 1 to 5 are implemented as follows: The terminal device executes a violent emergency classification method program based on multimodal information fusion through a built-in processor. After obtaining video surveillance and audio data, it uses I3D and VGGish models to extract features of video and audio data, fuses features of different modalities through a self-attention mechanism, and then performs late fusion through a temporal convolutional network. Finally, it performs classification and prediction through a graph neural network classifier, identifies the type of violent incident, and displays the results on the device interface.

9. A computer storage medium, characterized in that The computer storage medium stores a program for a method for classifying violent emergency events based on multimodal information fusion. When the program for classifying violent emergency events based on multimodal information fusion is executed by a processor, the steps of the method for classifying violent emergency events based on multimodal information fusion as described in any one of claims 1 to 5 are implemented.

Images