An intelligent violent behavior detection method and device based on multimodal feature fusion

Abstract

The present invention discloses an intelligent violent behavior detection method and device based on multimodal feature fusion, the method comprising: obtaining image feature data, voice feature data and optical flow feature data to be detected; cross-modally fusing the image feature data and voice feature data to obtain image fusion features and first voice fusion features; cross-modally fusing the optical flow feature data and voice feature data to obtain optical flow fusion features and second voice fusion features; inputting the image fusion features, the first voice fusion features, the optical flow fusion features and the second voice fusion features into a pre-trained multimodal fusion deep neural network model to obtain output violent behavior detection results. The present invention cross-modally fuses the image feature data, voice feature data and optical flow feature data, while retaining independent single-modal information, and performs video violent behavior detection by fusing different modalities, thereby improving the average accuracy of violent behavior detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of video content analysis, and in particular relates to an intelligent violent behavior detection method and device based on multimodal feature fusion. Background Art

[0002] In recent years, with the continuous development of Internet technology and the help of video big data technology, various videos have begun to appear in daily work and life, resulting in an exponential growth in the number of videos. While bringing great convenience to people and promoting social development, these videos are also full of violent, vulgar, and terrifying content, which has a negative impact on the physical and mental growth of immature minors. At present, how to efficiently and accurately identify these bad elements and protect the physical and mental health of minors has become an urgent issue. In addition, to find out the bad elements contained in massive videos, it is obviously very time-consuming and labor-intensive to rely solely on traditional human content review methods. Therefore, an intelligent analysis system with the function of detecting specific video content came into being, focusing on the research of automatic detection technology of video violence. This technology can not only be used for Internet video content review, but also can be applied to video surveillance of sensitive places in intelligent security.

[0003] So far, video violence detection methods still rely on processing a single mode, namely audio or video. Although recent studies have considered multimodal methods, most of them simply fuse video and audio emotion information at the feature level, ignoring the interactive information between different modes and the advantages of a single mode. In most cases, visual information, as an intuitive signal, can accurately identify and locate events, but in most scenarios, audio signals can distinguish visually ambiguous events. Therefore, these illustrate the importance and feasibility of developing a multimodal method. By studying a multimodal video violence detection method, we can cope with changing complex environments and improve the generalization ability and average detection accuracy of violence detection methods. Summary of the invention

[0004] In view of the above problems, the present invention proposes an intelligent violent behavior detection method, device and system based on multimodal feature fusion.

[0005] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides an intelligent violent behavior detection method based on multimodal feature fusion, comprising:

[0007] Acquire image feature data, speech feature data and optical flow feature data to be detected;

[0008] Cross-modally fusing the image feature data and the speech feature data to obtain an image fusion feature and a first speech fusion feature;

[0009] Cross-modally fuse the optical flow feature data and the speech feature data to obtain an optical flow fusion feature and a second speech fusion feature;

[0010] Inputting the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multi-modal fusion deep neural network model to obtain an output violent behavior detection result;

[0011] The training method of the multimodal fusion deep neural network model includes: training the multimodal fusion deep neural network model using labeled image feature data, speech feature data and optical flow feature data, and minimizing the loss function To optimize the model parameters and obtain a trained multimodal fusion deep neural network model.

[0012] In some embodiments, the acquiring of the image feature data, speech feature data and optical flow feature data to be detected includes:

[0013] Using the I3D network as an extractor of image data, image frame features are extracted to obtain image feature data to be detected;

[0014] Use the VGGish network as an extractor of speech data to extract speech data features and obtain the speech feature data to be detected;

[0015] The I3D network is used as the extractor of optical flow data, and the TV-L1 algorithm is used to calculate the optical flow map on the GPU to obtain the optical flow feature data to be detected.

[0016] In some embodiments, cross-modal fusion of the image feature data and the speech feature data includes:

[0017] The image feature data and the speech feature data are concatenated to obtain fusion features;

[0018] Calculating the correlation of the fused features through a cross-modal attention mechanism layer to obtain an output fused feature based on the attention mechanism;

[0019] The output fusion features based on the attention mechanism are split to obtain image fusion features and first speech fusion features.

[0020] In some embodiments, cross-modal fusion of the optical flow feature data and the speech feature data includes:

[0021] The optical flow feature data and the speech feature data are concatenated to obtain fusion features;

[0022] Calculating the correlation of the fused features through a cross-modal attention mechanism layer to obtain an output fused feature based on the attention mechanism;

[0023] The output fusion features based on the attention mechanism are split to obtain the optical flow fusion features and the second speech fusion features.

[0024] In some embodiments, the multimodal fusion deep neural network model includes three multimodal fusion modules and a fully connected layer connected in sequence; the multimodal fusion layer includes a cross-modal module, a bidirectional long short-term memory recursive neural network layer and a pooling layer; the cross-modal module includes a cross-modal attention layer and a graph convolution layer; the cross-modal attention layer includes a linear layer, a Softmax layer, a normalization layer and a linear layer connected in sequence.

[0025] In some embodiments, the cross-modal module is represented as:

[0026]

[0027] M i =Linear(M q )

[0028] M s =Softmax(M i , dim=1)

[0029] M n =Norm(M s , dim=2)

[0030] Out=Linear(M n )

[0031] Out′=GCN(Out)

[0032] Out1, Out2 = Split(Out′)

[0033] Among them, M1 and M2 represent the feature vectors of the two inputs of the cross-modal module, represents the concatenation of two feature vectors in the first dimension; Linear, Softmax, Norm and GCN represent the linear layer, Softmax layer, normalization layer and GCN layer respectively, M q 、M i are the input and output of the first linear layer, M s 、M nThey are the outputs of the Softmax layer and the normalization layer respectively, Out is the output of the second linear layer, and Out′ is the output of the GCN layer; Split represents the splitting operation of the matrix vector; Out1 and Out2 represent the two outputs of the cross-modal module respectively.

[0034] In some embodiments, the loss function It is expressed as:

[0035]

[0036] Among them, y i is the discrete distribution of real violent events, y′ i is the discrete distribution of the output video label prediction value of the multimodal feature fusion model, and N is the number of input samples.

[0037] In a second aspect, the present invention provides an intelligent violent behavior detection device based on multimodal feature fusion, comprising:

[0038] A data preprocessing module, used to obtain image feature data, speech feature data and optical flow feature data to be detected;

[0039] A feature fusion module, used to perform cross-modal fusion of image feature data and speech feature data, and to perform cross-modal fusion of optical flow feature data and speech feature data to obtain image fusion features, first speech fusion features, optical flow fusion features, and second speech fusion features;

[0040] The violent behavior detection module is used to input the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multi-modal fusion deep neural network model to obtain an output violent behavior detection result.

[0041] In a third aspect, the present invention provides an intelligent violent behavior detection system based on multimodal feature fusion, comprising: a storage medium and a processor;

[0042] The storage medium is used to store instructions;

[0043] The processor is used to operate according to the instructions to execute the steps of any one of the methods described in the first aspect.

[0044] According to a fourth aspect, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention proposes an intelligent violent behavior detection method based on multimodal feature fusion, and proposes a multimodal fusion deep neural network model for jointly optimizing multimodal fusion and violent behavior detection, which provides a significant improvement over the traditional multimodal violent behavior detection method. The present invention can effectively extract audio, optical flow and image frame information in violent behavior events. The present invention utilizes a cross-modal module to combine different modal features and model the relationship between different modalities to generate video features with stronger distinguishing power. The present invention effectively cross-modally fuses image feature data, speech feature data and optical flow feature data while retaining independent single modal information, forming speech fusion features, visual fusion features and optical flow fusion features. Video violent events are detected through the three fusion features, which improves the average accuracy of violent behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments and in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 It is a flow chart of an intelligent violent behavior detection method based on multimodal feature fusion of the present invention;

[0049] Figure 2 It is a framework diagram of an intelligent violent behavior detection method based on multimodal feature fusion of the present invention;

[0050] Figure 3 It is a schematic diagram of a cross-modal model used in an intelligent violent behavior detection method based on multimodal feature fusion of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0052] The application principle of the present invention is described in detail below in conjunction with the accompanying drawings.

[0053] In the description of the present invention, "several" means more than one, "many" means more than two, "greater than", "less than", "exceed" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0054] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0055] Example 1

[0056] An intelligent violent behavior detection method based on multimodal feature fusion, comprising:

[0057] Acquire image feature data, speech feature data and optical flow feature data to be detected;

[0058] Cross-modally fusing the image feature data and the speech feature data to obtain an image fusion feature and a first speech fusion feature;

[0059] Cross-modally fuse the optical flow feature data and the speech feature data to obtain an optical flow fusion feature and a second speech fusion feature;

[0060] Inputting the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multi-modal fusion deep neural network model to obtain an output violent behavior detection result;

[0061] The training method of the multimodal fusion deep neural network model includes: training the multimodal fusion deep neural network model using labeled image feature data, speech feature data and optical flow feature data, and minimizing the loss function To optimize the model parameters and obtain a trained multimodal fusion deep neural network model.

[0062] The present invention provides an intelligent violent behavior detection method based on multimodal feature fusion. Figure 2 As shown, the specific steps include:

[0063] Step 1: Obtain the image feature data, speech feature data and optical flow feature data to be detected, as described below:

[0064] We define the multimodal data samples and the corresponding labels as I = (V, A, O) and y, where V is the untrimmed image frame, A is the speech, O is the optical flow, and y∈{0,1}, y=1 indicates that I covers violent events.

[0065] Step 1.1: Using the I3D network as an extractor F for the image frame data VV , extract the image frame features and obtain the image feature data to be detected Where C and T are the number of channels and the length of the image frame sequence respectively.

[0066] Step 1.2: Use the VGGish network as an extractor F for speech data A A , extract the voice data features and obtain the voice feature data to be detected Where C and T are the number of channels and the length of the speech sequence respectively.

[0067] Step 1.3: Use the I3D network as an extractor F for the optical flow data O O , and use the TV-L1 algorithm to calculate the optical flow map on the GPU to obtain the optical flow feature data to be detected Where C and T are the number of channels and the length of the optical flow frame sequence respectively.

[0068] Step 2: Transform the image feature data X V and speech feature data X A Perform cross-modal fusion, such as Figure 3 The specific description is as follows:

[0069] Step 2.1: Transform the image feature data X V and speech feature data X A Perform stitching to obtain fused data

[0070] Step 2.2: Calculate the correlation of the fused data through the cross-modal attention mechanism layer to obtain the output fusion feature based on the attention mechanism

[0071] Step 2.3: Fuse the output of the attention mechanism with feature X M Split and obtain image fusion features And the first voice fusion feature

[0072] Step 3: Transform the optical flow feature data X O and speech feature data X A Perform cross-modal fusion, such as Figure 3 The specific description is as follows:

[0073] Step 3.1: Transform the optical flow feature data X O and speech feature data X A Perform stitching to obtain fused data

[0074] Step 3.2: Calculate the correlation of the fused data through the cross-modal attention mechanism layer to obtain the output fusion feature based on the attention mechanism

[0075] Step 3.3: Fuse the output of the attention mechanism with feature X M Split and obtain optical flow fusion features Fusion features with the second speech

[0076] Step 4: Build a multimodal fusion deep neural network model, such as Figure 2 As shown in the figure, it includes three multimodal fusion modules (MFB) and a fully connected layer (FC) connected in sequence, where the output channel parameters of the three multimodal fusion blocks are set to 128, 64, and 32 respectively. Each multimodal fusion module also includes multiple cross-modal modules, a bidirectional long short-term memory recurrent neural network layer (BiGRU) and a pooling layer (Pool). Figure 3 As shown in the figure, each cross-modal module includes a cross-modal attention layer (Cross-Modal Attention) and a graph convolution layer (GCN). The cross-modal attention layer includes a linear layer (Linear), a Softmax layer, a normalization layer (Norm) and a linear layer (Linear) connected in sequence. The cross-modal module accepts two inputs: feature maps and feature map Where T and C are the number of channels and sequence length respectively. The cross-modal module can be formulated by the following steps:

[0077]

[0078] M i =Linear(M q ) (2)

[0079] M s =Softmax(M i ,dim=1) (3)

[0080] M n =Norm(M s ,dim=2) (4)

[0081] Out=Linear(M n ) (5)

[0082] Out′=GCN(Out) (6)

[0083] Out1, Out2=Split(Out′) (7)

[0084] Among them, M1 and M2 represent the feature vectors of the two inputs of the cross-modal module, represents the concatenation of two feature vectors in the first dimension; Linear, Softmax, Norm and GCN represent the linear layer, Softmax layer, normalization layer and GCN layer respectively, M q 、M i are the input and output of the first linear layer, M s 、M n They are the outputs of the Softmax layer and the normalization layer respectively, Out is the output of the second linear layer, and Out′ is the output of the GCN layer; Split represents the splitting operation of the matrix vector; Out1 and Out2 represent the two outputs of the cross-modal module respectively.

[0085] Assuming that the graph convolution layer (GCN) has l output channels and k input channels, it is necessary to use kl convolution operations to convert the number of channels. The graph convolution operation formula is:

[0086]

[0087] in Represents the feature vector of k input channels; Represents the feature vector of the lth output channel; represents the two-dimensional convolution kernel of the Kth row and the Lth column; A represents the connection matrix between data nodes, which is used to determine whether there is a connection between two nodes vi and vj and the strength of the connection. Here, the normalized embedded Gaussian equation is used to measure the correlation between two nodes in space:

[0088]

[0089] Because the normalized Gaussian and softmax operations are equivalent, formula (9) is equivalent to formula (10):

[0090]

[0091] Among them A ij Represents node v i and node v j The correlation between them.

[0092] After each graph convolution operation, there is an additional ReLU operation, the purpose of which is to add nonlinearity to the graph convolution. Because the real-world problems solved using graph convolution are nonlinear, and the convolution operation is a linear operation, an activation function such as ReLU must be used to add nonlinear properties.

[0093] BiGRU layer: In the BiGRU layer, a bidirectional long short-term memory recurrent neural network layer (BiGRU) is used to extract the temporal information of violent videos from a global perspective. The formula function is: X = BiGRU (x). Where x is the input vector of the BiGRU layer, and X is the output vector, which is used as the input of the next layer.

[0094] Pooling layer: compresses the input feature map, making it smaller and simplifying the network calculation complexity; on the other hand, it compresses the features and extracts the main features. The pooling layer can reduce the dimension of the feature map while maintaining the most important information.

[0095] Step 5, the training method of the multimodal fusion deep neural network model, the specific steps are as follows:

[0096] Step 5.1, randomly initialize the parameters and weight values ​​of all multimodal fusion deep neural network models;

[0097] Step 5.2: Take the labeled image feature data, speech feature data, and optical flow feature data as the input of the multimodal fusion deep neural network model, and after the forward propagation step, finally reach the fully connected layer FC for detection to obtain the violent behavior detection result, that is, output a vector containing the predicted probability value. Since the weights are randomly assigned to the first training example, the output probability is also random;

[0098] Step 5.3, calculate the loss function of the fully connected layer FC As shown in formula (11), the cross entropy loss function is adopted, which is defined as follows:

[0099]

[0100] Among them, y i is the discrete distribution of real violent events. i is the discrete distribution of the output video label prediction value of the multimodal feature fusion model, and N is the number of input samples. Represents the loss function, which is used to evaluate the accuracy of the model's estimation of the true probability distribution. It can be achieved by minimizing the loss function To optimize the model and update all network parameters.

[0101] In step 5.4, back propagation is used to calculate the error gradient of all weights in the network. And all filter values, weights, and parameter values ​​are updated using gradient descent to minimize the output loss, that is, the value of the loss function is minimized. The weights are adjusted according to their contribution to the loss. When the same skeleton data is input again, the output probability may be closer to the target vector. This means that the network has learned to correctly classify this particular skeleton by adjusting its weights and filters, thereby reducing the output loss. Parameters such as the number of filters, filter size, and network structure have been fixed before step 5.1 and will not change during training. Only the filter matrix and connection weights are updated.

[0102] Step 5.5, repeat steps 5.2-5.4 for all skeleton data in the training set until the number of training reaches the set number of iterations. After completing the above steps, the training set data is trained and learned through the constructed spatiotemporal attention convolutional neural network, which actually means that all weights and parameters of the multimodal fusion deep neural network model have been optimized to correctly detect violent incidents.

[0103] Step 6: Use the trained multimodal fusion deep neural network model to identify the test samples and output the results of violent behavior detection.

[0104] In order to verify the advantages of the multimodal fusion deep neural network model, this experiment compared it with several publicly available baseline methods on the XD-Violence multimodal violence detection dataset, and selected the average precision (AP) as the experimental evaluation indicator.

[0105] Table 1 shows the comparison results of the intelligent violent behavior detection method based on multimodal feature fusion proposed in this embodiment and other baseline methods on the XD-Violence dataset. Here, the multimodal fusion deep neural network model is denoted as Ours. The intelligent violent behavior detection method based on multimodal feature fusion proposed in this embodiment connects the two stages (multimodal modeling and detection network) and jointly optimizes them to generate video features with stronger discriminative power. The feature fusion module based on the attention mechanism is used to interactively learn different modal fusion features, thereby obtaining better experimental results.

[0106] Table 1 Recognition results of the method of the present invention and other public methods on the XD-Violence dataset

[0107]

[0108]

[0109] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0110] Example 2

[0111] In an embodiment of the present invention, an intelligent violent behavior detection device based on multimodal feature fusion is provided, comprising the following modules:

[0112] A data preprocessing module, used to obtain image feature data, speech feature data and optical flow feature data to be detected;

[0113] A feature fusion module, used to perform cross-modal fusion of image feature data and speech feature data, and to perform cross-modal fusion of optical flow feature data and speech feature data to obtain image fusion features, first speech fusion features, optical flow fusion features, and second speech fusion features;

[0114] The violent behavior detection module is used to input the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multi-modal fusion deep neural network model to obtain an output violent behavior detection result.

[0115] Example 3

[0116] Based on the same inventive concept as that of Example 1, an intelligent violent behavior detection device based on multimodal feature fusion is provided in an embodiment of the present invention, including: a storage medium and a processor;

[0117] The storage medium is used to store instructions;

[0118] The processor is configured to operate according to the instructions to execute the steps of the method in the first aspect.

[0119] Example 4

[0120] A storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0121] In summary, the intelligent violent behavior detection method based on multimodal feature fusion proposed in the present invention solves the problem of multimodal data fusion in the process of video violent behavior detection. It extracts audio, optical flow and image frame features through deep learning technology, and then constructs a multimodal fusion deep neural network model to calculate multimodal fusion features and single modal features. This method can fuse multiple modes in the video and improve the accuracy of violence detection, with an accuracy rate of up to 80.59%.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent violent behavior detection method based on multimodal feature fusion, characterized in that: include: Acquire image feature data, speech feature data and optical flow feature data to be detected; Cross-modally fusing the image feature data and the speech feature data to obtain an image fusion feature and a first speech fusion feature, including: splicing the image feature data and the speech feature data to obtain a fusion feature; calculating the correlation of the fusion feature through a cross-modal attention mechanism layer to obtain an output fusion feature based on the attention mechanism; splitting the output fusion feature based on the attention mechanism to obtain an image fusion feature and a first speech fusion feature; Cross-modally fusing the optical flow feature data and the speech feature data to obtain an optical flow fusion feature and a second speech fusion feature, including: splicing the optical flow feature data and the speech feature data to obtain a fusion feature; calculating the correlation of the fusion feature through a cross-modal attention mechanism layer to obtain an output fusion feature based on the attention mechanism; splitting the output fusion feature based on the attention mechanism to obtain an optical flow fusion feature and a second speech fusion feature; Inputting the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multi-modal fusion deep neural network model to obtain an output violent behavior detection result; The training method of the multimodal fusion deep neural network model includes: training the multimodal fusion deep neural network model using labeled image feature data, speech feature data and optical flow feature data, and minimizing the loss function To optimize the model parameters and obtain a trained multimodal fusion deep neural network model.

2. The intelligent violent behavior detection method based on multimodal feature fusion according to claim 1 is characterized in that: The acquisition of the image feature data, speech feature data and optical flow feature data to be detected comprises: Using the I3D network as an extractor of image data, image frame features are extracted to obtain image feature data to be detected; Use the VGGish network as an extractor of speech data to extract speech data features and obtain the speech feature data to be detected; The I3D network is used as the extractor of optical flow data, and the TV-L1 algorithm is used to calculate the optical flow map on the GPU to obtain the optical flow feature data to be detected.

3. The intelligent violent behavior detection method based on multimodal feature fusion according to claim 1 is characterized in that: The multimodal fusion deep neural network model includes three multimodal fusion modules and a fully connected layer connected in sequence; the multimodal fusion module includes a cross-modal module, a bidirectional long short-term memory recursive neural network layer and a pooling layer; the cross-modal module includes a cross-modal attention layer and a graph convolution layer; the cross-modal attention layer includes a linear layer, a Softmax layer, a normalization layer and a linear layer connected in sequence.

4. The intelligent violent behavior detection method based on multimodal feature fusion according to claim 3 is characterized in that: The cross-modal module is represented as: M i =Linear(M q ) M s =Softmax(M i ,dim=1) M n =Norm(M s ,dim=2) Out=Linear(M n ) Out′=GCN(Out) Out1, Out2 = Split(Out′) Among them, M1 and M2 represent the feature vectors of the two inputs of the cross-modal module, represents the concatenation of two feature vectors in the first dimension; Linear, Softmax, Norm and GCN represent the linear layer, Softmax layer, normalization layer and GCN layer respectively, M q 、M i are the input and output of the first linear layer, M s 、M n They are the outputs of the Softmax layer and the normalization layer respectively, Out is the output of the second linear layer, and Out′ is the output of the GCN layer; Split represents the splitting operation of the matrix vector; Out1 and Out2 represent the two outputs of the cross-modal module respectively.

5. The intelligent violent behavior detection method based on multimodal feature fusion according to claim 4 is characterized in that: Loss Function It is expressed as: Among them, y i is the discrete distribution of real violent events, y i ′ is the discrete distribution of the output video label prediction value of the multimodal feature fusion model, and N is the number of input samples.

6. An intelligent violent behavior detection device based on multimodal feature fusion, characterized in that: include: A data preprocessing module, used to obtain image feature data, speech feature data and optical flow feature data to be detected; A feature fusion module, used to perform cross-modal fusion of image feature data and speech feature data, and to perform cross-modal fusion of optical flow feature data and speech feature data to obtain image fusion features, first speech fusion features, optical flow fusion features, and second speech fusion features; Specifically used for: splicing image feature data and speech feature data to obtain fusion features; Calculating the correlation of the fusion features through a cross-modal attention mechanism layer to obtain an output fusion feature based on the attention mechanism; splitting the output fusion feature based on the attention mechanism to obtain an image fusion feature and a first speech fusion feature; splicing the optical flow feature data and the speech feature data to obtain a fusion feature; Calculating the correlation of the fused features through a cross-modal attention mechanism layer to obtain an output fused feature based on the attention mechanism; The output fusion features based on the attention mechanism are split to obtain the optical flow fusion features and the second speech fusion features; The violent behavior detection module is used to input the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multi-modal fusion deep neural network model to obtain an output violent behavior detection result.

7. An intelligent violent behavior detection device based on multimodal feature fusion, characterized in that: include: Storage media and processors; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images