Multimodal information fusion video violence detection methods, systems and equipment

Abstract

This invention provides a method, system, and device for detecting violent behavior in videos using multimodal information fusion, relating to the field of computer vision. The method includes: acquiring videos from different sources and angles, segmenting the videos to generate video clips; using a weakly supervised labeling method to label video clips containing violent behavior as violent videos, creating weak labels; inputting the video to be detected into a violence detection model trained based on the weakly labeled video clips to identify violent behavior in the video; the weakly supervised video violence detection model includes a three-branch multimodal information optimizer combining graph convolution, multi-scale mechanisms, and a graph LSTM structure; the three-branch multimodal information optimizer extracts RGB features, optical flow features, and audio features from the video clips. This invention can reduce the difficulty and cost of sample collection and achieve multimodal information fusion.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, system, and device for detecting violent behavior in videos using multimodal information fusion. Background Technology

[0002] Violent behavior refers to fighting, brawling, or other negative acts that further threaten personal safety. Reducing violent crime and maintaining public safety are key concerns in the field of public security. Violent behavior detection in video refers to the use of image processing and computer vision technologies to automatically and intelligently analyze video content, accurately and quickly detect violent behavior in monitored scenes, and issue warnings to assist relevant personnel in quickly developing intervention plans and preventing dangerous situations from occurring.

[0003] Research on deep learning in the field of violence monitoring is still in its early stages, and the main challenges include:

[0004] (1) High-precision violent behavior video samples are scarce. Existing methods require marking each frame of the video to indicate whether violent behavior exists, specifying a specific number of frames. That is, video behavior monitoring requires detailed marking of each frame, which makes sample collection difficult and costly.

[0005] (2) The joint learning and mining of multimodal information is insufficient. Existing video analysis technology does not make full use of the temporal consistency and behavioral consistency of audio and visual features when the behavior occurs. Summary of the Invention

[0006] The purpose of this invention is to provide a method, system, and device for detecting violent behavior in videos by fusing multimodal information, so as to solve the problems of scarce high-precision video samples and insufficient joint learning of multimodal features.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A video violence detection method based on multimodal information fusion includes:

[0009] Acquire videos from different sources and from different angles, divide the videos, and generate video clips;

[0010] Weakly supervised labeling is used to mark video clips containing violent behavior as violent videos, thus creating weak tags;

[0011] The video to be detected is input into the violence detection model to identify violent behaviors in the video. The violence detection model is obtained by training a weakly supervised video violence detection model based on video segments with weak labels. The weakly supervised video violence detection model includes a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanism and graph LSTM structure. The three-branch multimodal information optimizer is used to extract RGB features, optical flow features and audio features from the video segments.

[0012] Optionally, the three-branch multimodal information optimizer specifically includes: an RGB feature optimizer, an optical flow feature optimizer, and an audio feature optimizer;

[0013] When the three-branch multimodal information optimizer extracts the RGB features, the I3D network is used to extract the RGB features and input them into the RGB feature optimizer;

[0014] The RGB feature optimizer includes four sequentially connected multi-scale convolutional layers and a graph convolutional structure. The four multi-scale convolutional layers include a 1×1 convolution that does not consider the neighborhood and integrates all time series, a 3×3 regular convolution that considers the neighborhood and integrates all time series, a 3×3 convolution that considers the neighborhood and is spaced 5 units apart on the time series, and a 3×3 convolution that considers the neighborhood and is spaced 9 units apart on the time series. The four multi-scale convolutional layers are used to determine the temporal boundaries of the RGB features indicating the presence of the violent behavior, highlighting the video anomaly features when the violent behavior exists. The graph convolution is used to receive the RGB features processed by the four multi-scale convolutional layers and reduce the scale of the RGB features.

[0015] Optionally, when the three-branch multimodal information optimizer extracts the optical flow features, an I3D network is used to extract the optical flow features and input them into the graph convolutional structure in the optical flow feature optimizer; the execution process of the graph convolutional structure is as follows: in, The features of node i in layer l+1; j is the neighboring node of node i; d i d is the degree of node i; j Let W be the degree of node j. l Let be the weight parameter matrix of the l-th layer, and σ be the nonlinear activation function.

[0016] Optionally, when the three-branch multimodal information optimizer extracts the audio features, the audio features are extracted through a VGGish network and input into the audio feature optimizer; the audio feature optimizer is a graph LSTM structure.

[0017] The graph LSTM structure includes input gate I. i Forgotten Gate iAdaptive forget gate Memory Gate R i and output gate O i The loop process of each gate in the graph LSTM structure is as follows:

[0018]

[0019] f i =δ(W f L i,t+1 +U f h i,t +b f ),

[0020]

[0021]

[0022]

[0023] Where i is the node; t is the layer number; δ represents the sigmoid function; W u L is the input gate weight matrix; i,t+1 The input node state; U u h represents the hidden state weight parameters of the input gate node. i,t U represents the hidden state of node i at layer t. In b represents the state weight parameters of the adjacent nodes of the input gate; u W is the input gate constant variable; f U is the forget gate weight matrix; f b represents the hidden state weight parameter for the forget gate node; f U is the forget gate constant variable; fn The state weight parameters for the adjacent nodes of the forget gate; Adaptive forget gate constant variable; W R U is the memory gate weight matrix; R U represents the hidden state weight parameter of the memory gate node; Rn The state weight parameters of the adjacent nodes of the memory gate; b represents the hidden state of the neighboring nodes of node i at level t; R W is the memory gate constant variable. o U is the output gate weight matrix; o U represents the hidden state weight parameters of the output gate node; on b represents the state weight parameters of the adjacent nodes of the output gate; o This is the output gate constant variable.

[0024] Optionally, the hidden state and storage state of the output nodes of the graph LSTM structure can be updated according to the five gating points;

[0025] The update process is as follows:

[0026]

[0027] h i,t+1 =O i ⊙tanh(C i,t+1 )

[0028] Among them, C i,t+1 The output node's storage state is represented by ⊙; the node-by-node product is represented by ⊙; N(i) is the number of adjacent nodes; h i,t+1 This represents the hidden state of the output node.

[0029] Optionally, the weakly supervised video violence detection model further includes:

[0030] The RGB features, optical flow features, and audio features are summed to determine the summed features;

[0031] The summed features are sequentially passed through a 1×1 convolution and a GELU activation function to determine the feature weights;

[0032] The product features are determined by multiplying the feature weights with the summed features.

[0033] The product feature and the summation feature are superimposed and output to a fully connected layer to determine the feature with a scale of Batch×2C; Batch is the batch size and C is the length of the feature time series.

[0034] The Batch×2C features are divided into two Batch×C features; one Batch×C feature is used to output the frame-by-frame result, and the other Batch×C feature is convolved again with 1×1 to output the global result. The difference between the two Batch×C features is used to characterize the frame-by-frame loss.

[0035] A multimodal information fusion video violence detection system includes:

[0036] The video clip generation module is used to acquire videos from different sources and angles, divide the videos, and generate video clips;

[0037] The weak labeling module is used to label video clips containing violent behavior as violent videos using a weak supervised labeling method, thus creating weak labels.

[0038] A violence behavior recognition module is used to input the video to be detected into a violence behavior detection model to identify violent behaviors in the video. The violence behavior detection model is obtained by training a weakly supervised video violence behavior detection model based on video segments with weak labels. The weakly supervised video violence behavior detection model includes a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanisms, and graph LSTM structure. The three-branch multimodal information optimizer is used to extract RGB features, optical flow features, and audio features from the video segments.

[0039] Optionally, the three-branch multimodal information optimizer specifically includes: an RGB feature optimizer, an optical flow feature optimizer, and an audio feature optimizer;

[0040] When the three-branch multimodal information optimizer extracts the RGB features, the I3D network is used to extract the RGB features and input them into the RGB feature optimizer;

[0041] The RGB feature optimizer includes four sequentially connected multi-scale convolutional layers and a graph convolutional structure. The four multi-scale convolutional layers include a 1×1 convolution that does not consider the neighborhood and integrates all time series, a 3×3 regular convolution that considers the neighborhood and integrates all time series, a 3×3 convolution that considers the neighborhood and is spaced 5 units apart on the time series, and a 3×3 convolution that considers the neighborhood and is spaced 9 units apart on the time series. The four multi-scale convolutional layers are used to determine the temporal boundaries of the RGB features indicating the presence of the violent behavior, highlighting the video anomaly features when the violent behavior exists. The graph convolution is used to receive the RGB features processed by the four multi-scale convolutional layers and reduce the scale of the RGB features.

[0042] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the aforementioned multimodal information fusion video violence detection method.

[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for detecting video violence through multimodal information fusion.

[0044] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides a method, system and device for detecting violent behavior in videos by multimodal information fusion. The present invention only labels whether violent behavior exists in the entire video segment, adopts a weakly supervised labeling method, adaptively learns global overall information and local proximity information, realizes the determination of the existence of violent behavior in the entire video segment (global) and the monitoring of violent behavior frame by frame, and reduces the difficulty and cost of sample collection.

[0045] This invention employs a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanisms, and graph LSTM structure to extract RGB features, optical flow features, and audio features from videos, thereby achieving the fusion of multimodal information. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the video violence detection method based on multimodal information fusion provided by the present invention;

[0048] Figure 2 This is a structural diagram of the weakly supervised video violence detection model provided by the present invention;

[0049] Figure 3 This is an example diagram of the detection results of the present invention; Figure 3 (a) is the first screenshot from the subway station surveillance camera showing violent behavior; Figure 3 (b) is the second screenshot from the subway station surveillance camera showing violent behavior; Figure 3 (c) is the first screenshot showing violence in a close-up scene; Figure 3 (d) is the second screenshot showing violent behavior in a close-up scene. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] The purpose of this invention is to provide a method, system, and device for detecting violent behavior in videos by fusing multimodal information, which can reduce the difficulty and cost of sample collection and achieve multimodal information fusion.

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1

[0054] This invention provides a video violence detection method based on multimodal information fusion, comprising:

[0055] Videos from different sources and angles are acquired and divided to generate video clips.

[0056] Weakly supervised labeling is used to mark video clips containing violent behavior as violent videos, thus creating weak labels.

[0057] In practical applications, a violent behavior training dataset is created using publicly available abnormal behavior datasets such as UCF-Crime, UCF-Fight, XD, and some movie clips. The dataset is then segmented into video clips of less than 3 minutes by video editing. Each video clip is manually reviewed to determine whether violent behavior exists. If violent behavior is found in a video, it is marked as a violent video. The subject of the behavior, the specific time of occurrence, and the duration are not marked.

[0058] The video to be detected is input into the violence detection model to identify violent behaviors in the video. The violence detection model is obtained by training a weakly supervised video violence detection model based on video segments with weak labels. The weakly supervised video violence detection model includes a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanism and graph LSTM structure. The three-branch multimodal information optimizer is used to extract RGB features, optical flow features and audio features from the video segments.

[0059] In practical applications, the multimodal information fusion video violence detection method provided by this invention is as follows: Figure 1 As shown.

[0060] Step 101: Obtain videos from different sources and angles, such as surveillance footage or film clips.

[0061] Step 102: Create a training dataset of weakly labeled violent behavior: Divide the video into video segments of less than 3 minutes and manually label the segments containing violent behavior.

[0062] Step 103: Design a multimodal information fusion video violence behavior detection model.

[0063] Step 104: Input the violent behavior training dataset into the model, iteratively optimize the model according to the loss function, and save the model with the highest detection accuracy as the violent behavior detection model.

[0064] Step 105: Input the video to be detected into the violence detection model to identify whether there is violence in the video.

[0065] like Figure 2As shown, the weakly supervised video violence detection model includes a three-branch feature extractor, which extracts optical flow, RGB and audio features from the video respectively. Then, feature optimization and fusion are used to achieve unified feature representation and finally output the detection result. The main structures include I3D, VGGish, graph convolution, multi-scale mechanism, graph LSTM structure, etc. Figure 2 In the diagram, Batch × number × C represents the feature map scale, Batch is the batch size of the model input, number is the feature scale, C is the length of the feature time series, x1-x4 after FLOW represents the input nodes, F1-F4 represents the output nodes, and x1-x9 after Audio represents the nodes.

[0066] In this embodiment, when training and predicting video segments, Batch = 128 and C = the number of frames in each input video segment. When predicting in real time, Batch = 1 and C = 20, that is, the 20 frames before the current time are taken as the data source.

[0067] In practical applications, the three-branch multimodal information optimizer specifically includes: an RGB feature optimizer, an optical flow feature optimizer, and an audio feature optimizer;

[0068] When the three-branch multimodal information optimizer extracts the RGB features, the I3D network is used to extract the RGB features and input them into the RGB feature optimizer;

[0069] The RGB feature optimizer comprises four sequentially connected multi-scale convolutional layers and a graph convolutional structure. The four multi-scale convolutional layers include a 1×1 convolution that does not consider the neighborhood and integrates all time series, a 3×3 regular convolution that considers the neighborhood and integrates all time series (stride s = 1), a 3×3 convolution that considers the neighborhood and time series with a 5-unit interval (stride s = 5), and a 3×3 convolution that considers the neighborhood and time series with a 9-unit interval (stride s = 9). The four multi-scale convolutional layers are used to determine the temporal boundaries of the RGB features indicating the presence of the violent behavior, highlighting the video's abnormal features when the violent behavior exists. The graph convolution is used to receive the RGB features processed by the four multi-scale convolutional layers and reduce the scale of the RGB features.

[0070] This four-layer multi-scale convolutional structure helps the model find the temporal boundaries where violent behavior features exist, highlighting the abnormal features of the video when the behavior exists. This structure does not change the feature scale. The output results are superimposed on the input to the graph convolution, reducing the scale from 1024 to 32. This graph convolutional structure is consistent with the graph convolutional structure in the optical flow feature optimizer.

[0071] In practical applications, when the three-branch multimodal information optimizer extracts the optical flow features, an I3D network is used to extract the optical flow features and input them into the graph convolutional structure in the optical flow feature optimizer; the execution process of the graph convolutional structure is as follows: in, The features of node i in layer l+1; j is the neighboring node of node i; d i d is the degree of node i; j Let W be the degree of node j. l Let be the weight parameter matrix of the l-th layer, and σ be the nonlinear activation function.

[0072] In practical applications, when the three-branch multimodal information optimizer extracts the audio features, the audio features are extracted through the VGGish network and input into the audio feature optimizer; the audio feature optimizer is a graph LSTM structure.

[0073] The graph LSTM structure includes input gate I. i Forgotten Gate i Adaptive forget gate Memory Gate R i and output gate O i The loop process of each gate in the graph LSTM structure is as follows:

[0074]

[0075] f i =δ(W f L i,t+1 +U f h i,t +b f ),

[0076]

[0077]

[0078]

[0079] Where i is the node; t is the layer number; δ represents the sigmoid function; W u L is the input gate weight matrix; i,t+1 The input node state; U u h represents the hidden state weight parameters of the input gate node. i,t U represents the hidden state of node i at layer t. In b represents the state weight parameters of the adjacent nodes of the input gate; u W is the input gate constant variable; f U is the forget gate weight matrix; fb represents the hidden state weight parameter for the forget gate node; f U is the forget gate constant variable; fu The state weight parameters for the adjacent nodes of the forget gate; Adaptive forget gate constant variable; W R U is the memory gate weight matrix; R U represents the hidden state weight parameter of the memory gate node; Rn The state weight parameters of the adjacent nodes of the memory gate; b represents the hidden state of the neighboring nodes of node i at level t; R W is the memory gate constant variable. o U is the output gate weight matrix; o U represents the hidden state weight parameters of the output gate node; on b represents the state weight parameters of the adjacent nodes of the output gate; o This is the output gate constant variable.

[0080] In practical applications, the hidden state and storage state of the output nodes of the graph LSTM structure are updated based on 5 gating points.

[0081] The update process is as follows:

[0082]

[0083] h i,t+1 =O i ⊙tanh(C i,t+1 )

[0084] Among them, C i,t+1 The output node's storage state is represented by ⊙; the node-by-node product is represented by ⊙; N(i) is the number of adjacent nodes; h i,t+1 This represents the hidden state of the output node.

[0085] All weight parameters are trained using backpropagation.

[0086] In practical applications, the weakly supervised video violence detection model further includes: summing the RGB features, optical flow features, and audio features to determine the summed features; sequentially passing the summed features through a 1×1 convolution and a GELU activation function to determine the feature weights; multiplying the feature weights with the summed features to determine the product features; and superimposing the product features with the summed features for output. This operation process is used to obtain the residual value between the summed features and the true features. The feature weights are used to fuse multi-channel information to achieve temporal scale context information, fully utilizing the correlation between features before and after time for information interaction and supplementation, thereby improving the accuracy of violence event prediction on a time series.

[0087] When the product feature and the summation feature are superimposed on the output structure, a feature of scale Batch×2C is obtained through a fully connected layer; Batch is the batch size, and C is the feature time series length; the Batch×2C feature is divided into two features of scale Batch×C; one Batch×C feature is used to output the frame-by-frame result, and the other Batch×C feature is convolved again with 1×1 to output the global result. The difference between the two Batch×C features is used to characterize the frame-by-frame loss.

[0088] The total loss L of the model designed in this invention consists of three components, including the global loss L. a Real-time loss per frame L m The BCE loss function is used to calculate the similarity L between the global result and the frame-by-frame result. s The cross-entropy loss function is used as the metric.

[0089] L = L a +L m +2L s

[0090] All training networks in this invention were implemented on the Python-based deep learning framework PyTorch 1.11, using an NVIDIA A40 processor. To ensure fairness in the method comparison, all network model hyperparameters were uniformly set as follows: training batch (epoch) of 40 and learning rate of 0.01. The dataset contains a total of 10,000 video clips. During the process of inputting video clips into the model for training, the network uses ffmpeg to convert the input video clips into images at 0.2 seconds per frame.

[0091] This invention is compared with the weakly supervised abnormal behavior monitoring method XD, and the loss value and detection accuracy of each model are output. When the training batch is 20, the loss of this invention is close to that of the comparison method XD when it is 30 batches, as shown in Table 1. This invention is superior to the comparison method in information mining, and the three-branch structure is helpful for feature extraction.

[0092] Table 1 Comparison of Iterative Loss Functions

[0093]

[0094]

[0095] Accuracy tests were conducted on the entire dataset. The frame-by-frame detection accuracy was determined manually after random sampling. In this example, 2000 video frames were randomly collected for result analysis. Compared with the comparison method, the present invention achieved a global detection accuracy of 79.583%, an improvement of about 3 percentage points, and a frame-by-frame detection accuracy of 76.9%, an improvement of 4.45 percentage points.

[0096] Table 2 Model Performance Comparison Table

[0097]

[0098] Figure 3 The example images of the detection results of this invention show screenshots of two video scenes where violent behavior was detected. Figure 3 (a) and Figure 3 (b) Showing the perspective of subway station surveillance cameras, with a large number of people and complex behaviors. Figure 3 (c) and Figure 3 (d) Displaying close-up scenes with a small number of people. This invention can identify violent behavior in videos under different scenarios, recognize the nodes where the behavior occurs and issue warnings, displaying "Anomaly Detected," and can be effectively applied to the detection of violent behavior in videos.

[0099] Example 2

[0100] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a video violence detection system based on multimodal information fusion is provided below.

[0101] A multimodal information fusion video violence detection system includes:

[0102] The video clip generation module is used to acquire videos from different sources and from different angles, divide the videos, and generate video clips.

[0103] The weak labeling module is used to label video clips containing violent behavior as violent videos using a weak supervised labeling method, thus creating weak labels.

[0104] A violence behavior recognition module is used to input the video to be detected into a violence behavior detection model to identify violent behaviors in the video. The violence behavior detection model is obtained by training a weakly supervised video violence behavior detection model based on video segments with weak labels. The weakly supervised video violence behavior detection model includes a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanisms, and graph LSTM structure. The three-branch multimodal information optimizer is used to extract RGB features, optical flow features, and audio features from the video segments.

[0105] In practical applications, the three-branch multimodal information optimizer specifically includes: an RGB feature optimizer, an optical flow feature optimizer, and an audio feature optimizer. When the three-branch multimodal information optimizer extracts the RGB features, an I3D network is used to extract the RGB features and input them into the RGB feature optimizer. The RGB feature optimizer includes four layers of multi-scale structured convolutions and a graph convolution structure connected in sequence. The four layers of multi-scale structured convolutions include a 1×1 convolution that does not consider the neighborhood and integrates all time series, a 3×3 regular convolution that considers the neighborhood and integrates all time series, a 3×3 convolution that considers the neighborhood and is spaced 5 units apart on the time series, and a 3×3 convolution that considers the neighborhood and is spaced 9 units apart on the time series. The four layers of multi-scale structured convolutions are used to determine the temporal boundaries of the RGB features where the violent behavior exists, highlighting the abnormal features of the video when the violent behavior exists. The graph convolution is used to receive the RGB features after passing through the four layers of multi-scale structured convolutions and reduce the scale of the RGB features.

[0106] Example 3

[0107] This invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the video violence detection method based on multimodal information fusion provided in Embodiment 1.

[0108] In practical applications, the aforementioned electronic devices can be servers.

[0109] In practical applications, electronic devices include: at least one processor, memory, bus, and communication interface.

[0110] The processor, communication interface, and memory communicate with each other via a communication bus.

[0111] A communication interface is used to communicate with other devices.

[0112] The processor is used to execute programs, specifically the methods described in the above embodiments.

[0113] Specifically, the program may include program code, which includes computer operation instructions.

[0114] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0115] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0116] Based on the description of the above embodiments, this application provides a storage medium storing computer program instructions thereon, which can be executed by a processor to implement the methods described in any embodiment.

[0117] The multimodal information fusion video violence detection system provided in this application exists in various forms, including but not limited to:

[0118] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0119] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access capabilities. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0120] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0121] (4) Other electronic devices with data interaction functions.

[0122] Specific embodiments of the subject matter have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0123] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0124] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0130] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, and CD-ROM.

[0131] Digital multifunction optical disc (DVD) or other optical storage, magnetic cassette tape, magnetic tape, disk storage or other magnetic storage devices

[0132] Or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0133] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific transactions or implement specific abstract data types. This application can also be practiced in distributed computing environments where transactions are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0135] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0136] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A video violence detection method based on multimodal information fusion, characterized in that, include: Acquire videos from different sources and from different angles, divide the videos, and generate video clips; Weakly supervised labeling is used to mark video clips containing violent behavior as violent videos, thus creating weak tags; The video to be detected is input into the violence detection model to identify violent behaviors in the video; the violence detection model is obtained by training a weakly supervised video violence detection model based on video segments with weak labels; the weakly supervised video violence detection model includes a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanism and graph LSTM structure; The three-branch multimodal information optimizer is used to extract RGB features, optical flow features, and audio features from the video clip; The three-branch multimodal information optimizer specifically includes: an RGB feature optimizer, an optical flow feature optimizer, and an audio feature optimizer; When the three-branch multimodal information optimizer extracts the RGB features, the I3D network is used to extract the RGB features and input them into the RGB feature optimizer; The RGB feature optimizer includes four sequentially connected multi-scale convolutional layers and a graph convolutional structure. The four multi-scale convolutional layers include a 1×1 convolution that does not consider the neighborhood and integrates all time series, a 3×3 regular convolution that considers the neighborhood and integrates all time series, a 3×3 convolution that considers the neighborhood and is spaced 5 units apart on the time series, and a 3×3 convolution that considers the neighborhood and is spaced 9 units apart on the time series. The four multi-scale convolutional layers are used to determine the temporal boundaries of the RGB features indicating the presence of the violent behavior, highlighting the video anomaly features when the violent behavior exists. The graph convolution is used to receive the RGB features processed by the four multi-scale convolutional layers and reduce the scale of the RGB features.

2. The video violence detection method based on multimodal information fusion according to claim 1, characterized in that, When the three-branch multimodal information optimizer extracts the optical flow features, an I3D network is used to extract the optical flow features and input them into the graph convolutional structure in the optical flow feature optimizer; the execution process of the graph convolutional structure is as follows: ;in, For the first Layer i Node characteristics; j For nodes i The adjacent nodes; For nodes i The degree; For nodes j The degree, For the first The layer's weight parameter matrix, It is a non-linear activation function.

3. The video violence detection method based on multimodal information fusion according to claim 1, characterized in that, When the three-branch multimodal information optimizer extracts the audio features, it extracts the audio features through a VGGish network and inputs them into the audio feature optimizer; the audio feature optimizer is a graph LSTM structure; The graph LSTM structure includes an input gate. Forgotten Gate Adaptive forget gate Memory gate and output gate The loop process of each gate in the graph LSTM structure is as follows: ； in, i For nodes; t Number of floors; Represents the sigmoid function; The input gate weight matrix; The node state is the input. The hidden state weight parameters of the input gate node; This represents the hidden state of node i at layer t. The state weight parameters of the adjacent nodes of the input gate; For input gate constant variables; Here is the forget gate weight matrix; The hidden state weight parameters for the forget gate nodes; Forget gate constant variables; The state weight parameters for adjacent nodes of the forget gate; Adaptive forget gate constant variable; This is the memory gate weight matrix; The hidden state weight parameters for memory gate nodes; The state weight parameters of the adjacent nodes of the memory gate; Let i be the hidden state of its neighboring nodes at level t. For memory gate constant variables; This is the output gate weight matrix; Output the hidden state weight parameters of the gate nodes; The output gate's adjacent node state weight parameters; This is the output gate constant variable.

4. The video violence detection method based on multimodal information fusion according to claim 3, characterized in that, The hidden and stored states of the output nodes of the graph LSTM structure are updated based on 5 gating points. The update process is as follows: in, ⊙ represents the storage state of the output node; ⊙ represents the pointwise product. This represents the number of adjacent nodes; This represents the hidden state of the output node.

5. The video violence detection method based on multimodal information fusion according to claim 1, characterized in that, The weakly supervised video violence detection model also includes: The RGB features, optical flow features, and audio features are summed to determine the summed features; The summed features are sequentially passed through a 1×1 convolution and a GELU activation function to determine the feature weights; The product features are determined by multiplying the feature weights with the summed features. The product feature and the summation feature are superimposed and output to a fully connected layer to determine the feature with a scale of Batch×2C; Batch is the batch size and C is the length of the feature time series. The Batch×2C features are divided into two Batch×C features; one Batch×C feature is used to output the frame-by-frame result, and the other Batch×C feature is convolved again with 1×1 to output the global result. The difference between the two Batch×C features is used to characterize the frame-by-frame loss.

6. A video violence detection system based on multimodal information fusion, characterized in that, include: The video clip generation module is used to acquire videos from different sources and angles, divide the videos, and generate video clips; The weak labeling module is used to label video clips containing violent behavior as violent videos using a weak supervised labeling method, thus creating weak labels. A violence behavior recognition module is used to input the video to be detected into a violence behavior detection model to identify violent behaviors in the video to be detected. The violence behavior detection model is obtained by training a weakly supervised video violence behavior detection model based on video segments with weak labels. The weakly supervised video violence behavior detection model includes a three-branch multimodal information optimizer that combines graph convolution, multi-scale mechanism and graph LSTM structure. The three-branch multimodal information optimizer is used to extract RGB features, optical flow features, and audio features from the video clip; The three-branch multimodal information optimizer specifically includes: an RGB feature optimizer, an optical flow feature optimizer, and an audio feature optimizer; When the three-branch multimodal information optimizer extracts the RGB features, the I3D network is used to extract the RGB features and input them into the RGB feature optimizer; The RGB feature optimizer includes four sequentially connected multi-scale convolutional layers and a graph convolutional structure. The four multi-scale convolutional layers include a 1×1 convolution that does not consider the neighborhood and integrates all time series, a 3×3 regular convolution that considers the neighborhood and integrates all time series, a 3×3 convolution that considers the neighborhood and is spaced 5 units apart on the time series, and a 3×3 convolution that considers the neighborhood and is spaced 9 units apart on the time series. The four multi-scale convolutional layers are used to determine the temporal boundaries of the RGB features indicating the presence of the violent behavior, highlighting the video anomaly features when the violent behavior exists. The graph convolution is used to receive the RGB features processed by the four multi-scale convolutional layers and reduce the scale of the RGB features.

7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the video violence detection method based on multimodal information fusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the video violence detection method based on multimodal information fusion as described in any one of claims 1-5.

Images