A multimodal video anomaly detection method and related equipment that integrates RGB appearance, skeletal pose, and audio information.

Abstract

This invention provides a multimodal video anomaly detection method and related equipment that combines RGB appearance, skeletal pose, and audio information. The method includes: segmenting a target video into several video segments; preprocessing each video segment to obtain multimodal data for each segment, the multimodal data including human skeletal trajectory, key appearance regions, and audio spectrograms; extracting features from the multimodal data of each video segment using a skeletal pose sub-network, an RGB appearance sub-network, and an audio information sub-network, respectively, to obtain skeletal pose features, appearance features, and audio features for each video segment; employing an adaptive multi-stream fusion strategy to fuse the skeletal pose features, appearance features, and audio features of each video segment and predicting an anomaly score for each video segment; if the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior.

Description

Technical Field

[0001] This invention relates to the field of video anomaly detection technology, and in particular to a multimodal video anomaly detection method and related equipment that combines RGB appearance, skeletal pose and audio information. Background Technology

[0002] Video anomaly detection is one of the research directions in intelligent video anomaly detection. In the era of "smart cities" and "smart transportation," a large number of video surveillance cameras are widely installed in public places such as subway entrances and campuses, generating massive amounts of video data. Due to the unbounded and fuzzy nature of video anomaly events, the traditional method of manually reviewing each frame is inefficient, costly, and has excessive latency. Therefore, utilizing computer vision technology to achieve intelligent analysis of video content and improve the accuracy of anomaly detection is an important research topic in the field of intelligent video analysis.

[0003] In recent years, with the rapid development of deep learning and computer vision, experts and scholars at home and abroad have conducted in-depth research on the semantic content features of videos and have achieved good results in video anomaly detection. Unlike images, which only contain static visual appearance information, videos are inherently multimodal. Fusing static features such as appearance and skeleton with dynamic features such as motion and time series is effective in improving detection accuracy. Current research usually uses multi-stream networks to extract static appearance features and short-term motion cues separately, and then directly fuses the output of the multi-stream network. This results in poor detection performance and deviation from the essential semantics of the event. In addition, videos generally contain temporal motion and auditory track information. For example, a "diving" video usually involves a series of event atoms, such as "jumping off the platform," "spinning in the air," and "falling into the water," accompanied by cheers or applause. However, existing anomaly detection methods rarely utilize the audio information in videos. Summary of the Invention

[0004] This invention provides a multimodal video anomaly detection method and related equipment that combines RGB appearance, skeletal pose and audio information, aiming to improve the accuracy of video abnormal behavior detection by adaptively fusing multimodal features using a multi-stream network structure.

[0005] In a first aspect, the present invention provides a multimodal video anomaly detection method that combines RGB appearance, skeletal pose, and audio information, comprising:

[0006] Step 1: Divide the target video into several video segments;

[0007] Step 2: Preprocess each video segment to obtain multimodal data for each video segment, including human skeletal trajectory, key appearance regions, and audio spectrograms;

[0008] Step 3: Use the skeletal pose subnetwork, RGB appearance subnetwork and audio information subnetwork to extract features from the multimodal data of each video segment to obtain the skeletal pose features, appearance features and audio features of each video segment;

[0009] Step 4: Use an adaptive multi-stream fusion strategy to fuse the skeletal pose features, appearance features, and audio features of each video segment and predict the anomaly score of each video segment;

[0010] Step 5: If the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior.

[0011] Furthermore, in step 2, the preprocessing of each video segment includes:

[0012] Human pose estimation algorithm is used to extract human skeleton key points in each video frame of the video segment. The human skeleton key points of all video frames constitute the human skeleton trajectory of the video segment.

[0013] The human skeleton key points extracted from each video frame in the video clip are used to determine the human rectangular region in the video frame. The human rectangular regions of all video frames constitute the appearance key region of the video clip.

[0014] The audio information corresponding to the video segment is converted into a sound spectrogram using short-time Fourier transform.

[0015] Furthermore, in step 3, an LSTM network is used as the skeletal pose subnetwork;

[0016] Correspondingly, an LSTM network is used to extract features from the human skeleton trajectory of each video segment, obtaining the skeletal pose features of each video segment, specifically including:

[0017] The global pose component x of the target human body in each video frame is calculated using the following formula. t g ,y t g ,h t ,w t Where t∈(1,…,T) represents the video frame number in the video segment, and T represents the total number of video frames in the video segment;

[0018] w t =max(x i,t )-min(x i,t )

[0019] h t =max(y i,t )-min(y i,t )

[0020]

[0021]

[0022] Where, x i,t and y i,t Let x and y represent the x and y coordinates of the i-th skeletal keypoint of the target human body in the t-th video frame in the given coordinate system, i∈(1,…,k), and k represents the total number of skeletal keypoints in the t-th video frame;

[0023] The local pose components of the target human body in each video frame are calculated using the following formula.

[0024]

[0025]

[0026] The global and local pose components of the target human body in each video frame are stitched together to form the skeletal pose feature of that video frame. The skeletal pose features of all video frames in the video segment constitute the skeletal pose feature of the video segment.

[0027] Furthermore, in step 3, a spatiotemporal generative adversarial network is used as the RGB appearance sub-network.

[0028] Furthermore, in step 3, a convolutional neural network is used as the audio information sub-network.

[0029] Furthermore, the adaptive multi-stream fusion strategy specifically includes:

[0030] For each category c and stream m, define a predicted score vector. For each modality, the feature vector output by the sub-network is used to stack the prediction score vectors from all streams to form a fusion vector for each training video sample n. Where C is the total number of categories and M is the total number of streams;

[0031] The optimal fusion weight matrix W is represented as:

[0032]

[0033] Among them, y n,c w represents the true label of training video sample n for class c. c This represents the weight vector for category c;

[0034] The parameters of the entire network are optimized by minimizing the following loss function, thereby learning the optimal fusion weight matrix W;

[0035]

[0036] Where S represents the predicted score of all training video samples in a training batch, Y represents the true label of all training video samples in a training batch, and V = [V 1 ,...,V m ,...,V M ] T V represents the class relation matrix, used to regularize the learning process of the weight matrix W. m V represents the class relation vector corresponding to the stream m of all training video samples in a training batch, where each element V is a class relation vector. i,j λ1 and λ2 represent the proportion of category i that is misclassified as category j, and λ1 and λ2 are weight parameters.

[0037] Secondly, the present invention provides a multimodal video anomaly detection device that combines RGB appearance, skeletal pose, and audio information, comprising:

[0038] The video segmentation module is used to divide a target video into several video segments;

[0039] The preprocessing module is used to preprocess each video segment to obtain multimodal data for each video segment, including human skeletal trajectory, key appearance regions, and sound spectrograms;

[0040] The feature extraction module is used to extract features from the multimodal data of each video segment using the skeletal pose subnetwork, RGB appearance subnetwork and audio information subnetwork respectively, to obtain the skeletal pose features, appearance features and audio features of each video segment;

[0041] The fusion module is used to fuse the skeletal pose features, appearance features, and audio features of each video segment using an adaptive multi-stream fusion strategy and predict the anomaly score of each video segment.

[0042] An anomaly detection module is used to detect whether the anomaly score of a video segment exceeds a set threshold. If so, the video segment is considered to contain abnormal behavior.

[0043] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in the first aspect.

[0044] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0045] The beneficial effects of this invention are:

[0046] (1) This invention models RGB appearance features, skeletal pose features and audio information features by training three different sub-networks respectively, which can extract multimodal features of the target video more comprehensively, enabling video anomaly detection based on more comprehensive multimodal features and improving detection accuracy.

[0047] (2) In order to combine the output of the multi-stream network, this invention proposes an adaptive multi-stream fusion strategy, which adaptively learns the optimal fusion weight for each category and uses the estimated class relationship as a regularization factor to influence the learning process of the weight, thereby strengthening the connection between categories and obtaining better video anomaly detection performance. Attached Figure Description

[0048] Figure 1 A flowchart illustrating a multimodal video anomaly detection method that combines RGB appearance, skeletal pose, and audio information, provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of skeletal pose feature extraction provided in an embodiment of the present invention;

[0050] Figure 3 Partial visualization of the detection method of the present invention provided in the embodiments of the present invention on HR-ShanghaiTech;

[0051] Figure 4 Visualization of the normal and abnormal effects of the detection method of the present invention in a continuous sequence, as provided in the embodiments of the present invention;

[0052] Figure 5 A flowchart illustrating a multimodal video anomaly detection method that combines RGB appearance, skeletal pose, and audio information, provided in an embodiment of the present invention.

[0053] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] like Figure 1As shown, this embodiment of the invention provides a multimodal video anomaly detection method that combines RGB appearance, skeletal pose, and audio information, including the following steps:

[0056] S101: The target video is divided into several video segments by means of resizing, image normalization and cropping; it can be understood that each video segment includes several consecutive video frames.

[0057] S102: Preprocess each video segment to obtain multimodal data for each video segment, the multimodal data including human skeletal trajectory, key appearance regions, and sound spectrogram;

[0058] S103: Use the skeletal pose subnetwork, RGB appearance subnetwork and audio information subnetwork to extract features from the multimodal data of each video segment, and obtain the skeletal pose features, appearance features and audio features of each video segment;

[0059] S104: An adaptive multi-stream fusion strategy is adopted to fuse the skeletal pose features, appearance features, and audio features of each video segment and predict the anomaly score of each video segment.

[0060] It is understandable that the anomaly score for each category is a fused score vector, where each component represents the degree of anomaly for the corresponding category.

[0061] S105: If the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior; otherwise, it is considered not to contain abnormal behavior. It is understood that the threshold can be customized by the user based on specific circumstances.

[0062] The multimodal video anomaly detection method provided in this invention combines RGB appearance, skeletal pose features, and audio information. It trains three different sub-networks to model RGB appearance features, skeletal pose features, and audio information features respectively, which can extract multimodal features of the target video more comprehensively. This enables video anomaly detection based on more comprehensive multimodal features and improves detection accuracy.

[0063] In one embodiment, preprocessing each video segment includes: extracting human skeleton key points from each video frame within the video segment using a human pose estimation algorithm, wherein the human skeleton key points from all video frames constitute the human skeletal trajectory of the video segment; determining the human rectangular region in the video frame based on the human skeleton key points extracted from each video frame within the video segment (e.g., the maximum and minimum values ​​of the skeleton key points), wherein the human rectangular regions from all video frames constitute the appearance key regions of the video segment; and converting the audio information corresponding to the video segment into a sound spectrogram using a short-time Fourier transform.

[0064] Specifically, the key points of the human skeleton extracted in this embodiment are 2D skeleton key points. The horizontal and vertical axes of the sound spectrogram are the time scale and frequency scale, respectively.

[0065] like Figure 2 As shown, in one embodiment, an LSTM network is used as the skeletal pose sub-network; correspondingly, the LSTM network is used to extract features from the human skeletal trajectory of each video segment to obtain the skeletal pose features of each video segment, specifically including:

[0066] For any target human body in any video frame, let f be the rectangular region of the human body. g =(x g ,y g x, w, h), where w and h represent the height and width of the rectangular human body region; x g ,y g Indicates the center of the rectangular area of ​​the human body;

[0067] The global pose components of the target human body in the t-th video frame are calculated using the following formula. Where t∈(1,…,T) represents the video frame number in the video segment, and T represents the total number of video frames in the video segment;

[0068] w t =max(x i,t )-min(x i,t )

[0069] h t =max(y i,t )-min(y i,t )

[0070]

[0071]

[0072] Where, x i,t and y i,t Let x and y represent the x and y coordinates of the i-th skeletal keypoint of the target human body in the t-th video frame in the given coordinate system, i∈(1,…,k), and k represents the total number of skeletal keypoints in the t-th video frame;

[0073] The local pose components of the target human body in the t-th video frame are calculated using the following formula.

[0074]

[0075]

[0076] The global and local pose components of the target human body in each video frame are stitched together to form the skeletal pose feature of that video frame. The skeletal pose features of all video frames in the video segment constitute the skeletal pose feature of the video segment.

[0077] It is understandable that if the local pose components of the t-th video frame are represented as... and global attitude components The skeletal pose features of the t-th video frame obtained by concatenating the two can be represented as follows:

[0078] In one embodiment, a spatiotemporal generative adversarial network (GAN) as disclosed in CN117593684A is used as the RGB appearance sub-network. This GAN extracts the RGB appearance features of each video frame and predicts the next frame. Correspondingly, the GAN is used to extract features from the key appearance regions of each video segment to obtain the appearance features of each video segment. Specifically, this includes: inputting the continuous video frames contained in the video segment into the GAN; continuously reducing the size through downsampling during the encoding stage to extract spatial structure and local detail features; and restoring the size through upsampling during the decoding stage to output the final result.

[0079] Specifically, to perform spatiotemporal modeling of the RGB appearance features of consecutive video frames, a spatiotemporal Transformer module is proposed. This module utilizes a multi-head attention mechanism to fuse patches in the same space at different time scales, and outputs feature Z after scale transformation. a A multi-scale channel module is proposed to concatenate features from the encoding and decoding stages, outputting a fused multi-scale feature Z. b Z a With Z b The images are stitched together, pooling layers in the original generator are removed, and upsampling is performed by stride of the convolutional layers to gradually expand the receptive field and restore the image size. All convolutional blocks contain 3D-Batchnorm layers and ReLU layers. The last convolutional block adds a Tanh activation layer and sets its temporal padding to 0 to ensure that the extracted appearance features are the same in dimension as the skeletal pose features.

[0080] In one embodiment, a convolutional neural network is used as the audio information subnetwork.

[0081] In one embodiment, to combine the outputs of the multi-stream network, an adaptive multi-stream fusion strategy is proposed. This strategy adaptively learns the optimal fusion weights for each category and uses the estimated class relationships as a regularization factor to influence the weight learning process, thereby strengthening the connections between categories and automatically identifying highly correlated categories to achieve better video anomaly detection performance.

[0082] Specifically, the adaptive multi-stream fusion strategy includes:

[0083] For each category c and stream m, define a predicted score vector. This represents the feature vector output by the sub-network for the corresponding modality; for n training video samples, the predicted score vectors from all streams are stacked to form a fusion vector. Where C is the total number of categories (in this embodiment, the video samples are divided into two categories: normal and abnormal, i.e., C=2), and M is the total number of streams (in this embodiment, they are divided into three feature streams: skeletal pose, appearance, and audio, i.e., M=3).

[0084] Based on the above, this embodiment represents the optimal fusion weight matrix W of the three feature streams as follows:

[0085]

[0086] Among them, y n,c w represents the true label of training video sample n for class c. c This represents the weight vector for category c. Specifically, the weight vector for abnormal cases is w1, and the weight vector for normal cases is w2.

[0087] To achieve adaptation, this embodiment continuously optimizes the parameters of the entire network by minimizing the following loss function, thereby learning the optimal fusion weight matrix W.

[0088]

[0089] Where S represents the predicted score of all training video samples in a training batch, Y represents the true label of all training video samples in a training batch, and V = [V 1 ,...,V m ,...,V M ] T V represents the class relation matrix, used to regularize the learning process of the weight matrix W. m V represents the class relation vector corresponding to the stream m of all training video samples in a training batch, where each element V is a class relation vector. i,j λ1 and λ2 represent the proportion of category i that is misclassified as category j, and λ1 and λ2 are weight parameters.

[0090] To verify the effectiveness of the present invention, the present invention also provides the following experimental data.

[0091] This invention takes into account the complexity, ambiguity, and imbalanced distribution of positive and negative samples of anomalous behavior. It employs a binary classification approach for anomalous behavior detection, distinguishing between abnormal and normal behavior within a specific segment of the target video dataset. Correspondingly, this invention uses frame-level AUC to evaluate the performance of anomaly detection; a higher AUC value indicates better detection performance. Generally, an AUC value greater than 0.5 is considered acceptable, while an AUC value close to 1 indicates even better performance. Within a video frame, if one or more anomalous events occur, the frame is marked as an anomalous frame.

[0092] The accuracy of the detection framework was calculated and compared with other state-of-the-art methods. The entire experiment was conducted on an NVIDIA RTX 3070ti, using the PyTorch framework to build the model. The AUC of different methods was compared on the ShanghaiTech, CUHK Avenue, and Corridor datasets. To better demonstrate the higher accuracy and robustness of this invention in identifying human-related abnormal behaviors, some irrelevant data was filtered and removed, resulting in datasets specifically focused on human behavior, including HR Avenue, HR ShanghaiTech, and HR Corridor. As shown in Table 1, the method of this invention outperforms other state-of-the-art mainstream methods on multiple datasets.

[0093] Table 1 Comparison of AUC (%) between the present invention and other methods

[0094]

[0095] like Figure 3 As shown, effectively fusing appearance features and skeletal features can effectively identify human-object interaction behaviors (such as riding a bicycle), thereby improving detection accuracy. Figure 4 As shown, two example sequences from the same scene are displayed. It can be seen that when dealing with abnormal behavior, the present invention predicts the skeleton joints according to the normal mode and visualizes the predicted skeleton results. As can be seen from the figure, when dealing with abnormal behavior, the predicted skeleton is significantly different from the historical skeleton; when dealing with normal behavior, the similarity between the predicted skeleton and the historical skeleton is significantly improved.

[0096] like Figure 5 As shown, this embodiment of the invention also provides a multimodal video anomaly detection device that combines RGB appearance, skeletal pose and audio information, including a video segmentation module, a preprocessing module, a feature extraction module, a fusion module and an anomaly detection module.

[0097] The video segmentation module divides the target video into several video segments; the preprocessing module preprocesses each video segment to obtain multimodal data for each video segment, including human skeletal trajectory, key appearance regions, and audio spectrograms; the feature extraction module uses a skeletal pose subnetwork, an RGB appearance subnetwork, and an audio information subnetwork to extract features from the multimodal data of each video segment, obtaining skeletal pose features, appearance features, and audio features for each video segment; the fusion module uses an adaptive multi-stream fusion strategy to fuse the skeletal pose features, appearance features, and audio features of each video segment and predict the anomaly score for each video segment; the anomaly detection module detects whether the anomaly score of a video segment exceeds a set threshold, and if so, considers the video segment to contain abnormal behavior.

[0098] It should be noted that the multimodal video anomaly detection device provided in this embodiment of the invention is for implementing the above method, and its specific functions can be referred to the above method embodiments, which will not be repeated here.

[0099] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call logical instructions in the memory 603 to execute a multimodal video anomaly detection method that combines RGB appearance, skeletal pose, and audio information. This method includes: segmenting the target video into several video segments; preprocessing each video segment to obtain multimodal data for each segment, the multimodal data including human skeletal trajectory, key appearance regions, and audio spectrograms; extracting features from the multimodal data of each video segment using a skeletal pose sub-network, an RGB appearance sub-network, and an audio information sub-network, respectively, to obtain skeletal pose features, appearance features, and audio features for each video segment; fusing the skeletal pose features, appearance features, and audio features of each video segment using an adaptive multi-stream fusion strategy and predicting the anomaly score for each video segment; and considering that the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior.

[0100] Furthermore, when the logical instructions in the aforementioned memory 603 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] This invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: segmenting a target video into several video segments; preprocessing each video segment to obtain multimodal data for each video segment, the multimodal data including human skeletal trajectory, key appearance regions, and audio spectrograms; using a skeletal pose subnetwork, an RGB appearance subnetwork, and an audio information subnetwork to extract features from the multimodal data of each video segment, respectively, to obtain skeletal pose features, appearance features, and audio features for each video segment; using an adaptive multi-stream fusion strategy to fuse the skeletal pose features, appearance features, and audio features of each video segment and predict the anomaly score of each video segment; if the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior.

[0102] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the methods provided in the above-described method embodiments, including, for example,: segmenting a target video into several video segments; preprocessing each video segment to obtain multimodal data for each video segment, the multimodal data including human skeletal trajectory, key appearance regions, and audio spectrograms; using a skeletal pose subnetwork, an RGB appearance subnetwork, and an audio information subnetwork to extract features from the multimodal data of each video segment, obtaining skeletal pose features, appearance features, and audio features for each video segment; employing an adaptive multi-stream fusion strategy to fuse the skeletal pose features, appearance features, and audio features of each video segment and predict the anomaly score for each video segment; if the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal video anomaly detection method that combines RGB appearance, skeletal pose, and audio information, characterized in that, include: Step 1: Divide the target video into several video segments; Step 2: Preprocess each video segment to obtain multimodal data for each video segment, including human skeletal trajectory, key appearance regions, and audio spectrograms; Step 3: Use the skeletal pose subnetwork, RGB appearance subnetwork and audio information subnetwork to extract features from the multimodal data of each video segment to obtain the skeletal pose features, appearance features and audio features of each video segment; Specifically, an LSTM network is used as the skeletal pose subnetwork; a spatiotemporal generative adversarial network is used as the RGB appearance subnetwork; and a convolutional neural network is used as the audio information subnetwork. Correspondingly, an LSTM network is used to extract features from the human skeleton trajectory of each video segment, obtaining the skeletal pose features of each video segment, specifically including: The global pose components of the target human body in each video frame are calculated using the following formula. ;in, This indicates the video frame number in the video clip, and T represents the total number of video frames in the video clip; in, and Let x and y represent the x and y coordinates of the i-th skeletal keypoint of the target human body in the t-th video frame of the given coordinate system, respectively. k represents the total number of skeleton keypoints in the t-th video frame; The local pose components of the target human body in each video frame are calculated using the following formula. ; The global pose component and local pose component of the target human body in each video frame are stitched together to form the skeletal pose feature of that video frame. The skeletal pose features of all video frames in the video segment constitute the skeletal pose feature of the video segment. Step 4: Use an adaptive multi-stream fusion strategy to fuse the skeletal pose features, appearance features, and audio features of each video segment and predict the anomaly score of each video segment; Step 5: If the anomaly score of a video segment exceeds a set threshold, the video segment is considered to contain abnormal behavior.

2. The multimodal video anomaly detection method combining RGB appearance, skeletal pose, and audio information according to claim 1, characterized in that, Step 2, the preprocessing of each video segment includes: Human pose estimation algorithm is used to extract human skeleton key points in each video frame of the video segment. The human skeleton key points of all video frames constitute the human skeleton trajectory of the video segment. The human skeleton key points extracted from each video frame in the video clip are used to determine the human rectangular region in the video frame. The human rectangular regions of all video frames constitute the appearance key region of the video clip. The audio information corresponding to the video segment is converted into a sound spectrogram using short-time Fourier transform.

3. A multimodal video anomaly detection method combining RGB appearance, skeletal pose, and audio information as described in claim 1 or 2, characterized in that, The adaptive multi-stream fusion strategy specifically includes: For each category Heliu Define the predicted score vector For the feature vector output by the sub-network corresponding to the modality, for the training video samples The predicted score vectors from all streams are stacked to form a fusion vector. ;in, It is the total number of categories. It is the total number of flows; The optimal fusion weight matrix W is represented as: in, Indicates training video samples For category The true label, This represents the weight vector for category c; The parameters of the entire network are optimized by minimizing the following loss function, thereby learning the optimal fusion weight matrix W; in, This represents the predicted score for all training video samples in a training batch. This represents the true labels corresponding to all training video samples in a training batch. The class relation matrix is ​​used to regularize the learning process of the weight matrix W. This represents the class relation vector corresponding to the stream m of all training video samples in a training batch, where each element in the class relation vector represents a class relation vector. This represents the proportion of category i that is misclassified as category j. and These are the weight parameters.

4. A multimodal video anomaly detection device that combines RGB appearance, skeletal pose, and audio information, applied to the method described in any one of claims 1 to 3, characterized in that, include: The video segmentation module is used to divide a target video into several video segments; The preprocessing module is used to preprocess each video segment to obtain multimodal data for each video segment, including human skeletal trajectory, key appearance regions, and sound spectrograms; The feature extraction module is used to extract features from the multimodal data of each video segment using the skeletal pose subnetwork, RGB appearance subnetwork and audio information subnetwork respectively, to obtain the skeletal pose features, appearance features and audio features of each video segment; The fusion module is used to fuse the skeletal pose features, appearance features, and audio features of each video segment using an adaptive multi-stream fusion strategy and predict the anomaly score of each video segment. An anomaly detection module is used to detect whether the anomaly score of a video segment exceeds a set threshold. If so, the video segment is considered to contain abnormal behavior.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

Images