Multi-mode video anomaly detection method combining RGB appearance, skeleton posture and audio information and related equipment

Through a multimodal video anomaly detection method combining RGB appearance, skeletal posture and audio information, the adaptive multi-stream fusion strategy is used to solve the problem of poor detection performance in the existing technology, and achieve higher accuracy of video anomaly behavior detection .

CN119007288AActive Publication Date: 2024-11-22HENAN UNIVERSITY

Patent Information

Application Number
CN202411077123.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-22
Estimated Expiration
2044-08-07

AI Technical Summary

Technical Problem

The existing video abnormal behavior detection methods have poor detection performance when fusing static visual appearance information and dynamic motion characteristics, and have not fully utilized the audio information in the video, resulting in low detection accuracy.

Method used

Multimodal video anomaly detection method combining RGB appearance, skeletal attitude and audio information is adopted to extract features through bone attitude subnet, RGB appearance subnet and audio information subnet respectively, and fusion features are adopted for adaptive multi-stream fusion strategy to improve the Detection accuracy.

Benefits of technology

By comprehensively extracting multimodal features, the accuracy of video anomaly behavior detection is improved, and the connection between categories is optimized through adaptive fusion strategies, which significantly improves detection performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007288A_ABST
    Figure CN119007288A_ABST
Patent Text Reader

Abstract

The invention provides a multi-mode video anomaly detection method combining RGB appearance, skeleton postures and audio information and related equipment. The method comprises the following steps: segmenting a target video into a plurality of video clips; preprocessing each video clip to obtain multi-modal data of each video clip, wherein the multi-modal data comprises a human skeleton track, an appearance key area and a sound spectrogram; performing feature extraction on the multi-modal data of each video clip by using a skeleton posture sub-network, an RGB appearance sub-network and an audio information sub-network to obtain a skeleton posture feature, an appearance feature and an audio feature of each video clip; fusing the skeleton posture feature, the appearance feature and the audio feature of each video clip by adopting a self-adaptive multi-stream fusion strategy, and predicting an abnormal score of each video clip; and if the abnormal score of the video clip exceeds a set threshold value, considering that the video clip contains an abnormal behavior.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information and related equipment. Background Art

[0002] Abnormal video behavior detection is one of the research directions in intelligent video anomaly detection. In the era of "smart city" and "smart transportation", a large number of video surveillance is widely installed in public places such as subway entrances and campuses, generating massive video data. Due to the unbounded and fuzzy characteristics of video abnormal events, the traditional manual backtracking method of viewing frame by frame has problems such as low efficiency, high cost and high latency. Therefore, using computer vision technology to realize intelligent analysis of video content and improve the accuracy of anomaly detection is an important research topic in the current field of intelligent video analysis.

[0003] In recent years, with the vigorous development of deep learning and computer vision, domestic and foreign experts and scholars have conducted in-depth research on video semantic content features, and have applied them to video abnormal behavior detection and achieved good results. Unlike images that only contain static visual appearance information, videos are multimodal in nature. Fusion of static features such as appearance and skeleton with dynamic features such as motion and time series is effective in improving detection accuracy. Current research usually uses multi-stream networks to extract static appearance features and short-term motion clues respectively, and then directly fuses the output of the multi-stream network, which has the problem of poor detection performance and deviation from the essential semantics of the event. In addition, in general, videos contain temporal motion and auditory track information. For example, "diving" videos usually involve a series of event atoms, such as "jumping off the platform", "spinning in the air" and "falling into the water", accompanied by cheers or applause. However, existing anomaly detection methods rarely use audio information in videos. Summary of the invention

[0004] The present invention provides a multimodal video anomaly detection method and related equipment combining RGB appearance, skeleton posture and audio information, aiming to utilize a multi-stream network structure to adaptively fuse multimodal features to improve the accuracy of video abnormal behavior detection.

[0005] In a first aspect, the present invention provides a multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information, comprising:

[0006] Step 1: Divide the target video into several video segments;

[0007] Step 2: Preprocess each video clip to obtain multimodal data of each video clip, wherein the multimodal data includes a human skeleton trajectory, a key appearance area, and a sound spectrum graph;

[0008] Step 3: Use the skeleton posture sub-network, RGB appearance sub-network and audio information sub-network to extract features from the multimodal data of each video clip, and obtain the skeleton posture features, appearance features and audio features of each video clip;

[0009] Step 4: Adopt an adaptive multi-stream fusion strategy to fuse the skeleton posture features, appearance features, and audio features of each video clip and predict the anomaly score of each video clip;

[0010] Step 5: If the anomaly score of the video clip exceeds a set threshold, the video clip is considered to contain abnormal behavior.

[0011] Furthermore, in step 2, preprocessing each video clip includes:

[0012] A human body posture estimation algorithm is used to extract the key points of the human skeleton in each video frame in the video clip, and the key points of the human skeleton in all video frames constitute the human skeleton trajectory of the video clip;

[0013] Determine a human rectangular region in the video frame according to the human skeleton key points extracted from each video frame in the video segment, and the human rectangular regions in all video frames constitute the appearance key region of the video segment;

[0014] The audio information corresponding to the video clip is converted into a sound spectrogram using short-time Fourier transform.

[0015] Furthermore, in step 3, the LSTM network is used as the skeleton posture subnetwork;

[0016] Correspondingly, the LSTM network is used to extract the features of the human skeleton trajectory of each video clip to obtain the skeleton posture features of each video clip, including:

[0017] The global pose component x of the target human body in each video frame is calculated as follows: t g ,y t g ,h t ,w t ; Wherein, t∈(1,…,T) represents the sequence number of the video frame in the video clip, and T represents the total number of video frames in the video clip;

[0018] w t =max(x i,t )-min(x i,t )

[0019] h t =max(y i,t )-min(y i,t )

[0020]

[0021]

[0022] Among them, x i,t and i,t Respectively represent the horizontal and vertical coordinates of the i-th skeleton key point of the target human body in the t-th video frame in the given coordinate system, i∈(1,…,k), k represents the total number of skeleton key points in the t-th video frame;

[0023] The local posture component of the target human body in each video frame is calculated according to the following formula:

[0024]

[0025]

[0026] The global posture component and the local posture component of the target human body in each video frame are spliced ​​as the skeleton posture feature of the video frame, and the skeleton posture features of all video frames in the video clip constitute the skeleton posture feature of the video clip.

[0027] Furthermore, in step 3, a spatiotemporal generative adversarial network is used as the RGB appearance sub-network.

[0028] Furthermore, in step 3, a convolutional neural network is used as the audio information subnetwork.

[0029] Furthermore, the adaptive multi-stream fusion strategy specifically includes:

[0030] For each class c and stream m, define the prediction score vector is the feature vector output by the sub-network of the corresponding modality. For the training video sample n, the prediction score vectors from all streams are stacked to form a fusion vector Where C is the total number of categories and M is the total number of flows;

[0031] The optimal fusion weight matrix W is expressed as:

[0032]

[0033] Among them, y n,c represents the true label of training video sample n for category c, w c Represents the weight vector of category c;

[0034] Minimize the following loss function to optimize the parameters of the entire network, so as to learn the optimal fusion weight matrix W;

[0035]

[0036] Among them, S represents the prediction scores corresponding to all training video samples in a training batch, Y represents the true labels corresponding to all training video samples in a training batch, V = [V 1 ,...,V m ,...,V M ] T Represents the class relationship matrix, which is used to regularize the learning process of the weight matrix W, V m Represents the class relationship vector corresponding to the stream m of all training video samples in a training batch. The element V in the class relationship vector i,j It represents the proportion of category i being misclassified as category j, and λ1 and λ2 are weight parameters.

[0037] In a second aspect, the present invention provides a multimodal video anomaly detection device combining RGB appearance, skeleton posture and audio information, comprising:

[0038] A video segmentation module is used to segment the target video into several video segments;

[0039] A preprocessing module, used for preprocessing each video clip to obtain multimodal data of each video clip, wherein the multimodal data includes a human skeleton trajectory, a key appearance area, and a sound spectrum diagram;

[0040] A feature extraction module is used to extract features from the multimodal data of each video clip using the skeleton posture subnetwork, the RGB appearance subnetwork and the audio information subnetwork, so as to obtain the skeleton posture features, the appearance features and the audio features of each video clip;

[0041] A fusion module, which is used to fuse the skeleton posture features, appearance features and audio features of each video clip using an adaptive multi-stream fusion strategy and predict the anomaly score of each video clip;

[0042] The anomaly detection module is used to detect whether the anomaly score of the video clip exceeds a set threshold. If so, it is considered that the video clip contains abnormal behavior.

[0043] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0044] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0045] Beneficial effects of the present invention:

[0046] (1) The present invention trains three different sub-networks to model RGB appearance features, skeletal posture features, and audio information features respectively, thereby being able to more comprehensively extract the multimodal features of the target video, thereby enabling video anomaly detection to be performed based on more comprehensive multimodal features, thereby improving detection accuracy.

[0047] (2) In order to combine the outputs of multi-stream networks, the present invention proposes an adaptive multi-stream fusion strategy to adaptively learn the optimal fusion weights for each category, use the estimated class relationship as a regularization factor to influence the weight learning process, and strengthen the connection between categories to obtain better video anomaly detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of feature extraction of skeletal posture features provided by an embodiment of the present invention;

[0050] Figure 3 Partial visualization effect diagram of the detection method of the present invention provided in the embodiment of the present invention on HR-ShanghaiTech;

[0051] Figure 4 A normal and abnormal visualization effect diagram of the detection method of the present invention in a continuous sequence provided by an embodiment of the present invention;

[0052] Figure 5 A flowchart of a multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information provided by an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] like Figure 1As shown, the embodiment of the present invention provides a multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information, comprising the following steps:

[0056] S101: Segment the target video into a number of video segments by means of size adjustment, image normalization, and cropping. It can be understood that each video segment includes a number of continuous video frames.

[0057] S102: Preprocess each video clip to obtain multimodal data of each video clip, wherein the multimodal data includes a human skeleton trajectory, a key appearance area, and a sound spectrum graph;

[0058] S103: using the skeleton posture sub-network, the RGB appearance sub-network and the audio information sub-network to respectively extract features from the multimodal data of each video clip, so as to obtain the skeleton posture features, the appearance features and the audio features of each video clip;

[0059] S104: Adopting an adaptive multi-stream fusion strategy to fuse the skeleton posture features, appearance features, and audio features of each video clip and predicting anomaly scores of each video clip;

[0060] It can be understood that the anomaly score of each category is a fused score vector, in which each component represents the degree of anomaly of the corresponding category.

[0061] S105: If the abnormality score of the video clip exceeds the set threshold, the video clip is considered to contain abnormal behavior, otherwise it is considered that the video clip does not contain abnormal behavior. It can be understood that the threshold can be customized by the user according to the specific situation.

[0062] The multimodal video anomaly detection method that combines RGB appearance, skeletal posture features and audio information provided in the embodiment of the present invention trains three different sub-networks to model RGB appearance features, skeletal posture features and audio information features respectively, and can more comprehensively extract the multimodal features of the target video, so that video anomaly detection can be performed based on more comprehensive multimodal features, thereby improving detection accuracy.

[0063] In one embodiment, preprocessing each video clip includes: using a human posture estimation algorithm to extract human skeleton key points in each video frame in the video clip, and the human skeleton key points of all video frames constitute the human skeleton trajectory of the video clip; determining the human body rectangular area in the video frame according to the human skeleton key points extracted from each video frame in the video clip (such as the maximum and minimum values ​​of the skeleton key points), and the human body rectangular areas of all video frames constitute the appearance key area of ​​the video clip; using short-time Fourier transform to convert the audio information corresponding to the video clip into a sound spectrum diagram.

[0064] Specifically, the human skeleton key points extracted in this embodiment are 2D skeleton key points. The horizontal axis and vertical axis of the sound spectrum graph are the time scale and the frequency scale respectively.

[0065] like Figure 2 As shown, in one embodiment, an LSTM network is used as a skeleton posture subnetwork; correspondingly, the LSTM network is used to extract features of the human skeleton trajectory of each video clip to obtain the skeleton posture features of each video clip, specifically including:

[0066] For the target human body in any video frame, assume that the human body rectangular area is represented as f g =(x g ,y g ,w,h), where w,h represent the height and width of the human body rectangular area; x g ,y g Indicates the center of the rectangular area of ​​the human body;

[0067] The global posture component of the target human body in the tth video frame is calculated according to the following formula: Where t∈(1,…,T) represents the sequence number of the video frame in the video clip, and T represents the total number of video frames in the video clip;

[0068] w t =max(x i,t )-min(x i,t )

[0069] h t =max(y i,t )-min(y i,t )

[0070]

[0071]

[0072] Among them, x i,t and i,t Respectively represent the horizontal and vertical coordinates of the i-th skeleton key point of the target human body in the t-th video frame in the given coordinate system, i∈(1,…,k), k represents the total number of skeleton key points in the t-th video frame;

[0073] The local posture component of the target human body in the tth video frame is calculated according to the following formula:

[0074]

[0075]

[0076] The global posture component and the local posture component of the target human body in each video frame are spliced ​​as the skeleton posture feature of the video frame, and the skeleton posture features of all video frames in the video clip constitute the skeleton posture feature of the video clip.

[0077] It can be understood that if the local posture component of the tth video frame is expressed as and the global attitude component Then the skeletal posture feature of the tth video frame obtained by splicing the two can be expressed as

[0078] In one embodiment, a spatiotemporal generative adversarial network such as that disclosed in CN117593684A is used as an RGB appearance subnetwork. The spatiotemporal generative adversarial network extracts the RGB appearance features of each video frame and predicts the next frame. Correspondingly, the spatiotemporal generative adversarial network is used to extract features of the key appearance areas of each video clip to obtain the appearance features of each video clip, specifically including: inputting the continuous video frames contained in the video clip into the spatiotemporal generative adversarial network, continuously reducing the size by downsampling in the encoding stage, extracting the spatial structure and local detail features; restoring the size by upsampling in the decoding stage, and outputting the final result.

[0079] Specifically, in order to perform spatiotemporal modeling of the RGB appearance features of consecutive video frames, a spatiotemporal Transformer module is proposed, which uses a multi-head attention mechanism to fuse patches of the same space at different time scales, and outputs the feature Z after the scale changes. a ; A multi-scale channel module is proposed to concatenate the features of the encoding stage and the decoding stage, and output the fused multi-scale feature Z b ; Z a With Z b Splicing, delete the pooling layer in the original generator, upsample through the stride of the convolution layer, gradually expand the receptive field to restore the image size, all convolution blocks contain 3D-Batchnorm layer and ReLU layer, the last convolution block adds a Tanh activation layer and sets the padding of its time dimension to 0 to ensure that the extracted appearance features have the same dimension as the skeleton posture features.

[0080] In one embodiment, a convolutional neural network is used as the audio information subnetwork.

[0081] In one embodiment, in order to combine the outputs of multi-stream networks, an adaptive multi-stream fusion strategy is proposed to adaptively learn the optimal fusion weights for each category, use the estimated class relationship as a regularization factor to influence the weight learning process, strengthen the connection between categories, and automatically identify highly related categories to obtain better video anomaly detection performance.

[0082] Specifically, the adaptive multi-stream fusion strategy includes:

[0083] For each class c and stream m, define the prediction score vector is the feature vector output by the sub-network of the corresponding modality; for the training video sample n, the prediction score vectors from all streams are stacked to form a fusion vector Among them, C is the total number of categories (in this embodiment, the categories of video samples are divided into two categories: normal and abnormal, that is, C=2), and M is the total number of streams (in this embodiment, it is divided into three feature streams: skeleton posture, appearance, and audio, that is, M=3).

[0084] Based on the above content, this embodiment expresses the optimal fusion weight matrix W of the three feature streams as:

[0085]

[0086] Among them, y n,c represents the true label of training video sample n for category c, w c Represents the weight vector of category c, specifically, the weight vector w1 for abnormal situations and the weight vector w2 for normal situations.

[0087] In order to achieve self-adaptation, this embodiment continuously optimizes the parameters of the entire network by minimizing the following loss function, thereby learning and obtaining the optimal fusion weight matrix W.

[0088]

[0089] Among them, S represents the prediction scores corresponding to all training video samples in a training batch, Y represents the true labels corresponding to all training video samples in a training batch, V = [V 1 ,...,V m ,...,V M ] T Represents the class relationship matrix, which is used to regularize the learning process of the weight matrix W, V m Represents the class relationship vector corresponding to the stream m of all training video samples in a training batch. The element V in the class relationship vector i,j It represents the proportion of category i being misclassified as category j, and λ1 and λ2 are weight parameters.

[0090] In order to verify the effectiveness of the solution of the present invention, the present invention also provides the following experimental data.

[0091] The present invention takes into account the complexity of abnormal behavior, the ambiguity of its definition, and the uneven distribution of positive and negative samples. The present invention performs abnormal behavior detection in a binary classification manner, that is, distinguishing whether a specific segment in the target video data set contains abnormal and normal behaviors. Correspondingly, the present invention uses the AUC at the frame level to evaluate the performance of anomaly detection, and the higher the AUC value, the better the detection effect. Generally speaking, when the AUC value is greater than 0.5, the performance of the model is considered acceptable, and when the AUC value is close to 1, the performance of the model is even better. In a video frame, if one or more abnormal events occur, the frame will be marked as an abnormal frame.

[0092] The accuracy of the detection framework was calculated and compared with other state-of-the-art methods. The entire experiment was conducted on NVIDIA RTX 3070ti, and the model was built using the PyTorch framework. The AUCs of different methods were compared on the ShanghaiTech, CUHK Avenue, and Corridor datasets, respectively. In order to better demonstrate that the present invention has higher accuracy and robustness in identifying abnormal behaviors related to humans, some irrelevant data were screened and eliminated to form datasets that specifically focus on human behavior, including HR Avenue, HR ShanghaiTech, and HR Corridor. As shown in Table 1, the method of the present invention is ahead of other state-of-the-art mainstream methods on multiple datasets.

[0093] Table 1 AUC (%) comparison between the present invention and other methods

[0094]

[0095] like Figure 3 As shown in , by effectively fusing appearance features and skeleton features, we can effectively identify the interaction between people and objects (such as riding a bicycle), thereby improving the detection accuracy. Figure 4 As shown, two example sequences from the same scene are displayed. It can be seen that: for abnormal behavior, the present invention predicts skeleton joints according to the normal mode and visualizes the predicted skeleton results. It can be seen from the figure that for abnormal behavior, the predicted skeleton is significantly different from the historical skeleton; for normal behavior, the similarity between the predicted skeleton and the historical skeleton is significantly improved.

[0096] like Figure 5 As shown, an embodiment of the present invention also provides a multimodal video anomaly detection device that combines RGB appearance, skeleton posture and audio information, including a video segmentation module, a preprocessing module, a feature extraction module, a fusion module and anomaly detection module.

[0097] The video segmentation module is used to segment the target video into several video clips; the preprocessing module is used to preprocess each video clip to obtain multimodal data of each video clip, and the multimodal data includes human skeleton trajectory, appearance key area and sound spectrum diagram; the feature extraction module is used to use the skeleton posture subnetwork, RGB appearance subnetwork and audio information subnetwork to extract features of the multimodal data of each video clip respectively, and obtain the skeleton posture features, appearance features and audio features of each video clip; the fusion module is used to adopt an adaptive multi-stream fusion strategy to fuse the skeleton posture features, appearance features and audio features of each video clip and predict the anomaly score of each video clip; the anomaly detection module is used to detect whether the anomaly score of the video clip exceeds the set threshold. If so, it is considered that the video clip contains abnormal behavior.

[0098] It should be noted that the multimodal video anomaly detection device provided in the embodiment of the present invention is to implement the above method. Its specific functions can be referred to the above method embodiments, which will not be repeated here.

[0099] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor (processor) 601, a communication interface (Communications Interface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call the logic instructions in the memory 603 to execute a multimodal video anomaly detection method that combines RGB appearance, skeletal posture and audio information, the method comprising: dividing the target video into a plurality of video clips; preprocessing each video clip to obtain multimodal data of each video clip, the multimodal data comprising a human skeleton trajectory, a key area of ​​appearance and a sound spectrum diagram; using a skeletal posture subnetwork, an RGB appearance subnetwork and an audio information subnetwork to perform feature extraction on the multimodal data of each video clip respectively to obtain skeletal posture features, appearance features and audio features of each video clip; using an adaptive multi-stream fusion strategy to fuse the skeletal posture features, appearance features and audio features of each video clip and predict the anomaly score of each video clip; if the anomaly score of a video clip exceeds a set threshold, the video clip is considered to contain abnormal behavior.

[0100] In addition, when the logic instructions in the above-mentioned memory 603 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0101] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: dividing the target video into several video clips; preprocessing each video clip to obtain multimodal data of each video clip, and the multimodal data includes human skeleton trajectory, appearance key area and sound spectrum diagram; using a skeleton posture subnetwork, an RGB appearance subnetwork and an audio information subnetwork to respectively extract features of the multimodal data of each video clip to obtain the skeleton posture features, appearance features and audio features of each video clip; using an adaptive multi-stream fusion strategy to fuse the skeleton posture features, appearance features and audio features of each video clip and predict the anomaly score of each video clip; if the anomaly score of the video clip exceeds a set threshold, the video clip is considered to contain abnormal behavior.

[0102] An embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods provided by the above-mentioned method embodiments are implemented, for example, including: dividing the target video into several video clips; preprocessing each video clip to obtain multimodal data of each video clip, wherein the multimodal data includes a human skeleton trajectory, a key area of ​​appearance, and a sound spectrum diagram; using a skeleton posture subnetwork, an RGB appearance subnetwork, and an audio information subnetwork to perform feature extraction on the multimodal data of each video clip respectively to obtain the skeleton posture features, appearance features, and audio features of each video clip; using an adaptive multi-stream fusion strategy to fuse the skeleton posture features, appearance features, and audio features of each video clip and predict the anomaly score of each video clip; if the anomaly score of a video clip exceeds a set threshold, the video clip is considered to contain abnormal behavior.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information, characterized in that: include: Step 1: Divide the target video into several video segments; Step 2: Preprocess each video clip to obtain multimodal data of each video clip, wherein the multimodal data includes a human skeleton trajectory, a key appearance area, and a sound spectrum graph; Step 3: Use the skeleton posture sub-network, RGB appearance sub-network and audio information sub-network to extract features from the multimodal data of each video clip, and obtain the skeleton posture features, appearance features and audio features of each video clip; Step 4: Adopt an adaptive multi-stream fusion strategy to fuse the skeleton posture features, appearance features, and audio features of each video clip and predict the anomaly score of each video clip; Step 5: If the anomaly score of the video clip exceeds a set threshold, the video clip is considered to contain abnormal behavior.

2. According to claim 1, a multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information is characterized in that: In step 2, preprocessing each video clip includes: A human body posture estimation algorithm is used to extract the key points of the human skeleton in each video frame in the video clip, and the key points of the human skeleton in all video frames constitute the human skeleton trajectory of the video clip; Determine a human rectangular region in the video frame according to the human skeleton key points extracted from each video frame in the video segment, and the human rectangular regions in all video frames constitute the appearance key region of the video segment; The audio information corresponding to the video clip is converted into a sound spectrogram using short-time Fourier transform.

3. The multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information according to claim 1 is characterized in that: In step 3, the LSTM network is used as the skeleton posture subnetwork; Correspondingly, the LSTM network is used to extract the features of the human skeleton trajectory of each video clip to obtain the skeleton posture features of each video clip, including: The global pose component of the target human body in each video frame is calculated according to the following formula: Where t∈(1,…,T) represents the sequence number of the video frame in the video clip, and T represents the total number of video frames in the video clip; w t =max(x i,t )-min(x i,t ) h t =max(y i,t )-min(y i,t ) Among them, x i,t and i,t Respectively represent the horizontal and vertical coordinates of the i-th skeleton key point of the target human body in the t-th video frame in the given coordinate system, i∈(1,…,k), k represents the total number of skeleton key points in the t-th video frame; The local posture component of the target human body in each video frame is calculated according to the following formula: The global posture component and the local posture component of the target human body in each video frame are spliced ​​as the skeleton posture feature of the video frame, and the skeleton posture features of all video frames in the video clip constitute the skeleton posture feature of the video clip.

4. The multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information according to claim 1 is characterized in that: In step 3, a spatiotemporal generative adversarial network is used as the RGB appearance sub-network.

5. The multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information according to claim 1 is characterized in that: In step 3, a convolutional neural network is used as the audio information subnetwork.

6. A multimodal video anomaly detection method combining RGB appearance, skeleton posture and audio information according to any one of claims 1 to 5, characterized in that: The adaptive multi-stream fusion strategy specifically includes: For each class c and stream m, define the prediction score vector is the feature vector output by the sub-network of the corresponding modality. For the training video sample n, the prediction score vectors from all streams are stacked to form a fusion vector Where C is the total number of categories and M is the total number of flows; The optimal fusion weight matrix W is expressed as: Among them, y n,c represents the true label of training video sample n for category c, w c Represents the weight vector of category c; Minimize the following loss function to optimize the parameters of the entire network, so as to learn the optimal fusion weight matrix W; Among them, S represents the prediction scores corresponding to all training video samples in a training batch, Y represents the true labels corresponding to all training video samples in a training batch, V = [V 1 ,...,V m ,...,V M ] T Represents the class relationship matrix, which is used to regularize the learning process of the weight matrix W, V m Represents the class relationship vector corresponding to the stream m of all training video samples in a training batch. The element V in the class relationship vector i,j It represents the proportion of category i being misclassified as category j, and λ1 and λ2 are weight parameters.

7. A multimodal video anomaly detection device combining RGB appearance, skeleton posture and audio information, characterized in that: include: A video segmentation module is used to segment the target video into several video segments; A preprocessing module, used for preprocessing each video clip to obtain multimodal data of each video clip, wherein the multimodal data includes a human skeleton trajectory, a key appearance area, and a sound spectrum diagram; A feature extraction module is used to extract features from the multimodal data of each video clip using the skeleton posture subnetwork, the RGB appearance subnetwork and the audio information subnetwork, so as to obtain the skeleton posture features, the appearance features and the audio features of each video clip; A fusion module, which is used to fuse the skeleton posture features, appearance features and audio features of each video clip using an adaptive multi-stream fusion strategy and predict the anomaly score of each video clip; The anomaly detection module is used to detect whether the anomaly score of the video clip exceeds a set threshold. If so, it is considered that the video clip contains abnormal behavior.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Video anomaly detection method of space-time Transform fused with multi-scale attention

    CN117593684A

  • Abnormal action determination method and device, electronic equipment and storage medium

    CN113392742A

  • Human body abnormal behavior detection method combining appearance texture and motion skeleton

    CN115601841A

  • Network traffic anomaly detection algorithm based on multi-modal feature fusion

    CN118353660A

  • Method and system for detecting abnormal behavior of pedestrian, and terminal and storage medium

    WO2022067606A1

Cited By

  • Space interaction accurate identification method and system based on multi-modal fusion

    CN120234654A

  • Spatial interaction precise recognition method and system based on multimodal fusion

    CN120234654B