Abnormal behavior detection method and system based on cross-modal fusion

Abstract

The invention provides an abnormal behavior detection method and system based on cross-modal fusion, belongs to the technical field of abnormal behavior detection, and processes acquired video data by using a pre-trained abnormal behavior detection model to obtain an abnormal behavior detection result. The abnormal behavior detection model comprises a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network. According to the method, a new cross-modal fusion algorithm is introduced, the fusion of audio features and visual features is realized in the time dimension, and audio information is utilized to further distinguish visual dimension fuzzy events; according to the method, two branches based on hyperbolic graph convolution are designed, the similarity and the time incidence relation of the features are processed respectively, the distinction degree of the features is improved in the hyperbolic space, and the accuracy of abnormal behavior detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal behavior detection, and in particular to an abnormal behavior detection method and system based on cross-modal fusion. Background Art

[0002] In recent years, with the popularity of intelligent surveillance camera systems, the demand for intelligent systems that can automatically and in real time analyze video surveillance has increased in all walks of life. In complex and crowded scenes (such as markets, shopping malls, hospitals, banks, streets, educational institutions, urban administrative offices, etc.), automatic detection of abnormal events is one of the most challenging tasks in modern intelligent video surveillance systems. In most cases, timely and accurate detection of abnormal behavior is the main goal of anomaly detection. Anomalies in videos usually include abnormal behaviors and abnormal entities. Abnormal behaviors refer to riots, fighting, trampling, and shootings, while abnormal entities refer to people with suspicious behaviors and controlled knives in public places. Abnormal entities and activities should be detected and alarmed by intelligent security systems in a timely and automatic manner. However, due to the ambiguity of the definition of anomalies, the diversity of human behaviors, and the complexity of environmental conditions, the task of video anomaly and abnormal behavior detection is challenging.

[0003] Although the research on abnormal behavior detection based on surveillance video has been carried out for many years, and many excellent solutions have emerged, it is still a very challenging task to design and optimize a deep learning network model with high recognition accuracy, strong universality and good performance. Unlike conventional detection tasks, abnormal behavior detection involves dealing with a small number of unpredictable and rare events, which brings the following difficulties to the design of deep learning models: 1) Unknown: Abnormal situations often involve many unknown factors, such as some instances with unknown behaviors, structures and distributions, which remain unknown until they actually occur, such as abnormal situations in complex places, new abnormal attacks, etc. In addition, abnormal behaviors are usually irregular, and different types of abnormalities usually show completely different characteristics. For example, in surveillance videos, abnormal events such as major traffic accidents, robberies and burglaries have obvious differences in visual dimensions. 2) Rarity: The probability of abnormal behavior occurring in daily life is much lower than the probability of normal behavior. Therefore, it is quite difficult to collect a large amount of labeled abnormal data. In addition, most of the videos in the dataset used to train deep learning models are composed of normal clips, which will result in far fewer abnormal events than normal events during the model training process, making the model's ability to fit abnormal events worse. 3) Diversity: The situations of abnormal events in different scenarios are diverse, such as environmental conditions (lighting changes, shadow effects of objects, object occlusion, complex backgrounds, etc.), crowd density, noise interference, complexity of human behavior, camera settings, etc., which will affect the extraction of visual features, making it more difficult for the model to locate and identify abnormal events. Therefore, the exploration of abnormal behavior detection is still a very challenging topic in the current industry and academia, and it also has extraordinary practical and general value.

[0004] Existing abnormal behavior detection algorithms are mainly divided into fully supervised abnormal behavior detection algorithms, weakly supervised abnormal behavior detection algorithms, semi-supervised abnormal behavior detection algorithms, and unsupervised abnormal behavior detection algorithms. Fully supervised learning has a higher accuracy for video abnormal behavior detection because the method can learn more accurate patterns from labeled normal and abnormal data, thereby achieving better detection results in the training stage. In addition, the fully supervised learning method needs to rely on labeled training data, so it can be applied in various application scenarios as long as there is enough training data available. However, due to the high cost of label acquisition for fully supervised learning, it may face the problem of lack of relevant data sets. In addition, when training data, the method may overfit, thereby reducing the generalization ability on unknown data. Weakly supervised abnormal behavior detection only requires rough annotation of the data set, which can improve the efficiency of data utilization and help the model learn more generalized feature representations instead of over-relying on detailed annotated video data, thereby improving the generalization ability of the model. In addition, weakly supervised learning models are usually more robust and can better handle noise and incomplete annotations, making them more suitable for application in practical scenarios. Compared with weakly supervised learning methods, semi-supervised abnormal behavior detection methods have lower requirements for data annotation because the training set only contains normal data. Abnormal behavior detection can be completed by simply identifying data that is different from the normal data learned in the training set as abnormal. However, during the training process, this method only learns the characteristics of abnormal data and it is difficult to obtain the differences between the two types of data, so it may cause more false positives. There are relatively few unsupervised abnormal behavior detection methods. Such methods lack supervised information, resulting in poor ability to distinguish abnormal behaviors, which may lead to a decrease in algorithm performance.

[0005] In summary, after investigation, we found that the current mainstream abnormal behavior detection algorithms have the following main defects:

[0006] (1) Many early anomaly detection systems rely on a single modality, focusing primarily on video information, such as the analysis of the motion or appearance features of people in the video. However, these methods ignore the importance of audio information. Audio in audio and video, such as screams, gunshots, and explosions, is also very important for the model to identify abnormal behavior.

[0007] (2) The extracted features lack robustness. Traditional anomaly detection methods rely on hand-crafted features, such as HOG and SIFT, which require specific expertise and a lot of debugging work. Therefore, this method has poor robustness, and the model trained with such features may not be able to adapt to the diversity and complexity of video data.

[0008] (3) Poor context association ability. Abnormal events usually involve rich context information, which requires comprehensive processing of related audio and video information. However, many existing methods do not effectively utilize this multimodal context association. At the same time, in multimodal videos, the temporal synchronization of audio and visual information is a challenge. Existing methods have not effectively solved this problem, making it difficult for the model to accurately associate audio and visual events. Summary of the invention

[0009] The object of the present invention is to provide an abnormal behavior detection method and system based on cross-modal fusion to solve at least one technical problem existing in the above-mentioned background technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect, the present invention provides an abnormal behavior detection method based on cross-modal fusion, comprising:

[0012] Obtain the video data to be detected;

[0013] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network, and a behavior prediction network; the visual feature extraction network is used to extract visual features from video data; the audio feature extraction network is used to extract audio features from video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space so as to better understand and identify abnormal behaviors in the video. The behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0014] Furthermore, the attention mechanism is used to combine audio and visual features to capture the global correlation between the two. The global attention map is constructed as follows:

[0015]

[0016] Where A represents the global attention map, Q a =X a W Q is the result of linear change of audio features, K v =X v W k is the result of linear changes in video features, W Q and W k are all learnable parameter matrices, D is the dimension of the hidden layer, Represents the similarity between the audio feature vector and the video feature vector.

[0017] Furthermore, the weight value is dynamically adjusted according to the time distance between each video clip to strengthen or weaken the feature information at a specific time point, which is expressed as:

[0018]

[0019] Where R ij represents the influence weight of the j-th segment on the i-th segment, w and b represent the weight parameter and bias term respectively.

[0020] Furthermore, the audio features and video features are fused to generate enhanced video features, which are expressed as:

[0021]

[0022]

[0023] Among them, X v is the original visual feature, W f and W v is a trainable weight matrix.

[0024] Furthermore, in the feature similarity branch, in order to learn the similarity relationship between features, the fused features obtained from the cross-modal fusion module are first projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space. The adjacency matrix is ​​defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the node, a hyperbolic graph convolution operation is performed so that each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips.

[0025] Furthermore, in the temporal correlation branch, the feature vector is projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space, and a temporal relationship matrix is ​​constructed; wherein the temporal relationship between the video clips is represented in the temporal relationship matrix, and the absolute time difference between the clips is calculated, that is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater; after obtaining the temporal relationship matrix, a hyperbolic graph convolution operation is applied to update the feature vector of each node, and update the feature representation of the current node.

[0026] In a second aspect, the present invention provides an abnormal behavior detection system based on cross-modal fusion, comprising:

[0027] An acquisition module, used for acquiring video data to be detected;

[0028] The prediction module is used to process the acquired video data using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate an enhanced Video features; The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space so as to better understand and identify abnormal behaviors in the video; The behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behaviors.

[0029] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the abnormal behavior detection method based on cross-modal fusion as described in the first aspect is implemented.

[0030] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the abnormal behavior detection method based on cross-modal fusion as described in the first aspect.

[0031] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the abnormal behavior detection method based on cross-modal fusion as described in the first aspect.

[0032] Terminology explanation:

[0033] Cross-modal fusion: Cross-modal fusion technology refers to the fusion of audio features and visual features of a video, realizing multimodal fusion of the two features in the time dimension, and using audio information to further distinguish events that are blurred in the visual dimension.

[0034] Hyperbolic Graph Convolutional Network: is a deep learning model for processing graph data, especially data in non-Euclidean space. In hyperbolic space, the distance metric is different from that in Euclidean space, which can better capture the relationship between hierarchical structures. The core idea of ​​hyperbolic graph convolution is to perform convolution operations in hyperbolic space so that the model can better capture the nonlinear structure of graph data.

[0035] Abnormal behavior detection: refers to the use of technical means to identify and monitor abnormal behaviors in scenes. This technology is usually used in public places, prisons, schools and other places to detect and respond to abnormal events in a timely manner and ensure people's safety. Abnormal behavior detection usually includes multiple steps such as data collection, feature extraction, classification and detection, as well as early warning and response. In addition, abnormal behavior detection technology can be combined with a variety of sensors and data sources, such as surveillance cameras, microphones and sensors, to improve the accuracy and timeliness of detection. At the same time, real-time data analysis and pattern recognition technology can be used to continuously optimize the detection algorithm and improve the performance and reliability of the system. The application of abnormal behavior detection technology can help prevent the occurrence of abnormal events and ensure public safety.

[0036] The beneficial effects of the present invention are as follows: a new cross-modal fusion algorithm is introduced to realize the fusion of audio features and visual features in the time dimension, and audio information is used to further distinguish events that are blurred in the visual dimension; two branches based on hyperbolic graph convolution are designed to respectively process the similarity and time correlation of features, and the discrimination of features in the hyperbolic space is improved, thereby improving the accuracy of abnormal behavior detection.

[0037] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0039] Figure 1 This is a functional framework diagram of the cross-modal fusion and hyperbolic graph convolution abnormal behavior detection model based on weakly supervised learning described in an embodiment of the present invention.

[0040] Figure 2 The figure is an overall structural diagram of the abnormal behavior detection algorithm described in the embodiment of the present invention.

[0041] Figure 3 Schematic diagram of the cross-modal fusion module described in an embodiment of the present invention.

[0042] Figure 4 This is a structural diagram of the cross-modal fusion module described in an embodiment of the present invention.

[0043] Figure 5 Schematic diagram of feature similarity branches of the hyperbolic graph convolution module described in an embodiment of the present invention.

[0044] Figure 6 Schematic diagram of the time-correlated branches of the hyperbolic graph convolution module described in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0046] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0047] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0048] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0049] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0050] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0051] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0052] The present invention provides an abnormal behavior detection algorithm based on cross-modal fusion and hyperbolic graph convolution, which can be applied to smart cities, smart transportation, intelligent security systems, Internet live broadcast content review, film and television work review, etc. The method includes: using an attention mechanism to extract a global attention map of audio features and video features to measure the degree of correlation between the two features, then realizing local awakening through adaptive position priors, dynamically adjusting weight values ​​according to the time distance between each video clip, so as to strengthen or weaken the feature information at a specific time point; then realizing cross-modal fusion, the purpose is to fuse audio features and video features to generate enhanced video features. Using the two branches of the hyperbolic graph convolution, the obtained fusion features are further processed, and the feature similarity branch and the temporal correlation branch are used to help the model effectively learn rich feature representations from multimodal data. Finally, a multi-instance learning algorithm is used to obtain the abnormal score of the video clip to complete a specific target task. The present invention uses the global feature extraction capability of the attention mechanism to complete the cross-modal fusion of video features and audio features by involving a new modal fusion framework. Specifically, we designed a cross-modal fusion module to optimize the modal fusion effect, and used the auxiliary role of audio information in abnormal event recognition to enhance the model's ability to distinguish between normal and abnormal events. In order to further improve the distinguishability of features, we designed two hyperbolic graph convolution branches, which are used to capture the complex similarity relationship between video clips and the temporal relationship between video clips. Combining the above two modules, we adopted a weakly supervised learning method to realize the model's recognition of abnormal video clips.

[0053] Example 1

[0054] In this embodiment 1, firstly, an abnormal behavior detection system based on cross-modal fusion is provided, including: an acquisition module, used to acquire video data to be detected; a prediction module, used to process the acquired video data using a pre-trained abnormal behavior detection model to obtain an abnormal behavior detection result.

[0055] The abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from video data; the audio feature extraction network is used to extract audio features from video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features; the hyperbolic graph convolution network includes feature similarity branches and The time correlation branch, after the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space so as to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0056] In this embodiment, the above system is used to implement abnormal behavior detection based on cross-modal fusion, including: using an acquisition module to acquire video data to be detected, and then using a prediction module to process the acquired video data using a pre-trained abnormal behavior detection model to obtain an abnormal behavior detection result.

[0057] The cross-modal fusion network combines audio and visual features using the attention mechanism to capture the global correlation between the two. The global attention map is constructed as follows:

[0058]

[0059] Where A represents the global attention map, Q a =X a W Q is the result of linear change of audio features, K v =X v W k is the result of linear changes in video features, W Q and W k are all learnable parameter matrices, D is the dimension of the hidden layer, Represents the similarity between the audio feature vector and the video feature vector.

[0060] The weight value is dynamically adjusted according to the time distance between each video clip to strengthen or weaken the feature information at a specific time point, which is expressed as:

[0061] R ij =exp(-|w(ij) 2 +b|)

[0062] Where R ij represents the influence weight of the j-th segment on the i-th segment, w and b represent the weight parameter and bias term respectively.

[0063] The audio features and video features are fused to generate enhanced video features, which can be expressed as:

[0064]

[0065]

[0066] Among them, X v is the original visual feature, W f and W v is a trainable weight matrix.

[0067] In the feature similarity branch, in order to learn the similarity relationship between features, the fused features obtained from the cross-modal fusion module are first projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space. The adjacency matrix is ​​defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the node, a hyperbolic graph convolution operation is performed so that each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips.

[0068] In the temporal correlation branch, the feature vector is projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space and construct a temporal relationship matrix; wherein, the temporal relationship between the video clips is represented in the temporal relationship matrix, and the absolute time difference between the clips is calculated, that is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater; after obtaining the temporal relationship matrix, the hyperbolic graph convolution operation is applied to update the feature vector of each node and update the feature representation of the current node.

[0069] In summary, this embodiment proposes an abnormal behavior detection algorithm based on cross-modal fusion and hyperbolic graph convolution, which is an audio-guided multimodal anomaly detection framework. After obtaining the fused features by fusing audio features and visual features in the time dimension, they are processed by hyperbolic graph convolution to improve the model discrimination, thereby improving the accuracy of abnormal behavior detection.

[0070] Example 2

[0071] Most of the existing work focuses on the analysis of visual information, while ignoring the auxiliary role of audio information. Compared with the input of a single modality, the input of multimodal information is more meaningful for improving the performance of abnormal behavior detection algorithms. In general, abnormal behavior can be accurately located using visual information, but accurate judgments cannot be made by relying solely on visual information. The effective use of audio information can further distinguish events that are blurred in the visual dimension, thereby improving algorithm performance. Therefore, in this embodiment, an audio-guided multimodal anomaly detection method is proposed, which realizes the multimodal fusion of audio features and visual features in the time dimension, and then inputs the fused features into the hyperbolic graph convolution module, using two branches to process feature similarity and time correlation respectively, and improves the distinguishability of features in the hyperbolic space, thereby improving the accuracy of abnormal behavior detection.

[0072] Figure 1 As shown, in this embodiment, an abnormal behavior detection framework based on cross-modal fusion of weakly supervised learning is proposed. In this detection framework, the visual features and audio features of the video are first extracted using the I3D (Inflated 3D) network and the VGGish network, respectively. These features are further optimized through the cross-modal fusion module and the hyperbolic graph convolutional network module, and then the score of abnormal behavior is obtained using the multi-instance learning algorithm. This framework can fuse audio and video features, improve the separability of features through the hyperbolic graph convolutional network, avoid background noise, and thus enhance the model's ability to recognize abnormal behavior.

[0073] Specifically, the cross-modal fusion module mainly consists of the following operations: 1) generating a global attention map; 2) local awakening; 3) cross-modal fusion. Generating a global attention map is a key step in this module, which aims to combine audio and visual features to capture the global correlation between the two. The local awakening operation is achieved through adaptive position priors, and the weight value can be dynamically adjusted according to the temporal distance between each video clip to strengthen or weaken the feature information at a specific time point. The cross-modal fusion operation is to fuse audio features with video features to generate enhanced video features.

[0074] The hyperbolic graph convolution module mainly consists of two branches: 1) feature similarity branch; 2) temporal correlation branch. After the feature similarity branch is operated through multiple layers of hyperbolic graph convolution, each node integrates the information of its neighboring nodes, thereby helping the model capture the complex similarity relationship between video clips. The goal of the temporal correlation branch is to model the temporal relationship between video clips in the hyperbolic space in order to better understand and identify abnormal behaviors in the video.

[0075] In this embodiment, Figure 2 As shown, the abnormal behavior detection method based on cross-modal fusion of weakly supervised learning specifically includes the following steps:

[0076] Step S1: Construct a cross-modal fusion module, which mainly uses the attention mechanism to fuse multi-modal features. The specific structure is as follows Figures 3 to 5 The main steps are as follows:

[0077] Step S1-1: Generate global attention map

[0078] After obtaining the video features and audio features output by the 13D model and the VGGish model, the cross-modal fusion module fuses the multimodal features. The attention mechanism is used to combine audio and visual features to capture the global correlation between the two. The construction of the global attention map can be expressed as:

[0079]

[0080] Where A represents the global attention map, Q a =X a W Q is the result of linear change of audio features, K v =X v W k is the result of linear changes in video features, W Q and W k are all learnable parameter matrices, whose purpose is to project the corresponding features into a suitable space for calculating attention scores. D is the dimension of the hidden layer, which is used to stabilize the scores of the attention mechanism and prevent problems such as gradient explosion or disappearance during training. Represents the similarity between the audio feature vector and the video feature vector. The result is an attention score that measures the degree of association between audio and video features.

[0081] Step S1-2: Local wake-up operation

[0082] This process is achieved through adaptive position priors, which can dynamically adjust the weight value according to the time distance between each video clip to strengthen or weaken the feature information at a specific time point. The operation can be expressed as:

[0083] R ij =exp(-|w(ij) 2 +b|)

[0084] Where R ij represents the influence weight of the j-th segment on the ith segment, w and b represent the weight parameter and bias term respectively. The former controls the influence degree of the position prior. The larger the parameter is, the stronger the influence of the position prior is and the more obvious the effect of local wake-up is. The latter is used to adjust the influence position of the position prior and further fine-tune the effect of the local wake-up operation.

[0085] Step S1-3: Cross-modal fusion operation

[0086] The cross-modal fusion operation is the last step of this module. Its purpose is to fuse audio features and video features to generate enhanced video features. The specific details can be expressed as:

[0087]

[0088]

[0089] Where X v is the original visual feature, W f and W v is a trainable weight matrix that maps audio and video features to a specific feature space, and then adds an adaptive position prior to the global attention map. In addition, in order to ensure the original distribution of visual features, it is necessary to v Finally, in order to stabilize the model training process and prevent gradient explosion, the features need to be normalized.

[0090] Step S2: Hyperbolic Graph Convolution Module

[0091] After the operation of step S1, an enhanced visual feature can be obtained, which is supplemented by audio information in the time dimension, thereby helping the model to better capture multimodal information related to abnormal behaviors. Next, the two branches of the hyperbolic graph convolution are used to learn feature similarity and temporal correlation respectively, helping the model to effectively learn rich feature representations from multimodal data.

[0092] Step S2-1: Feature similarity branch

[0093] In order to learn the similarity relationship between features, we first introduce the feature similarity branch, such as Figure 6 As shown, first, the fusion features obtained from the cross-modal fusion module Projecting into the hyperbolic space through exponential mapping, we can get the feature representation of the feature in the hyperbolic space. Next, we define the adjacency matrix by calculating the feature similarity in hyperbolic space. The calculation method is as follows:

[0094]

[0095] The feature similarity is obtained by calculating the Lorentz distance of the features in the hyperbolic space, and the calculation method is as follows:

[0096]

[0097] Before the introduction of the activation function, a thresholding operation was used to eliminate the weak relationship between features in the hyperbolic space and strengthen the correlation between more similar features:

[0098]

[0099] Here, τ is a pre-set threshold used to distinguish whether there is a strong similarity between nodes. It can remove connections with poor correlation when composing a graph, which helps the graph convolutional network to learn the feature similarity between nodes more effectively.

[0100] After obtaining the adjacency matrix of the node, the next step is to perform a hyperbolic graph convolution operation:

[0101]

[0102]

[0103] It means that a hyperbolic transformation is performed on the i-th node of the l-1th layer. After multiple layers of transformation operations, the i-th node of the lth layer integrates the information of all adjacent nodes.

[0104] After the feature similarity branch is operated through multiple layers of hyperbolic graph convolution, each node integrates the information of its neighboring nodes, thereby helping the model capture the complex similarity relationship between video clips. Finally, in order to increase the nonlinear relationship, the output value needs to be operated as follows:

[0105]

[0106] Step S2-2: Time-dependent branching

[0107] The goal of the temporal relevance branch is to model the temporal relationships between video clips in a hyperbolic space in order to better understand and identify anomalous and abnormal behaviors in videos.

[0108] Similar to the similarity branch, the temporal correlation branch also projects the feature vector into the hyperbolic space through exponential mapping to obtain the feature representation in the hyperbolic space. The next step is to construct the temporal relationship matrix:

[0109]

[0110] The temporal relationship matrix represents the temporal relationship between video clips and calculates the absolute time difference between clip i and clip j. That is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater.

[0111] After obtaining the temporal relationship matrix, the next step is to apply the hyperbolic graph convolution operation to update the feature vector of each node. This operation takes into account the adjacent nodes that have a temporal relationship with the current node and updates the feature representation of the current node:

[0112]

[0113] Through the temporal correlation branch, the model can effectively capture the temporal relationship between video clips in the hyperbolic space, making it easier to understand the dynamic changes in the video.

[0114] Step S3: Behavior prediction

[0115] After obtaining the output results of the feature similarity branch and the temporal correlation branch, the next step is behavior prediction:

[0116]

[0117] First, the results of the two branches are concatenated to form a joint feature representation. Next, the similarity between the concatenated features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0118] In summary, in this embodiment, by introducing the cross-modal fusion module and the hyperbolic graph convolution module, the model's ability to identify abnormal behaviors is improved, and it can be put into practical use in the future. Considering the use of audio and visual modal data for the identification of abnormal behaviors, compared with the input of a single modality, the input of multimodal information is more meaningful for improving the performance of the abnormal behavior detection algorithm. An innovative cross-modal fusion module is proposed to fuse the two modal information, realizing the multimodal fusion of audio features and visual features in the time dimension, and taking advantage of the hyperbolic space, using two branches to process feature similarity and time correlation.

[0119] Example 3

[0120] This embodiment 3 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned abnormal behavior detection method based on cross-modal fusion is implemented. The method includes:

[0121] Obtain the video data to be detected;

[0122] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network, and a behavior prediction network; the visual feature extraction network is used to extract visual features from video data; the audio feature extraction network is used to extract audio features from video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space so as to better understand and identify abnormal behaviors in the video. The behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0123] Example 4

[0124] This embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, the processor calls the program instructions to execute the above-mentioned abnormal behavior detection method based on cross-modal fusion, the method comprising:

[0125] Obtain the video data to be detected;

[0126] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network, and a behavior prediction network; the visual feature extraction network is used to extract visual features from video data; the audio feature extraction network is used to extract audio features from video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space so as to better understand and identify abnormal behaviors in the video. The behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0127] Example 5

[0128] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the abnormal behavior detection method based on cross-modal fusion as described above, and the method includes:

[0129] Obtain the video data to be detected;

[0130] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network, and a behavior prediction network; the visual feature extraction network is used to extract visual features from video data; the audio feature extraction network is used to extract audio features from video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space so as to better understand and identify abnormal behaviors in the video. The behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0131] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0135] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. An abnormal behavior detection method based on cross-modal fusion, characterized in that: include: Obtain the video data to be detected; The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from video data; the audio feature extraction network is used to extract audio features from video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features; the hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is operated through multiple layers of hyperbolic graph convolution, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch is to model the temporal relationship between video clips in the hyperbolic space, so as to better understand and identify abnormal behaviors in the video; The behavior prediction network is used to concatenate the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and to represent the similarity between the concatenated features and the weight matrix by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

2. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: The attention mechanism is used to combine audio and visual features to capture the global correlation between the two; the global attention map is constructed as follows: Where A represents the global attention map, Q a =X a W Q is the result of linear change of audio features, K v =X v W k is the result of linear changes in video features, W Q and W k are all learnable parameter matrices, D is the dimension of the hidden layer, Represents the similarity between the audio feature vector and the video feature vector.

3. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: The weight value is dynamically adjusted according to the time distance between each video clip to strengthen or weaken the feature information at a specific time point, which is expressed as: R ij =exp(-|w(i-j) 2 +b|) Where R ij represents the influence weight of the j-th segment on the i-th segment, w and b represent the weight parameter and bias term respectively.

4. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: The audio features and video features are fused to generate enhanced video features, which can be expressed as: Among them, X v is the original visual feature, W f and W v is a trainable weight matrix.

5. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: In the feature similarity branch, in order to learn the similarity relationship between features, the fused features obtained from the cross-modal fusion module are first projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space. The adjacency matrix is ​​defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the node, a hyperbolic graph convolution operation is performed so that each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips.

6. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: In the temporal correlation branch, the feature vector is projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space and construct a temporal relationship matrix; wherein, the temporal relationship between the video clips is represented in the temporal relationship matrix, and the absolute time difference between the clips is calculated, that is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater; after obtaining the temporal relationship matrix, the hyperbolic graph convolution operation is applied to update the feature vector of each node and update the feature representation of the current node.

7. An abnormal behavior detection system based on cross-modal fusion, characterized in that: include: An acquisition module, used for acquiring video data to be detected; A prediction module is used to process the acquired video data using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features in the video data; the audio feature extraction network is used to extract audio features in the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features; the hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is operated through multiple layers of hyperbolic graph convolution, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch is to model the temporal relationship between video clips in the hyperbolic space, so as to better understand and identify abnormal behaviors in the video; The behavior prediction network is used to concatenate the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and to represent the similarity between the concatenated features and the weight matrix by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the abnormal behavior detection method based on cross-modal fusion as described in any one of claims 1-6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the abnormal behavior detection method based on cross-modal fusion as described in any one of claims 1-6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the abnormal behavior detection method based on cross-modal fusion as described in any one of claims 1-6.

Images