Abnormal behavior detection method and system based on cross-modal fusion

Abstract

The application provides an abnormal behavior detection method and system based on cross-modal fusion, and belongs to the technical field of abnormal behavior detection. The video data obtained is processed by using a pre-trained abnormal behavior detection model to obtain an abnormal behavior detection result. The abnormal behavior detection model comprises a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network. A new cross-modal fusion algorithm is introduced, the fusion of audio features and visual features is realized in the time dimension, the audio information is used to further distinguish the events in the visual dimension, two branches based on hyperbolic graph convolution are designed to process the similarity and time correlation of the features respectively, and the distinguishing degree of the features is improved in the hyperbolic space, so that the accuracy of the abnormal behavior detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal behavior detection, and in particular to an abnormal behavior detection method and system based on cross-modal fusion. Background Art

[0002] In recent years, with the widespread adoption of intelligent surveillance camera systems, demand for intelligent systems capable of automatically and in real time analyzing video surveillance has increased across various industries. Automatically detecting unusual events in complex and crowded scenarios (such as markets, shopping malls, hospitals, banks, streets, educational institutions, and city administrative offices) is a particularly challenging task in modern intelligent video surveillance systems. In most cases, timely and accurate detection of unusual behavior is the primary goal of anomaly detection. Anomalies in video typically include unusual behaviors and unusual entities. Abnormal behaviors include riots, fighting, stampedes, and shootings, while unusual entities include suspicious individuals and the presence of controlled knives in public places. Both unusual entities and activities should be detected and alerted by intelligent security systems in a timely and automatic manner. However, the task of detecting unusual behavior in video is challenging due to the ambiguity of anomaly definitions, the diversity of human behavior, and the complexity of environmental conditions.

[0003] Although research on abnormal behavior detection based on surveillance video has been ongoing for many years and many excellent solutions have emerged, designing and optimizing deep learning network models with high recognition accuracy, strong generalizability, and good performance remains a challenging task. Unlike conventional detection tasks, abnormal behavior detection involves dealing with a small number of unpredictable and rare events, which poses the following difficulties for deep learning model design: 1) Unknowns: Abnormal situations often involve many unknown factors, such as instances with unknown behavior, structure, and distribution, which remain unknown until they actually occur. Examples include abnormal situations in complex locations and new types of unusual attacks. Furthermore, abnormal behavior is often irregular, and different types of anomalies often exhibit distinct characteristics. For example, in surveillance video, abnormal events such as major traffic accidents, robberies, and burglaries exhibit significant visual differences. 2) Rarity: The probability of abnormal behavior occurring in daily life is far lower than that of normal behavior. Therefore, collecting large amounts of labeled anomaly data is quite difficult. Furthermore, the majority of videos in the datasets used to train deep learning models consist of normal clips, which results in far fewer abnormal events than normal events during model training, impairing the model's ability to fit abnormal events. 3) Diversity: Abnormal events in different scenarios vary widely, such as environmental conditions (lighting changes, object shadows, object occlusions, complex backgrounds, etc.), crowd density, noise interference, the complexity of human behavior, and camera settings. These factors all affect the extraction of visual features, making it more difficult for the model to locate and identify abnormal events. Therefore, the exploration of abnormal behavior detection remains a very challenging topic in both industry and academia, yet it also has extraordinary practical and universal value.

[0004] Existing abnormal behavior detection algorithms are mainly categorized as fully supervised, weakly supervised, semi-supervised, and unsupervised. Fully supervised learning offers high accuracy for abnormal behavior detection in videos because it can learn relatively accurate patterns from labeled normal and abnormal data, resulting in better detection results during the training phase. Furthermore, fully supervised learning methods rely on labeled training data, making them applicable to a variety of application scenarios as long as sufficient training data is available. However, due to the high cost of label acquisition for fully supervised learning, relevant datasets may be lacking. Furthermore, during training, these methods may suffer from overfitting, which reduces their generalization ability to unseen data. Weakly supervised abnormal behavior detection requires only coarse dataset annotation, which improves data utilization efficiency and helps the model learn more generalized feature representations rather than relying excessively on detailed annotated video data, thereby improving the model's generalization ability. Furthermore, weakly supervised learning models are generally more robust and better able to handle noise and incomplete annotations, making them more suitable for real-world applications. Compared to weakly supervised learning methods, semi-supervised anomaly detection methods require less data labeling because the training set contains only normal data. Anomaly detection is accomplished by simply identifying data that differs from the normal data learned in the training set as anomalies. However, during training, these methods only learn the characteristics of the anomaly data and struggle to discern the differences between the two types of data, potentially leading to a high number of false positives. Unsupervised anomaly detection methods are relatively rare, and their lack of supervised information results in poor ability to distinguish abnormal behavior, potentially reducing algorithm performance.

[0005] In summary, after investigation, we found that the current mainstream abnormal behavior detection algorithms have the following main defects:

[0006] (1) Many early anomaly detection systems relied on a single modality, focusing primarily on video information, such as analyzing the motion or appearance characteristics of people in the video. However, these methods ignored the importance of audio information. Audio in audio and video, such as screams, gunshots, and explosions, is also very important for the model to identify abnormal behavior.

[0007] (2) The extracted features lack robustness. Traditional anomaly detection methods rely on hand-crafted features, such as HOG and SIFT, which require specific expertise and extensive debugging. As a result, this method has poor robustness, and the model trained with such features may not be able to adapt to the diversity and complexity of video data.

[0008] (3) Poor contextual association capabilities. Abnormal events often involve rich contextual information, requiring comprehensive processing of related audio and video information. However, many existing methods do not effectively utilize this multimodal contextual association. Furthermore, in multimodal videos, the temporal synchronization of audio and visual information is a challenge. Existing methods do not effectively address this issue, making it difficult for models to accurately associate audio and visual events. Summary of the Invention

[0009] The purpose of the present invention is to provide a method and system for abnormal behavior detection based on cross-modal fusion to solve at least one technical problem existing in the above-mentioned background technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a method for detecting abnormal behavior based on cross-modal fusion, comprising:

[0012] Obtain the video data to be detected;

[0013] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein, the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features. ; The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space in order to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0014] Furthermore, the attention mechanism is used to combine audio and visual features to capture the global correlation between the two. The global attention map is constructed as follows:

[0015]

[0016] where A denotes the global attention map, Q a = X a W Q is the result of linear transformation of audio features, K v = X v W k is the result of linear transformation of video features, W Q and W k are both learnable parameter matrices, and D is the dimension of the hidden layer. represents the similarity between the audio feature vector and the video feature vector.

[0017] Further, the weight value is dynamically adjusted according to the time distance between each video segment to strengthen or weaken the feature information at a specific time point, which is represented as:

[0018]

[0019] where R ij represents the influence weight of the jth segment on the ith segment, and w and b represent the weight parameter and the bias term, respectively.

[0020] Further, the audio features and the video features are fused to generate enhanced video features, which is represented as:

[0021]

[0022]

[0023] where X v is the original visual feature, W f and W v are trainable weight matrices.

[0024] Further, in the feature similarity branch, in order to learn the similarity relationship between the features, the fusion features obtained from the cross-modal fusion module are first projected into hyperbolic space through exponential mapping to obtain the feature representation of the features in the hyperbolic space. The adjacency matrix is defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the nodes, the hyperbolic graph convolution operation is performed to integrate the information of the neighborhood nodes of each node, so as to capture the complex similarity relationship between the video segments.

[0025] Furthermore, in the temporal correlation branch, the feature vector is projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space and construct a temporal relationship matrix; wherein, the temporal relationship between the video clips is represented in the temporal relationship matrix, and the absolute time difference between the clips is calculated, that is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater; after obtaining the temporal relationship matrix, the hyperbolic graph convolution operation is applied to update the feature vector of each node and update the feature representation of the current node.

[0026] In a second aspect, the present invention provides an abnormal behavior detection system based on cross-modal fusion, comprising:

[0027] An acquisition module, used to acquire video data to be detected;

[0028] The prediction module is used to process the acquired video data using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein, the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate an enhanced Video features; the hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0029] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the abnormal behavior detection method based on cross-modal fusion as described in the first aspect is implemented.

[0030] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the abnormal behavior detection method based on cross-modal fusion as described in the first aspect.

[0031] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the abnormal behavior detection method based on cross-modal fusion as described in the first aspect.

[0032] Explanation of terms:

[0033] Cross-modal fusion: Cross-modal fusion technology refers to the fusion of audio and visual features of a video, achieving multimodal fusion of the two features in the temporal dimension, and using audio information to further distinguish events that are blurred in the visual dimension.

[0034] Hyperbolic Graph Convolutional Network (HGCN): A deep learning model for processing graph data, particularly data in non-Euclidean space. In hyperbolic space, the distance metric differs from that in Euclidean space, better capturing relationships between hierarchical structures. The core idea of ​​HGCN is to perform convolution operations in hyperbolic space, enabling the model to better capture the nonlinear structure of graph data.

[0035] Abnormal behavior detection refers to the use of technology to identify and monitor unusual behavior in a setting. This technology is commonly used in public places, prisons, schools, and other locations to promptly detect and respond to unusual incidents and ensure public safety. Abnormal behavior detection typically involves multiple steps, including data collection, feature extraction, classification and detection, as well as early warning and response. Furthermore, abnormal behavior detection technology can combine multiple sensors and data sources, such as surveillance cameras, microphones, and sensors, to improve detection accuracy and timeliness. Furthermore, real-time data analysis and pattern recognition techniques can be used to continuously optimize detection algorithms and enhance system performance and reliability. The application of abnormal behavior detection technology can help prevent unusual incidents and ensure public safety.

[0036] The beneficial effects of the present invention are as follows: a new cross-modal fusion algorithm is introduced to achieve the fusion of audio features and visual features in the time dimension, and audio information is used to further distinguish events that are blurred in the visual dimension; two branches based on hyperbolic graph convolution are designed to process the similarity and temporal correlation of features respectively, and the discrimination of features in the hyperbolic space is improved, thereby improving the accuracy of abnormal behavior detection.

[0037] Advantages of the additional aspects of the application will become apparent in the following description, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0039] Figure 1 A function block diagram of the abnormal behavior detection model based on cross-modal fusion and hyperbolic graph convolution of weakly supervised learning is shown in the embodiment of the present application.

[0040] Figure 2 A whole structure diagram of the abnormal behavior detection algorithm is shown in the embodiment of the present application.

[0041] Figure 3 A schematic diagram of the cross-modal fusion module is shown in the embodiment of the present application.

[0042] Figure 4 A structure diagram of the cross-modal fusion module is shown in the embodiment of the present application.

[0043] Figure 5 A schematic diagram of the feature similarity branch of the hyperbolic graph convolution module is shown in the embodiment of the present application.

[0044] Figure 6 A schematic diagram of the time correlation branch of the hyperbolic graph convolution module is shown in the embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application will be described in detail below, and the examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation of the present application.

[0046] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs.

[0047] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as such.

[0048] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0049] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise contradictory.

[0050] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0051] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0052] The application provides an abnormal behavior detection algorithm based on cross-modal fusion and hyperbolic graph convolution, which can be applied to smart cities, intelligent transportation, intelligent security systems, Internet live content review, film and television work review, etc. The method comprises the following steps: using an attention mechanism to extract global attention maps of audio features and video features, which are used to measure the correlation between the two features; then, local wake-up is realized by adaptive position prior, and the weight value is dynamically adjusted according to the time distance between video segments to strengthen or weaken the feature information at a specific time point; next, cross-modal fusion is realized, the purpose of which is to fuse audio features and video features to generate enhanced video features. The two branches of hyperbolic graph convolution are used to further process the obtained fusion features, and the feature similarity branch and the time sequence correlation branch are used to help the model effectively learn rich feature representations from multi-modal data. Finally, the multi-instance learning algorithm is used to obtain the abnormal score of the video segment, and the specific target task is completed. The application uses the global feature extraction capability of the attention mechanism to complete the cross-modal fusion of video features and audio features through a new modal fusion framework. Specifically, a cross-modal fusion module is designed to optimize the modal fusion effect, and the auxiliary role of audio information for abnormal event recognition is used to enhance the model's ability to distinguish normal events and abnormal events. In order to further improve the distinguishability of the features, two hyperbolic graph convolution branches are designed to capture the complex similarity relationship between video segments and the time sequence relationship between video segments. In combination with the above two modules, a weakly supervised learning method is used to realize the recognition of video abnormal segments by the model.

[0053] Embodiment 1

[0054] In this embodiment 1, first, an abnormal behavior detection system based on cross-modal fusion is provided, comprising: an acquisition module for acquiring video data to be detected. A prediction module is used to process the acquired video data by using a pre-trained abnormal behavior detection model to obtain an abnormal behavior detection result.

[0055] The abnormal behavior detection model comprises a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network. The visual feature extraction network is used to extract visual features in video data. The audio feature extraction network is used to extract audio features in the video data. The cross-modal fusion network is used to combine the audio features and the visual features, capture global correlation between the audio features and the visual features, dynamically adjust a weight value according to a time distance between video clips, fuse the audio features and the visual features, and generate enhanced video features. The hyperbolic graph convolution network comprises a feature similarity branch and a time correlation branch. The feature similarity branch integrates information of neighbor nodes of each node after a plurality of hyperbolic graph convolution operations, thereby capturing complex similarity between the video clips. The time correlation branch models a time sequence relationship between the video clips in a hyperbolic space, so as to better understand and identify abnormal behaviors in the video. The behavior prediction network is used to splice results of the feature similarity branch and the time correlation branch to form a joint feature representation. Similarity between the spliced features and a weight matrix is represented by calculating an inner product in the hyperbolic space, so as to output a score for judging the abnormal behavior.

[0056] In the embodiment, the system is used to realize the abnormal behavior detection based on the cross-modal fusion, which comprises: using the acquisition module to acquire video data to be detected. Then, the prediction module is used to process the acquired video data by using the pre-trained abnormal behavior detection model, so as to obtain an abnormal behavior detection result.

[0057] The cross-modal fusion network combines the audio features and the visual features by using an attention mechanism, so as to capture global correlation between the audio features and the visual features. The global attention graph is constructed in the following manner:

[0058]

[0059] wherein A represents the global attention graph, Q a = X a W Q is a result of the audio features after linear transformation, K v = X v W k is a result of the video features after linear transformation, W Q and W k are both learnable parameter matrices, D is a dimension of a hidden layer, represents similarity between the audio feature vector and the video feature vector.

[0060] The weight value is dynamically adjusted according to the time distance between the video clips, so as to strengthen or weaken feature information at a specific time point, which is represented as:

[0061] R ij =exp(-|w(ij) 2 +b|)

[0062] where R ij represents the influence weight of the jth segment on the ith segment, w and b represent the weight parameter and bias term respectively.

[0063] The audio features and video features are fused to generate enhanced video features, which can be expressed as:

[0064]

[0065]

[0066] Among them, X v is the original visual feature, W f and W v is a trainable weight matrix.

[0067] In the feature similarity branch, in order to learn the similarity relationship between features, the fused features obtained from the cross-modal fusion module are first projected into the hyperbolic space through exponential mapping to obtain the feature representation of the features in the hyperbolic space. The adjacency matrix is ​​defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the node, a hyperbolic graph convolution operation is performed so that each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips.

[0068] In the temporal correlation branch, the feature vector is projected into the hyperbolic space through an exponential mapping to obtain the feature representation of the feature in the hyperbolic space and construct a temporal relationship matrix. The temporal relationship matrix represents the temporal relationship between video clips and calculates the absolute time difference between the clips. That is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater. After obtaining the temporal relationship matrix, a hyperbolic graph convolution operation is applied to update the feature vector of each node and the feature representation of the current node.

[0069] In summary, this embodiment proposes an abnormal behavior detection algorithm based on cross-modal fusion and hyperbolic graph convolution. This is an audio-guided multimodal anomaly detection framework. By fusing audio features and visual features in the time dimension, after obtaining the fused features, it is processed by hyperbolic graph convolution to improve the model's discrimination and thus improve the accuracy of abnormal behavior detection.

[0070] Example 2

[0071] Most of the existing work focuses on the analysis of visual information, while ignoring the auxiliary role of audio information. Compared with single-modal input, multimodal information input is more meaningful for improving the performance of abnormal behavior detection algorithms. Generally speaking, abnormal behavior can be accurately located using visual information, but accurate judgment cannot be made by relying solely on visual information. The effective use of audio information can further distinguish events that are blurred in the visual dimension, thereby improving algorithm performance. Therefore, in this embodiment, an audio-guided multimodal anomaly detection method is proposed, which realizes the multimodal fusion of audio features and visual features in the time dimension, and then inputs the fused features into the hyperbolic graph convolution module, using two branches to process feature similarity and time correlation respectively, and improves the feature discrimination in the hyperbolic space, thereby improving the accuracy of abnormal behavior detection.

[0072] Figure 1 As shown in the figure, this embodiment proposes an abnormal behavior detection framework based on cross-modal fusion of weakly supervised learning. In this detection framework, the visual and audio features of the video are first extracted using the I3D (Inflated 3D) network and the VGGish network, respectively. These features are further optimized using the cross-modal fusion module and the hyperbolic graph convolutional network module. Subsequently, a multi-instance learning algorithm is used to obtain an abnormal behavior score. This framework can fuse audio and video features, improve the separability of features through the hyperbolic graph convolutional network, avoid background noise, and thus enhance the model's ability to identify abnormal behavior.

[0073] Specifically, the cross-modal fusion module consists of the following operations: 1) generating a global attention map; 2) local attention; and 3) cross-modal fusion. Generating a global attention map is a key step in this module, aiming to combine audio and visual features to capture the global correlation between them. The local attention operation is implemented through adaptive position priors, dynamically adjusting weights based on the temporal distance between video clips to strengthen or weaken feature information at specific time points. The cross-modal fusion operation fuses audio and video features to generate enhanced video features.

[0074] The hyperbolic graph convolution module consists of two main branches: 1) feature similarity branch; 2) temporal correlation branch. The feature similarity branch, through multi-layer hyperbolic graph convolution operations, allows each node to integrate information from its neighboring nodes, helping the model capture complex similarity relationships between video clips. The temporal correlation branch aims to model the temporal relationships between video clips in hyperbolic space to better understand and identify abnormal behavior in videos.

[0075] In this embodiment, Figure 2 As shown in FIG, the abnormal behavior detection method based on cross-modal fusion of weakly supervised learning specifically includes the following steps:

[0076] Step S1: Construct a cross-modal fusion module, which mainly uses the attention mechanism to fuse multimodal features. The specific structure is as follows Figures 3 to 5 The main steps are as follows:

[0077] Step S1-1: Generate global attention map

[0078] After obtaining the video and audio features output by the 13D model and the VGGish model, the cross-modal fusion module fuses the multimodal features. The attention mechanism is used to combine the audio and visual features to capture the global correlation between the two. The construction of the global attention map can be expressed as:

[0079]

[0080] Where A represents the global attention map, Q a =X a W Q is the result of linear change of audio features, K v =X v W k It is the result of linear change of video features, W Q and W k are all learnable parameter matrices whose purpose is to project the corresponding features into a suitable space for calculating attention scores. D is the dimension of the hidden layer, which is used to stabilize the scores of the attention mechanism and prevent problems such as gradient explosion or disappearance during training. Represents the similarity between the audio feature vector and the video feature vector. The result is an attention score that measures the degree of association between audio and video features.

[0081] Step S1-2: Partial wake-up operation

[0082] This process is achieved through adaptive position priors, which can dynamically adjust the weight value according to the temporal distance between each video segment to strengthen or weaken the feature information at a specific time point. The operation can be expressed as:

[0083] R ij =exp(-|w(ij) 2 +b|)

[0084] where R ij represents the influence weight of the j-th segment on the i-th segment, w and b represent the weight parameter and bias term respectively. The former controls the influence degree of the position prior. The larger the parameter, the stronger the influence of the position prior and the more obvious the effect of local wake-up. The latter is used to adjust the influence position of the position prior and further fine-tune the effect of the local wake-up operation.

[0085] Step S1-3: Cross-modal fusion operation

[0086] The cross-modal fusion operation is the last step of this module. Its purpose is to fuse audio features and video features to generate enhanced video features. The specific details can be expressed as follows:

[0087]

[0088]

[0089] where X v is the original visual feature, W f and W v It is a trainable weight matrix that maps audio features and video features into a specific feature space, and then adds an adaptive position prior to the global attention map. In addition, in order to ensure the original distribution of visual features, it is necessary to v Finally, in order to stabilize the model training process and prevent gradient explosion, the features need to be normalized.

[0090] Step S2: Hyperbolic Graph Convolution Module

[0091] After step S1, enhanced visual features are obtained. These features are supplemented by audio information in the temporal dimension, helping the model better capture multimodal information related to abnormal behavior. Next, the two branches of hyperbolic graph convolution are used to learn feature similarity and temporal correlation, respectively, helping the model effectively learn rich feature representations from multimodal data.

[0092] Step S2-1: Feature similarity branch

[0093] In order to learn the similarity relationship between features, we first introduce the feature similarity branch, such as Figure 6 As shown, first the fusion features obtained from the cross-modal fusion module are Projecting the feature into the hyperbolic space through exponential mapping, we can get the feature representation in the hyperbolic space. The next step is to define the adjacency matrix by calculating the feature similarity in hyperbolic space. The calculation method is as follows:

[0094]

[0095] Feature similarity is obtained by calculating the Lorentz distance of features in hyperbolic space. The calculation method is as follows:

[0096]

[0097] Before the introduction of the activation function, a thresholding operation was used to eliminate the weak relationship between features in the hyperbolic space and strengthen the correlation between more similar features:

[0098]

[0099] Here, τ is a pre-set threshold used to distinguish whether there is strong similarity between nodes. It can remove connections with poor correlation when composing the graph, which helps the graph convolutional network to learn the feature similarity between nodes more effectively.

[0100] After obtaining the node adjacency matrix, the next step is to perform a hyperbolic graph convolution operation:

[0101]

[0102]

[0103] It means that a hyperbolic transformation is performed on the i-th node of the l-1th layer. After multiple layers of transformation operations, the i-th node of the l-th layer integrates the information of all adjacent nodes.

[0104] After the feature similarity branch passes through multiple layers of hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby helping the model capture the complex similarity relationships between video clips. Finally, in order to increase the nonlinear relationship, the output value needs to be operated as follows:

[0105]

[0106] Step S2-2: Time-dependent branching

[0107] The goal of the temporal correlation branch is to model the temporal relationships between video clips in hyperbolic space in order to better understand and identify unusual and abnormal behaviors in videos.

[0108] Similar to the similarity branch, the temporal correlation branch also projects the feature vector into the hyperbolic space through exponential mapping to obtain the feature representation in the hyperbolic space. The next step is to construct the temporal relationship matrix:

[0109]

[0110] The temporal relationship matrix represents the temporal relationship between video clips and calculates the absolute time difference between clip i and clip j. That is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater.

[0111] After obtaining the temporal relationship matrix, the next step is to apply the hyperbolic graph convolution operation to update the feature vector of each node. This operation takes into account the adjacent nodes that have a temporal relationship with the current node and updates the feature representation of the current node:

[0112]

[0113] Through the temporal correlation branch, the model can effectively capture the temporal relationship between video clips in hyperbolic space, making it easier to understand dynamic changes in the video.

[0114] Step S3: Behavior prediction

[0115] After obtaining the output results of the feature similarity branch and the temporal correlation branch, the next step is behavior prediction:

[0116]

[0117] First, the results of the two branches are concatenated to form a joint feature representation. Next, the similarity between the concatenated features and the weight matrix is ​​expressed by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0118] In summary, in this embodiment, by introducing a cross-modal fusion module and a hyperbolic graph convolution module, the model's ability to identify abnormal behavior is improved, and it can be put into practical use in the future. Considering the use of audio and visual modal data for abnormal behavior identification, compared with single-modal input, multi-modal information input is more meaningful for improving the performance of abnormal behavior detection algorithms. An innovative cross-modal fusion module is proposed to fuse the two modal information, realizing multi-modal fusion of audio features and visual features in the time dimension, and taking advantage of the hyperbolic space, using two branches to process feature similarity and temporal correlation.

[0119] Example 3

[0120] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the above-mentioned abnormal behavior detection method based on cross-modal fusion is implemented. The method includes:

[0121] Obtain the video data to be detected;

[0122] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein, the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features. ; The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space in order to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0123] Example 4

[0124] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the above-mentioned abnormal behavior detection method based on cross-modal fusion, which includes:

[0125] Obtain the video data to be detected;

[0126] The acquired video data is processed by using a pre-trained abnormal behavior detection model to obtain an abnormal behavior detection result; wherein the abnormal behavior detection model comprises a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features in the video data; the audio feature extraction network is used to extract audio features in the video data; the cross-modal fusion network is used to combine the audio features and the visual features, capture the global correlation between them, dynamically adjust the weight value according to the time distance between each video segment, fuse the audio features and the video features, and generate enhanced video features; the hyperbolic graph convolution network comprises a feature similarity branch and a time correlation branch; the feature similarity branch integrates the information of the neighborhood nodes of each node after a plurality of hyperbolic graph convolution operations, thereby capturing the complex similarity relationship between the video segments; the time correlation branch models the time sequence relationship between the video segments in the hyperbolic space, so as to better understand and identify the abnormal behavior in the video; the behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, calculate the similarity between the spliced features and the weight matrix in the hyperbolic space, and output a score for judging the abnormal behavior.

[0127] Embodiment 5

[0128] The embodiment 5 provides an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for realizing an abnormal behavior detection method based on cross-modal fusion as described above, the method comprising:

[0129] acquiring video data to be detected;

[0130] The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein, the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features. ; The hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space in order to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to splice the results of the feature similarity branch and the time correlation branch to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​represented by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior.

[0131] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A method for abnormal behavior detection based on cross-modal fusion, characterized in that: include: Obtain the video data to be detected; The acquired video data is processed using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein, the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features; the hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch models the temporal relationship between video clips in the hyperbolic space to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to combine the results of the feature similarity branch and the time correlation branch. The two modules are connected to form a joint feature representation, and the similarity between the spliced ​​features and the weight matrix is ​​expressed by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior. In the feature similarity branch, in order to learn the similarity relationship between features, the fusion features obtained from the cross-modal fusion module are first projected into the hyperbolic space through exponential mapping to obtain the feature representation of the features in the hyperbolic space. The adjacency matrix is ​​defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the node, a hyperbolic graph convolution operation is performed to enable each node to integrate the information of its neighboring nodes. This captures the complex similarity relationship between video clips; in the temporal correlation branch, the feature vector is projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space and construct a temporal relationship matrix; wherein, the temporal relationship between the video clips is represented in the temporal relationship matrix, and the absolute time difference between the clips is calculated, that is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater; after obtaining the temporal relationship matrix, the hyperbolic graph convolution operation is applied to update the feature vector of each node and update the feature representation of the current node.

2. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: The attention mechanism is used to combine audio and visual features to capture the global correlation between the two. The global attention map is constructed as follows: Where A represents the global attention map, Q a =X a W Q is the result of linear change of audio features, K v =X v W k It is the result of linear change of video features, W Q and W k are all learnable parameter matrices, D is the dimension of the hidden layer, Represents the similarity between the audio feature vector and the video feature vector.

3. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: The weight value is dynamically adjusted according to the temporal distance between each video clip to strengthen or weaken the feature information at a specific time point, which is expressed as: R ij =exp(-|w(i-j) 2 +b|) where R ij represents the influence weight of the jth segment on the ith segment, w and b represent the weight parameter and bias term respectively.

4. The abnormal behavior detection method based on cross-modal fusion according to claim 1 is characterized in that: The audio features and video features are fused to generate enhanced video features, which can be expressed as: Among them, X v is the original visual feature, W f and W v is a trainable weight matrix.

5. An abnormal behavior detection system based on cross-modal fusion, characterized in that: include: An acquisition module, used to acquire video data to be detected; A prediction module is used to process the acquired video data using a pre-trained abnormal behavior detection model to obtain abnormal behavior detection results; wherein, the abnormal behavior detection model includes a visual feature extraction network, an audio feature extraction network, a cross-modal fusion network, a hyperbolic graph convolution network and a behavior prediction network; the visual feature extraction network is used to extract visual features from the video data; the audio feature extraction network is used to extract audio features from the video data; the cross-modal fusion network is used to combine audio features and visual features to capture the global correlation between the two, dynamically adjust the weight value according to the time distance between each video clip, fuse the audio features with the video features, and generate enhanced video features; the hyperbolic graph convolution network includes a feature similarity branch and a time correlation branch. After the feature similarity branch is subjected to multi-layer hyperbolic graph convolution operations, each node integrates the information of its neighboring nodes, thereby capturing the complex similarity relationship between video clips. The time correlation branch is to model the temporal relationship between video clips in the hyperbolic space in order to better understand and identify abnormal behaviors in the video; the behavior prediction network is used to combine the feature similarity branch and the time correlation branch. The results are concatenated to form a joint feature representation, and the similarity between the concatenated features and the weight matrix is ​​expressed by calculating the inner product in the hyperbolic space, thereby outputting a score for judging abnormal behavior; in the feature similarity branch, in order to learn the similarity relationship between features, the fusion features obtained from the cross-modal fusion module are first projected into the hyperbolic space through exponential mapping to obtain the feature representation of the features in the hyperbolic space, and the adjacency matrix is ​​defined by calculating the feature similarity in the hyperbolic space. After obtaining the adjacency matrix of the node, a hyperbolic graph convolution operation is performed to enable each node to integrate the information of its neighboring nodes. information, thereby capturing the complex similarity relationship between video clips; in the temporal correlation branch, the feature vector is projected into the hyperbolic space through exponential mapping to obtain the feature representation of the feature in the hyperbolic space and construct a temporal relationship matrix; wherein, the temporal relationship between the video clips is represented in the temporal relationship matrix, and the absolute time difference between the clips is calculated. That is, the smaller the time difference between the two videos, the larger the weight value in the temporal relationship matrix, indicating that the temporal correlation between the two clips is greater; after obtaining the temporal relationship matrix, the hyperbolic graph convolution operation is applied to update the feature vector of each node and the feature representation of the current node.

6. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the abnormal behavior detection method based on cross-modal fusion as described in any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the abnormal behavior detection method based on cross-modal fusion as described in any one of claims 1 to 4.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the abnormal behavior detection method based on cross-modal fusion as described in any one of claims 1-4.

Images