Video anomaly detection method and device based on multi-modal feature alignment

Abstract

The invention relates to a video anomaly detection technology, and discloses a multi-modal feature alignment-based video anomaly detection method, which comprises the following steps of: extracting a visual text category label of a target video, generating a visual complete statement prompt, and inputting the visual complete statement prompt into a text encoder to obtain a visual text feature; extracting visual features of the target video by using a visual time network, and obtaining a visual-text alignment graph and cross-modal enhanced visual features according to the visual text features and the visual features; extracting an audio file in the target video, and obtaining an audio-text alignment graph and cross-modal enhanced audio features; splicing the cross-modal enhanced visual features and the cross-modal enhanced audio features, and inputting the spliced features into an anomaly classifier to obtain an anomaly classification result; and obtaining an event category classification result by using the vision-text alignment graph and the audio-text alignment graph. The invention further provides a video anomaly detection device and equipment based on multi-modal feature alignment and a medium. According to the invention, the accuracy of video anomaly detection can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method and device based on multimodal feature alignment. Background Art

[0002] In existing weakly supervised video anomaly detection, models based on visual information have achieved significant anomaly detection results. However, the classification labels in weakly supervised video anomaly detection datasets are often not well utilized, and this cross-modal visual-linguistic association has been neglected in previous studies.

[0003] In recent years, vision-language models have made tremendous progress in their development, with the CLIP model being particularly prominent in modeling generalized visual representations and semantic concepts. The core idea of the CLIP model is to achieve cross-modal alignment of images and texts through contrastive learning. This involves reducing the distance between image features and their matching text features in feature space, while increasing the distance between image features and their mismatched text features. Thanks to the massive amount of training data, the large-scale pre-trained CLIP model, based on hundreds of millions of image-text pairs, demonstrates powerful vision-language alignment capabilities. In existing technologies, the cross-modal alignment capabilities of the CLIP model have also been applied to weakly supervised video anomaly detection in order to better utilize the classification labels in weakly supervised video anomaly detection datasets. However, existing technologies are still not perfect in cross-modal alignment and fusion, resulting in poor multi-classification performance based on cross-modal information. Summary of the Invention

[0004] The present invention provides a video anomaly detection method and device based on multimodal feature alignment, which can improve the accuracy of video anomaly detection.

[0005] To achieve the above objectives, the present invention provides a video anomaly detection method based on multimodal feature alignment, comprising:

[0006] Get the target video and extract the visual text category label of the target video;

[0007] Generate visual complete sentence prompts based on visual text category labels, and input the visual complete sentence prompts into a preset text encoder to obtain visual text features;

[0008] The visual features of the target video are extracted using a visual temporal network, and the visual-text features and visual features are input into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map. The visual-text alignment map and visual-text features are then used to construct cross-modal enhanced visual features.

[0009] Extract the audio file from the target video and use the preset audio text extractor to extract the audio text features of the audio file;

[0010] The audio time network is used to extract audio sequence features from audio files, and the audio text features and audio sequence features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph. The audio-text alignment graph and audio text features are then used to construct cross-modal enhanced audio features.

[0011] Performing a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain a cross-modal splicing feature, and inputting the cross-modal splicing feature into a pre-built anomaly classifier to obtain an anomaly classification result;

[0012] The visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map. The event category probability map of the multimodal fusion alignment map in the feature dimension is calculated, and the event category is determined based on the event category probability map to obtain the event category classification result.

[0013] Optionally, generating a visual complete sentence prompt according to the visual text category label includes:

[0014] Convert visual text category labels into category tags using the word segmenter of the pre-built multimodal pre-training model;

[0015] Inserting category tags into a learnable context tag sequence yields visually complete sentence hints.

[0016] Optionally, extracting visual features of the target video using a visual temporal network includes:

[0017] The target video is processed into non-overlapping windows of preset length to obtain multi-window video segments, and the visual features of each window video segment are extracted using the image encoder in the pre-built multimodal pre-trained model. The visual features of each window video segment are spliced in the order of the windows to obtain a visual feature sequence;

[0018] Position encoding is performed on multi-window video clips, and the position encoding is added to the visual feature sequence to obtain the position-embedded visual features;

[0019] The position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features;

[0020] Input the local visual features into the pre-built global graph convolutional network, and use the feature similarity network in the pre-built global graph convolutional network to calculate the feature similarity feature matrix, and use the relative distance network in the pre-built global graph convolutional network to calculate the relative distance feature matrix;

[0021] The feature similarity feature matrix and the relative distance feature matrix are concatenated in the feature dimension to obtain the visual features of the target video.

[0022] Optionally, the position-embedded visual features and a pre-built attention mask matrix are input into a multi-head attention module to obtain local visual features, including:

[0023] Divide the position-embedded visual features into multiple positions of the same number of attention heads as the multi-head attention module;

[0024] The embedded visual features of each position are input into each attention head for calculation to obtain the query matrix, key matrix and value matrix of each attention head;

[0025] The query matrix is dot-producted with the key matrix and scaled before being added to the pre-built attention mask matrix to obtain the attention map.

[0026] After normalizing the attention map, perform dot product calculation with the value matrix to obtain the output visual features of each attention head;

[0027] The output visual features of each attention head are concatenated to obtain local visual features.

[0028] Optionally, the step of concatenating the output visual features of each attention head to obtain the local visual features includes:

[0029] Local visual feature F′ v It is expressed using the following formula:

[0030] F′ v =FC2(GELU(FC1(LN(F v-attn ))))+F v-embed

[0031] Among them, FC1 is the input dimension f v The output dimension is 4f v The fully connected layer, FC2 is the input dimension of 4f v The output dimension is f v The fully connected layer, GELU is the nonlinear activation function GELU, F v-embed is the visual feature after position embedding, F v-attn is the output of the multi-head attention module.

[0032] Optionally, inputting the visual-text features and the visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment graph includes:

[0033] The nonlinear activation function in the multi-layer perceptron of the pre-built visual-text cross-modal attention module is used to activate the visual text features to obtain nonlinear text features;

[0034] Use convolutional neural network to perform convolution processing on nonlinear text features to obtain text feature key matrix and text feature value matrix;

[0035] Use convolutional neural network to convolve visual features to obtain visual feature query matrix;

[0036] The text feature key matrix is transposed to obtain a transposed text feature key matrix, the similarity matrix between the transposed text feature key matrix and the visual feature query matrix is calculated, and the matrix is scaled according to the feature dimension of the similarity matrix to obtain a visual-text alignment graph.

[0037] Optionally, constructing cross-modal enhanced visual features using the visual-text alignment graph and visual-text features includes:

[0038] After performing dot product operations on the visual-text alignment map and the text feature value matrix, a convolutional neural network is used to perform convolution operations and batch normalization processing to obtain preliminary cross-modal enhanced visual features;

[0039] The visual features of the target video are jump-connected with the preliminary cross-modal enhanced visual features to obtain the cross-modal enhanced visual features.

[0040] In order to solve the above problems, the present invention further provides a video anomaly detection device based on multimodal feature alignment, the device comprising:

[0041] The visual feature acquisition module is used to acquire the target video and extract the visual text category label of the target video; generate a visual complete sentence prompt based on the visual text category label, and input the visual complete sentence prompt into a preset text encoder to obtain visual text features; use the visual time network to extract the visual features of the target video, and input the visual text features and visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map, and use the visual-text alignment map and visual text features to construct cross-modal enhanced visual features;

[0042] The audio feature acquisition module is used to extract the audio file in the target video and use the preset audio-text extractor to extract the audio-text features of the audio file; the audio sequence features in the audio file are extracted using the audio time network, and the audio-text features and audio sequence features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph, and the audio-text alignment graph and audio-text features are used to construct cross-modal enhanced audio features;

[0043] The video anomaly detection module is used to perform a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain cross-modal splicing features, and input the cross-modal splicing features into a pre-built anomaly classifier to obtain anomaly classification results; fuse the visual-text alignment map and the audio-text alignment map to obtain a multimodal fusion alignment map, calculate the event category probability map of the multimodal fusion alignment map in the feature dimension, and judge the event category based on the event category probability map to obtain the event category classification result.

[0044] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the above-mentioned video anomaly detection method based on multimodal feature alignment.

[0048] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one computer program. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned video anomaly detection method based on multimodal feature alignment.

[0049] The present invention generates visual complete sentence prompts based on visual text category labels, which can convert discrete visual text category labels into complete sentence descriptions, and make full use of the semantic information of the category labels at the visual level in the target video. In addition, the visual features of the target video are extracted by using the visual time network, which can realize the correlation of the visual information in the target video in the time domain dimension. In addition, the visual text features and visual features are input into the pre-built visual-text cross-modal attention module to obtain the visual-text alignment graph, and the visual-text alignment graph and visual text features are used to construct cross-modal enhanced visual features, which can improve the correlation between visual features and text features and realize visual-text cross-modal alignment. In addition, the audio text features and audio sequence The column features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment map, and the audio-text alignment map and audio-text features are used to construct cross-modal enhanced audio features, which can expand the perception dimension of the target video. The cross-modal enhanced visual features and the cross-modal enhanced audio features are spliced on the feature dimension to obtain cross-modal splicing features. Combining the splicing features of multiple modalities and performing anomaly classification can improve the accuracy of anomaly classification. In addition, the visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map, and the event category probability map of the multimodal fusion alignment map on the feature dimension is calculated. The event category is judged based on the event category probability map, which can improve the accuracy of event category classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a flow chart of a video anomaly detection method based on multimodal feature alignment provided by one embodiment of the present invention;

[0051] Figure 2 A schematic diagram of an anomaly detection model for a video anomaly detection method based on multimodal feature alignment provided by one embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a local Transformer for a video anomaly detection method based on multimodal feature alignment provided by one embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a global GCN network for a video anomaly detection method based on multimodal feature alignment provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of visual-text cross-modal attention for a video anomaly detection method based on multimodal feature alignment provided by one embodiment of the present invention;

[0055] Figure 6 A functional module diagram of a video anomaly detection device based on multimodal feature alignment provided by one embodiment of the present invention;

[0056] Figure 7 A schematic structural diagram of an electronic device for implementing the video anomaly detection method based on multimodal feature alignment provided by one embodiment of the present invention.

[0057] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] The embodiment of the present application provides a video anomaly detection method based on multimodal feature alignment. The execution subject of the video anomaly detection method based on multimodal feature alignment includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the video anomaly detection method based on multimodal feature alignment can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0060] Reference Figure 1 FIG. 1 is a flow chart of a method for detecting anomalies in a video based on multimodal feature alignment according to an embodiment of the present invention. In this embodiment, the method for detecting anomalies in a video based on multimodal feature alignment includes:

[0061] S1. Obtain the target video and extract the visual text category label of the target video.

[0062] In the embodiment of the present invention, the target video refers to a video file having multiple abnormal event categories, wherein the event category can be a normal event category or an abnormal event category.

[0063] In the embodiment of the present invention, the visual text category label refers to an overall summary of the video content in the target video.

[0064] S2. Generate a visual complete sentence prompt based on the visual text category label, and input the visual complete sentence prompt into a preset text encoder to obtain visual text features.

[0065] As an embodiment of the present invention, generating a visual complete sentence prompt based on a visual text category label includes:

[0066] Convert visual text category labels into category tags using the word segmenter of the pre-built multimodal pre-trained model;

[0067] Inserting category tags into a learnable context tag sequence yields visually complete sentence hints.

[0068] For example, generating a visual complete sentence prompt based on the visual text category label and inputting the visual complete sentence prompt into a preset text encoder to obtain visual text features can be implemented using the following steps:

[0069] Step 1: Use CLIP's tokenizer to convert text category labels into category tokens. The formula is as follows:

[0070] t class =Tokenizer(Label)

[0071] Among them, Label represents the text category label, t class For category labels, for example, if the target video is an XD-Violence video, the text category labels are normal, fighting, shooting, riot, abuse, car accident, and explosion;

[0072] Step 2: Mark the obtained category t class Insert a learnable context token sequence of length len {c1,...,c len}, get the visual complete sentence prompt t p , and its formula is as follows:

[0073] t p ={c1,...,t class ,...,c len}

[0074] The visual complete sentence prompts corresponding to n categories are t p Input CLIP's text encoder to get visual text features Among them, f v For dimension.

[0075] In an embodiment of the present invention, the preset text encoder may adopt the text encoder in the CLIP (Contrastive Language-Image Pre-training) multimodal pre-training model.

[0076] S3. Use the visual temporal network to extract the visual features of the target video, and input the visual-text features and visual features into the pre-built visual-text cross-modal attention module to obtain the visual-text alignment map, and use the visual-text alignment map and visual-text features to construct cross-modal enhanced visual features.

[0077] In an embodiment of the present invention, a visual temporal network refers to a network model that processes the visual information in a target video in the time domain. The visual temporal network in an embodiment of the present invention is composed of a local Transformer encoder for extracting short-term temporal associations and a global GCN (Graph Convolutional Network) for encoding long-term temporal associations.

[0078] In one embodiment of the present invention, extracting visual features of a target video using a visual temporal network includes:

[0079] The target video is processed into non-overlapping windows of preset length to obtain multi-window video segments, and the visual features of each window video segment are extracted using the image encoder in the pre-built multimodal pre-trained model. The visual features of each window video segment are spliced in the order of the windows to obtain a visual feature sequence;

[0080] Position encoding is performed on multi-window video clips, and the position encoding is added to the visual feature sequence to obtain the position-embedded visual features;

[0081] The position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features;

[0082] Input the local visual features into the pre-built global graph convolutional network, and use the feature similarity network in the pre-built global graph convolutional network to calculate the feature similarity feature matrix, and use the relative distance network in the pre-built global graph convolutional network to calculate the relative distance feature matrix;

[0083] The feature similarity feature matrix and the relative distance feature matrix are concatenated in the feature dimension to obtain the visual features of the target video.

[0084] In an embodiment of the present invention, the pre-built multimodal pre-training model may adopt a CLIP (Contrastive Language-Image Pre-training) model.

[0085] Furthermore, the position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features, including:

[0086] Divide the position-embedded visual features into multiple positions of the same number of attention heads as the multi-head attention module;

[0087] The embedded visual features of each position are input into each attention head for calculation to obtain the query matrix, key matrix and value matrix of each attention head;

[0088] The query matrix is dot-producted with the key matrix and scaled before being added to the pre-built attention mask matrix to obtain the attention map.

[0089] After normalizing the attention map, perform dot product calculation with the value matrix to obtain the output visual features of each attention head;

[0090] The output visual features of each attention head are concatenated to obtain local visual features.

[0091] Furthermore, the output visual features of each attention head are concatenated to obtain local visual features, including:

[0092] Local visual feature F′ v It is expressed using the following formula:

[0093] F′ v =FC2(GELU(FC1(LN(F v-attn ))))+F v-embed

[0094] Among them, FC1 is the input dimension f v The output dimension is 4f v The fully connected layer, FC2 is the input dimension of 4f v The output dimension is f v The fully connected layer, GELU is the nonlinear activation function GELU, F v-embed is the visual feature after position embedding, F v-attn is the output of the multi-head attention module.

[0095] For example, the following steps may be used to extract visual features of a target video using a visual temporal network:

[0096] Step a) Local Transformer:

[0097] A non-overlapping sliding window of length win is used on the video feature sequence to limit the Transformer's self-attention calculation to the inside of the sliding window. For the input visual feature sequence F with a time sequence length win, v , first generate the corresponding position code P = {p1,...,p win}, where p i=i, and after embedding it, the embedded position code is added to the visual feature. The formula is as follows:

[0098] F v-embed =F v +FPE(P)

[0099] Among them F v-embed It represents the visual features after position embedding, and FPE stands for Frame Position Embeddings.

[0100] The visual features after position embedding are normalized by the layer and the length of the attention window is win attn Attention mask Mask attn Input the multi-head attention module together. For n heads heads The multi-head attention first divides the input features into n heads Servings, that is Enter n respectively heads The calculation is performed by the attention head, and the formula is as follows:

[0101] Q i =FC Q,i (LN(F v-embed,i ))

[0102] K i =FC K,i (LN(F v-embed,i ))

[0103] V i =FC V,i (LN(F v-embed,i ))

[0104] Among them F v-embed,i represents the input of the i-th head, Q i is the query matrix obtained by the i-th multi-attention head, K i is the key matrix obtained by the i-th multi-attention head, V i is the value matrix obtained by the i-th multi-attention head, FC Q,i 、FC K,i 、FC V,i represents a fully connected layer with equal input and output dimensions for the i-th attention head, and LN represents a layer normalization layer.

[0105] Based on the obtained Q i With K iCalculate the dot product and scale it, then add it to the attention mask. The purpose of the attention mask is to make the model ignore the attention calculation of some areas. Here, the attention mask only focuses on the attention calculation within the non-overlapping attention window, only focuses on the attention value of the light gray part, and ignores the attention value of the dark gray part. The specific method is to create a matrix equal to the target size, and make the mask value of the part that needs to be paid attention to 0, and the mask value of the unnecessary part is set to -∞. Based on the attention mask, the attention map M is obtained attn,i , and its formula is as follows:

[0106]

[0107] The obtained attention map is normalized by the Softmax function and then compared with V i Perform dot product calculation to get the output F of the i-th attention head v-attn,i , and n heads The outputs of the heads are spliced together to obtain the complete attention module output F v-attn , and its formula is as follows:

[0108] F v-attn,i =Softmax(M attn,i )·V i

[0109]

[0110] Where Concat represents the concatenation operation, and the output F of the multi-head attention module v-attn After layer normalization, two-layer MLP and jump connection, local visual features can be obtained, and the formula is as follows:

[0111] F′ v =FC2(GELU(FC1(LN(F v-attn ))))+F v-embed

[0112] Among them, FC1 is the input dimension f v The output dimension is 4f v The fully connected layer, FC2 is the input dimension of 4f v The output dimension is f v The fully connected layer, GELU is the nonlinear activation function GELU, F v-embed is the visual feature after position embedding, F v-attn is the output of the multi-head attention module.

[0113] Step b) Global GCN:

[0114] In order to further capture the global temporal association, the global GCN network is used after the local Transformer obtains the local visual features. The second-layer feature similarity GCN and the second-layer relative distance GCN are used to model the temporal association.

[0115] First, construct the feature similarity adjacency matrix A based on cosine feature similarity sim , where the similarity sim between the i-th segment feature and the j-th segment feature i,j is defined as follows:

[0116]

[0117] Here, ||·|| represents the L2 norm calculation. The obtained similarity is thresholded to ignore weakly associated pairs and focus on feature pairs with higher similarity. The thresholding is as follows:

[0118]

[0119] Where τ represents the threshold. After the above processing, the similarity matrix is normalized by Softmax to obtain the feature similarity adjacency matrix, which is defined as follows:

[0120] A sim =Softmax(sim)

[0121] Based on the feature similarity adjacency matrix, the update formula of feature similarity GCN can be obtained:

[0122]

[0123] in and Represent the node feature matrices of the lth layer and the l+1th layer respectively, is the learnable weight matrix of the lth layer, and GELU represents the nonlinear activation function GELU.

[0124] Construct relative distance adjacency matrix A based on relative distance dis , where the relative distance between the i-th segment feature and the j-th segment feature is dis i,j is defined as follows:

[0125]

[0126] Where e represents a natural constant, |·| represents an absolute value function, and exp(·) represents an exponential function with e as the base. Similar to the feature similarity GCN, the relative distance adjacency matrix and the relative distance GCN update formula are constructed:

[0127] A dis =Softmax(dis)

[0128]

[0129] The feature dimensions of the two GCN outputs are half of the input features. The outputs of the two branches are spliced in the feature dimension to obtain the visual features of the target video.

[0130] As an embodiment of the present invention, inputting the visual-text features and the visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment graph includes:

[0131] The nonlinear activation function in the multi-layer perceptron of the pre-built visual-text cross-modal attention module is used to activate the visual text features to obtain nonlinear text features;

[0132] Use convolutional neural network to perform convolution processing on nonlinear text features to obtain text feature key matrix and text feature value matrix;

[0133] Use convolutional neural network to convolve visual features to obtain visual feature query matrix;

[0134] The text feature key matrix is transposed to obtain a transposed text feature key matrix, the similarity matrix between the transposed text feature key matrix and the visual feature query matrix is calculated, and the matrix is scaled according to the feature dimension of the similarity matrix to obtain a visual-text alignment graph.

[0135] For example, the visual-text features and the visual features are input into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment graph, and the following implementation steps can be used:

[0136] Step 1: First, use the two-layer MLP and skip connection to perform preliminary processing on the text features. The formula is described as follows:

[0137]

[0138] in The input dimension is f v The output dimension is 2f v The fully connected layer, The input dimension is 2f v The output dimension is f v The fully connected layer, GELU represents the nonlinear activation function GELU.

[0139] Step 2: Calculate Q based on a one-dimensional convolution with equal input and output dimensions t-v , K t-v With V t-v , where Q t-v Calculated by visual features, and K t-v With Vt-v It is calculated from text features and its formula is as follows:

[0140]

[0141] K t-v =Conv K (F′ t-v )

[0142] V t-v =Conv V (F′ t-v )

[0143] Among them, Conv Q It is a one-dimensional convolution in the t dimension, Conv K and Conv V It is a one-dimensional convolution in n dimensions, and the convolution kernel and convolution stride of the three one-dimensional convolutions are all 1. t-v K obtained with text features t-v Calculate the similarity matrix and scale it based on the dimension T of the matrix to obtain the visual-text alignment graph M v , and its formula is as follows: in, is the transpose of the text feature key matrix.

[0144] Furthermore, we use the visual-text alignment graph and visual-text features to construct cross-modal enhanced visual features, including:

[0145] After performing dot product operations on the visual-text alignment map and the text feature value matrix, a convolutional neural network is used to perform convolution operations and batch normalization processing to obtain preliminary cross-modal enhanced visual features;

[0146] The visual features of the target video are jump-connected with the preliminary cross-modal enhanced visual features to obtain the cross-modal enhanced visual features.

[0147] In the embodiment of the present invention, a skip connection refers to directly transmitting the output of a certain layer in the network to the input of a subsequent deeper layer, bypassing several intermediate layers and directly transmitting information across layers.

[0148] For example, the following steps may be used to construct cross-modal enhanced visual features using the visual-text alignment graph and visual-text features:

[0149] Aligning Visual-Textual Graph M v With v t-v Perform dot product operations and obtain preliminary cross-modal enhanced visual features through one-dimensional convolution and batch normalization The calculation formula is as follows:

[0150]

[0151] Among them, Conv W It is a one-dimensional convolution on the feature dimension, where both the convolution kernel and the convolution stride are 1, and BN stands for batch normalization.

[0152] Finally, the output of the visual temporal network Perform jump connections to obtain cross-modal enhanced visual features The formula is as follows:

[0153] S4. Extract the audio file from the target video, and use a preset audio-to-text extractor to extract the audio-to-text features of the audio file.

[0154] The embodiment of the present invention extracts the audio text features of the audio file by using the audio text extractor of AudioCLIP.

[0155] Exemplarily, the following implementation steps may be used to extract the audio text features of an audio file using a preset audio text extractor:

[0156] The learnable audio description words are largely similar to the learnable visual description prompt words, differing primarily in the use of different learnable prompt words and text encoders. Considering that the visual and audio descriptions of the same event may differ—for example, a video clip of lightning and thunder would have a visual description of "lightning" and an audio description of "thunder"—the visual and audio branches do not share learnable prompt word variables. Furthermore, given that the audio features come from the AudioCLIP audio encoder, an AudioCLIP-adapted text encoder was used to ensure alignment of the text features with the audio features.

[0157] S5. Use the audio time network to extract audio sequence features from the audio file, and input the audio text features and audio sequence features into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph, and use the audio-text alignment graph and audio text features to construct cross-modal enhanced audio features.

[0158] Exemplarily, audio sequence features are extracted from audio files using an audio time network, and the audio text features and audio sequence features are input into a pre-built audio-text cross-modal attention module to obtain an audio-text alignment graph. The audio-text alignment graph and audio text features are then used to construct cross-modal enhanced audio features. The following implementation steps can be used:

[0159] The audio-text cross-modal attention module is basically the same as the visual-text cross-modal attention module, except that the visual features are replaced by audio features and the audio-text alignment graph Ma and cross-modal enhanced audio features The calculation formula is as follows:

[0160]

[0161] Among them CrossAttn a Represents the audio-text cross-modal attention module.

[0162] S6. Perform a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain a cross-modal splicing feature, and input the cross-modal splicing feature into a pre-built anomaly classifier to obtain an anomaly classification result.

[0163] In the embodiment of the present invention, the pre-built anomaly classifier is constructed using a single fully connected layer and a Sigmoid normalization function.

[0164] For example, the cross-modal concatenation features are input into a pre-built anomaly classifier to obtain an anomaly classification result. The following implementation steps can be used:

[0165] Step 1: Combine the cross-modal enhanced features obtained from the visual branch and the audio branch and Splicing is performed on the feature dimension and input into the anomaly classifier consisting of a single-layer fully connected layer and a Sigmoid normalization function to obtain the binary anomaly score S classifier , and its formula is as follows:

[0166]

[0167] Concat represents the concatenation operation, FC c Represents a fully connected layer with an output dimension of 1.

[0168] Step 2: Determine the abnormal classification results based on the binary abnormality score.

[0169] Furthermore, for the binary classification task branch, the MIL loss based on Top-K is adopted. Specifically, the K scores with the highest binary classification anomaly scores in abnormal videos and normal videos are selected to calculate the binary cross entropy loss. The loss function is calculated as follows:

[0170]

[0171] where Ω K Representative from S classifier The largest K scores are selected from , and y∈{0,1} represents the label of the training video.

[0172] S7. Fuse the visual-text alignment map and the audio-text alignment map to obtain a multimodal fusion alignment map, calculate the event category probability map of the multimodal fusion alignment map in the feature dimension, and determine the event category based on the event category probability map to obtain the event category classification result.

[0173] For example, the visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map, an event category probability map of the multimodal fusion alignment map in the feature dimension is calculated, and the event category is determined based on the event category probability map to obtain an event category classification result. The following implementation steps can be used:

[0174] Combine the cross-modal enhanced features obtained from the visual branch and the audio branch and Splicing is performed on the feature dimension and input into the anomaly classifier consisting of a single-layer fully connected layer and a Sigmoid normalization function to obtain the binary anomaly score S classifier , and its formula is as follows:

[0175]

[0176] Concat represents the concatenation operation, FC c Represents a fully connected layer with an output dimension of 1.

[0177] (5) Multi-classification task branch

[0178] The visual-text alignment graph M obtained by the visual branch and the audio branch v With audio-text alignment graph M a Add element by element to get the multimodal fusion alignment map Using the Softmax normalization function in n dimensions, we can get the probability graph of n events corresponding to t moments. Multi-classification tasks can be achieved based on the highest event probability.

[0179] The calculation process of the multimodal fusion alignment map and the multi-classification probability map is as follows:

[0180] M m =M v +M a

[0181] P=Softmax(M m )

[0182] Furthermore, based on the multi-classification probability graph, the binary classification task can also be realized. Specifically, the probability sum of abnormal events is equivalent to the binary abnormal probability at that moment. Considering that the sum of the probabilities of all events after Softmax normalization is 1, the probability sum of abnormal events can be obtained by subtracting the probability of normal events from 1. Therefore, the binary abnormality score S of the multi-classification task branch is align The calculation formula is as follows:

[0183] S align =1-P normal

[0184] in Represents the probability of a normal event corresponding to time t.

[0185] For example, for the multi-classification task branch, the MIL-Align mechanism is adopted. Multi-classification probability map Represents the cross-modal similarity between vision, audio and all event text descriptions. Therefore, the largest K cross-modal similarities are selected in each row, and the average is taken. The Softmax function is used for normalization to obtain the alignment degree Z = {z1,...,z n}, and its formula is as follows:

[0186]

[0187] where Ω i,K represents the largest K cross-modal similarities in the i-th row.

[0188] The cross entropy loss is calculated based on the degree of alignment, and the loss function is calculated as follows:

[0189]

[0190] Contrastive loss is used to increase the distance between the text features of abnormal events and normal events. The distance between features is measured using cosine similarity. The contrastive loss function of the visual branch is calculated as follows:

[0191]

[0192] Among them, F t-v,1 Visual description text features representing normal events, F t-v,i ,i={2,..,n} represents the visual description text features of n-1 abnormal events, |·| represents the absolute value calculation, and ||·|| represents the L2 norm calculation. Similarly, the contrast loss of the audio branch can be obtained, and its loss function is calculated as follows:

[0193]

[0194] Combining all the above losses can get the final target optimization function The formula is as follows:

[0195]

[0196] Among them, λ is a hyperparameter.

[0197] In an embodiment of the present invention, the MIL-Align (Multi-Instance Learning Alignment) mechanism refers to an alignment optimization mechanism designed for weakly supervised scenarios, which is mainly used to achieve cross-modal (such as vision and language) or cross-time segment feature alignment in multimodal or time series data.

[0198] The present invention generates visual complete sentence prompts based on visual text category labels, which can convert discrete visual text category labels into complete sentence descriptions, and make full use of the semantic information of the category labels at the visual level in the target video. In addition, the visual features of the target video are extracted by using the visual time network, which can realize the correlation of the visual information in the target video in the time domain dimension. In addition, the visual text features and visual features are input into the pre-built visual-text cross-modal attention module to obtain the visual-text alignment graph, and the visual-text alignment graph and visual text features are used to construct cross-modal enhanced visual features, which can improve the correlation between visual features and text features and realize visual-text cross-modal alignment. In addition, the audio text features and audio sequence The column features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment map, and the audio-text alignment map and audio-text features are used to construct cross-modal enhanced audio features, which can expand the perception dimension of the target video. The cross-modal enhanced visual features and the cross-modal enhanced audio features are spliced on the feature dimension to obtain cross-modal splicing features. Combining the splicing features of multiple modalities and performing anomaly classification can improve the accuracy of anomaly classification. In addition, the visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map, and the event category probability map of the multimodal fusion alignment map on the feature dimension is calculated. The event category is judged based on the event category probability map, which can improve the accuracy of event category classification.

[0199] Reference Figure 2 , which is a schematic diagram of an anomaly detection model of a video anomaly detection method based on multimodal feature alignment provided by an embodiment of the present invention.

[0200] Reference Figure 3 , which is a schematic diagram of a local Transformer of a video anomaly detection method based on multimodal feature alignment provided by an embodiment of the present invention.

[0201] Reference Figure 4, which is a global GCN network diagram of a video anomaly detection method based on multimodal feature alignment provided by an embodiment of the present invention.

[0202] Reference Figure 5 , which is a schematic diagram of visual-text cross-modal attention for a video anomaly detection method based on multimodal feature alignment provided by an embodiment of the present invention.

[0203] like Figure 6 , which is a functional module diagram of a video anomaly detection device based on multimodal feature alignment provided by one embodiment of the present invention.

[0204] The video anomaly detection device 100 based on multimodal feature alignment of the present invention can be installed in an electronic device. Depending on the functions implemented, the video anomaly detection device 100 based on multimodal feature alignment can include a visual feature acquisition module 101, an audio feature acquisition module 102, and a video anomaly detection module 103.

[0205] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0206] In this embodiment, the functions of each module / unit are as follows:

[0207] The visual feature acquisition module 101 is used to acquire a target video and extract the visual text category label of the target video; generate a visual complete sentence prompt based on the visual text category label, and input the visual complete sentence prompt into a preset text encoder to obtain visual text features; use a visual time network to extract the visual features of the target video, and input the visual text features and visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map, and use the visual-text alignment map and the visual text features to construct cross-modal enhanced visual features.

[0208] In the embodiment of the present invention, the target video refers to a video file having multiple abnormal event categories, wherein the event category can be a normal event category or an abnormal event category.

[0209] In the embodiment of the present invention, the visual text category label refers to an overall summary of the video content in the target video.

[0210] As an embodiment of the present invention, generating a visual complete sentence prompt based on a visual text category label includes:

[0211] Convert visual text category labels into category tags using the word segmenter of the pre-built multimodal pre-trained model;

[0212] Inserting category tags into a learnable context tag sequence yields visually complete sentence hints.

[0213] For example, generating a visual complete sentence prompt based on the visual text category label and inputting the visual complete sentence prompt into a preset text encoder to obtain visual text features can be implemented using the following steps:

[0214] Step 1: Use CLIP's tokenizer to convert text category labels into category tokens. The formula is as follows:

[0215] t class =Tokenizer(Label)

[0216] Among them, Label represents the text category label, t class For category labels, for example, if the target video is an XD-Violence video, the text category labels are normal, fighting, shooting, riot, abuse, car accident, and explosion;

[0217] Step 2: Mark the obtained category t class Insert a learnable context token sequence of length len {c1,...,c len}, get the visual complete sentence prompt t p , and its formula is as follows:

[0218] t p ={c1,...,t class ,...,c len}

[0219] The visual complete sentence prompts corresponding to n categories are t p Input CLIP's text encoder to get visual text features Among them, f v For dimension.

[0220] In an embodiment of the present invention, the preset text encoder may adopt the text encoder in the CLIP (Contrastive Language-Image Pre-training) multimodal pre-training model.

[0221] In an embodiment of the present invention, a visual temporal network refers to a network model that processes the visual information in a target video in the time domain. The visual temporal network in an embodiment of the present invention is composed of a local Transformer encoder for extracting short-term temporal associations and a global GCN (Graph Convolutional Network) for encoding long-term temporal associations.

[0222] In one embodiment of the present invention, extracting visual features of a target video using a visual temporal network includes:

[0223] The target video is processed into non-overlapping windows of preset length to obtain multi-window video segments, and the visual features of each window video segment are extracted using the image encoder in the pre-built multimodal pre-trained model. The visual features of each window video segment are spliced in the order of the windows to obtain a visual feature sequence;

[0224] Position encoding is performed on multi-window video clips, and the position encoding is added to the visual feature sequence to obtain the position-embedded visual features;

[0225] The position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features;

[0226] Input the local visual features into the pre-built global graph convolutional network, and use the feature similarity network in the pre-built global graph convolutional network to calculate the feature similarity feature matrix, and use the relative distance network in the pre-built global graph convolutional network to calculate the relative distance feature matrix;

[0227] The feature similarity feature matrix and the relative distance feature matrix are concatenated in the feature dimension to obtain the visual features of the target video.

[0228] In an embodiment of the present invention, the pre-built multimodal pre-training model may adopt a CLIP (Contrastive Language-Image Pre-training) model.

[0229] Furthermore, the position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features, including:

[0230] Divide the position-embedded visual features into multiple positions of the same number of attention heads as the multi-head attention module;

[0231] The embedded visual features of each position are input into each attention head for calculation to obtain the query matrix, key matrix and value matrix of each attention head;

[0232] The query matrix is dot-producted with the key matrix and scaled before being added to the pre-built attention mask matrix to obtain the attention map.

[0233] After normalizing the attention map, perform dot product calculation with the value matrix to obtain the output visual features of each attention head;

[0234] The output visual features of each attention head are concatenated to obtain local visual features.

[0235] Furthermore, the output visual features of each attention head are concatenated to obtain local visual features, including:

[0236] Local visual feature F′ v It is expressed using the following formula:

[0237] F′ v =FC2(GELU(FC1(LN(F v-attn ))))+F v-embed

[0238] Among them, FC1 is the input dimension f v The output dimension is 4f v The fully connected layer, FC2 is the input dimension of 4f v The output dimension is f v The fully connected layer, GELU is the nonlinear activation function GELU, F v-embed is the visual feature after position embedding, F v-attn is the output of the multi-head attention module.

[0239] For example, the following steps may be used to extract visual features of a target video using a visual temporal network:

[0240] Step a) Local Transformer:

[0241] A non-overlapping sliding window of length win is used on the video feature sequence to limit the Transformer's self-attention calculation to the inside of the sliding window. For the input visual feature sequence F with a time sequence length win, v , first generate the corresponding position code P = {p1,...,p win}, where p i =i, and after embedding it, the embedded position code is added to the visual feature. The formula is as follows:

[0242] F v-embed =F v +FPE(P)

[0243] Among them F v-embedIt represents the visual features after position embedding, and FPE stands for Frame Position Embeddings.

[0244] The visual features after position embedding are normalized by the layer and the length of the attention window is win attn Attention mask Mask attn Input the multi-head attention module together. For n heads heads The multi-head attention first divides the input features into n heads Servings, that is Enter n respectively heads The calculation is performed by the attention head, and the formula is as follows:

[0245] Q i =FC Q,i (LN(F v-embed,i ))

[0246] K i =FC K,i (LN(F v-embed,i ))

[0247] V i =FC V,i (LN(F v-embed,i ))

[0248] Among them F v-embed,i represents the input of the i-th head, Q i is the query matrix obtained by the i-th multi-attention head, K i is the key matrix obtained by the i-th multi-attention head, V i is the value matrix obtained by the i-th multi-attention head, FC Q,i 、FC K,i 、FC V,i represents a fully connected layer with equal input and output dimensions for the i-th attention head, and LN represents a layer normalization layer.

[0249] Based on the obtained Q i With K i Calculate the dot product and scale it, then add it to the attention mask. The purpose of the attention mask is to make the model ignore the attention calculation of some areas. Here, the attention mask only focuses on the attention calculation within the non-overlapping attention window, only focuses on the attention value of the light gray part, and ignores the attention value of the dark gray part. The specific method is to create a matrix equal to the target size, and make the mask value of the part that needs to be paid attention to 0, and the mask value of the unnecessary part is set to -∞. Based on the attention mask, the attention map M is obtained attn,i , and its formula is as follows:

[0250]

[0251] The obtained attention map is normalized by the Softmax function and then compared with V i Perform dot product calculation to get the output F of the i-th attention head v-attn,i , and n heads The outputs of the heads are spliced together to obtain the complete attention module output F v-attn , and its formula is as follows:

[0252] f v-attn,i =Softmax(M attn,i )·V i

[0253]

[0254] Where Concat represents the concatenation operation, and the output F of the multi-head attention module v-attn After layer normalization, two-layer MLP and jump connection, local visual features can be obtained, and the formula is as follows:

[0255] F′ v =FC2(GELU(FC1(LN(F v-attn ))))+F v-embed

[0256] Among them, FC1 is the input dimension f v The output dimension is 4f v The fully connected layer, FC2 is the input dimension of 4f v The output dimension is f v The fully connected layer, GELU is the nonlinear activation function GELU, F v-embed is the visual feature after position embedding, F v-attn is the output of the multi-head attention module.

[0257] Step b) Global GCN:

[0258] In order to further capture the global temporal association, the global GCN network is used after the local Transformer obtains the local visual features. The second-layer feature similarity GCN and the second-layer relative distance GCN are used to model the temporal association.

[0259] First, construct the feature similarity adjacency matrix A based on cosine feature similarity sim , where the similarity sim between the i-th segment feature and the j-th segment feature i,j is defined as follows:

[0260]

[0261] Here, ||·|| represents the L2 norm calculation. The obtained similarity is thresholded to ignore weakly associated pairs and focus on feature pairs with higher similarity. The thresholding is as follows:

[0262]

[0263] Where τ represents the threshold. After the above processing, the similarity matrix is normalized by Softmax to obtain the feature similarity adjacency matrix, which is defined as follows:

[0264] A sim =Softmax(sim)

[0265] Based on the feature similarity adjacency matrix, the update formula of feature similarity GCN can be obtained:

[0266]

[0267] in and Represent the node feature matrices of the lth layer and the l+1th layer respectively, is the learnable weight matrix of the lth layer, and GELU represents the nonlinear activation function GELU.

[0268] Construct relative distance adjacency matrix A based on relative distance dis , where the relative distance between the i-th segment feature and the j-th segment feature is dis i,j is defined as follows:

[0269]

[0270] Where e represents a natural constant, |·| represents an absolute value function, and exp(·) represents an exponential function with e as the base. Similar to the feature similarity GCN, the relative distance adjacency matrix and the relative distance GCN update formula are constructed:

[0271] A dis =Softmax(dis)

[0272]

[0273] The feature dimensions of the two GCN outputs are half of the input features. The outputs of the two branches are spliced in the feature dimension to obtain the visual features of the target video.

[0274] As an embodiment of the present invention, inputting the visual-text features and the visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment graph includes:

[0275] The nonlinear activation function in the multi-layer perceptron of the pre-built visual-text cross-modal attention module is used to activate the visual text features to obtain nonlinear text features;

[0276] Use convolutional neural network to perform convolution processing on nonlinear text features to obtain text feature key matrix and text feature value matrix;

[0277] Use convolutional neural network to convolve visual features to obtain visual feature query matrix;

[0278] The text feature key matrix is transposed to obtain a transposed text feature key matrix, the similarity matrix between the transposed text feature key matrix and the visual feature query matrix is calculated, and the matrix is scaled according to the feature dimension of the similarity matrix to obtain a visual-text alignment graph.

[0279] For example, the visual-text features and the visual features are input into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment graph, and the following implementation steps can be used:

[0280] Step 1: First, use the two-layer MLP and skip connection to perform preliminary processing on the text features. The formula is described as follows:

[0281]

[0282] in The input dimension is f v The output dimension is 2f v The fully connected layer, The input dimension is 2f v The output dimension is f v The fully connected layer, GELU represents the nonlinear activation function GELU.

[0283] Step 2: Calculate Q based on a one-dimensional convolution with equal input and output dimensions t-v , K t-v With V t-v , where Q t-v Calculated by visual features, and K t-v With V t-v It is calculated from text features and its formula is as follows:

[0284]

[0285] K t-v =Conv K (F′ t-v )

[0286] V t-v =Conv V (F′ t-v )

[0287] Among them, Conv Q It is a one-dimensional convolution in the t dimension, Conv K and Conv V It is a one-dimensional convolution in n dimensions, and the convolution kernel and convolution stride of the three one-dimensional convolutions are all 1. t-v K obtained with text features t-v Calculate the similarity matrix and scale it based on the dimension t of the matrix to obtain the visual-text alignment graph M v , and its formula is as follows: in, is the transpose of the text feature key matrix.

[0288] Furthermore, we use the visual-text alignment graph and visual-text features to construct cross-modal enhanced visual features, including:

[0289] After performing dot product operations on the visual-text alignment map and the text feature value matrix, a convolutional neural network is used to perform convolution operations and batch normalization processing to obtain preliminary cross-modal enhanced visual features;

[0290] The visual features of the target video are jump-connected with the preliminary cross-modal enhanced visual features to obtain the cross-modal enhanced visual features.

[0291] In the embodiment of the present invention, a skip connection refers to directly transmitting the output of a certain layer in the network to the input of a subsequent deeper layer, bypassing several intermediate layers and directly transmitting information across layers.

[0292] For example, the following steps may be used to construct cross-modal enhanced visual features using the visual-text alignment graph and visual-text features:

[0293] Aligning Visual-Textual Graph M v With V t-v Perform dot product operations and obtain preliminary cross-modal enhanced visual features through one-dimensional convolution and batch normalization The calculation formula is as follows:

[0294]

[0295] Among them, Conv W It is a one-dimensional convolution on the feature dimension, where both the convolution kernel and the convolution stride are 1, and BN stands for batch normalization.

[0296] Finally, the output of the visual temporal network Perform jump connections to obtain cross-modal enhanced visual features The formula is as follows:

[0297] The audio feature acquisition module 102 is used to extract the audio file in the target video and use a preset audio-text extractor to extract the audio-text features of the audio file; use the audio time network to extract the audio sequence features in the audio file, and input the audio-text features and audio sequence features into a pre-built audio-text cross-modal attention module to obtain an audio-text alignment graph, and use the audio-text alignment graph and audio-text features to construct cross-modal enhanced audio features.

[0298] The embodiment of the present invention extracts the audio text features of the audio file by using the audio text extractor of AudioCLIP.

[0299] Exemplarily, the following implementation steps may be used to extract the audio text features of an audio file using a preset audio text extractor:

[0300] The learnable audio description words are largely similar to the learnable visual description prompt words, differing primarily in the use of different learnable prompt words and text encoders. Considering that the visual and audio descriptions of the same event may differ—for example, a video clip of lightning and thunder would have a visual description of "lightning" and an audio description of "thunder"—the visual and audio branches do not share learnable prompt word variables. Furthermore, given that the audio features come from the AudioCLIP audio encoder, an AudioCLIP-adapted text encoder was used to ensure alignment of the text features with the audio features.

[0301] Exemplarily, audio sequence features are extracted from audio files using an audio time network, and the audio text features and audio sequence features are input into a pre-built audio-text cross-modal attention module to obtain an audio-text alignment graph. The audio-text alignment graph and audio text features are then used to construct cross-modal enhanced audio features. The following implementation steps can be used:

[0302] The audio-text cross-modal attention module is basically the same as the visual-text cross-modal attention module, except that the visual features are replaced by audio features and the audio-text alignment graph M a and cross-modal enhanced audio features The calculation formula is as follows:

[0303]

[0304] Among them CrossAttn a Represents the audio-text cross-modal attention module.

[0305] The video anomaly detection module 103 is used to perform a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain a cross-modal splicing feature, and input the cross-modal splicing feature into a pre-built anomaly classifier to obtain an anomaly classification result; fuse the visual-text alignment map and the audio-text alignment map to obtain a multimodal fusion alignment map, calculate the event category probability map of the multimodal fusion alignment map in the feature dimension, and judge the event category based on the event category probability map to obtain an event category classification result.

[0306] In the embodiment of the present invention, the pre-built anomaly classifier is constructed using a single fully connected layer and a Sigmoid normalization function.

[0307] For example, the cross-modal concatenation features are input into a pre-built anomaly classifier to obtain an anomaly classification result. The following implementation steps can be used:

[0308] Step 1: Combine the cross-modal enhanced features obtained from the visual branch and the audio branch and Splicing is performed on the feature dimension and input into the anomaly classifier consisting of a single-layer fully connected layer and a Sigmoid normalization function to obtain the binary anomaly score S classifier , and its formula is as follows:

[0309]

[0310] Concat represents the concatenation operation, FC c Represents a fully connected layer with an output dimension of 1.

[0311] Step 2: Determine the abnormal classification results based on the binary abnormality score.

[0312] Furthermore, for the binary classification task branch, the MIL loss based on Top-K is adopted. Specifically, the K scores with the highest binary classification anomaly scores in abnormal videos and normal videos are selected to calculate the binary cross entropy loss. The loss function is calculated as follows:

[0313]

[0314] where Ω K Representative from S classifier The largest K scores are selected from , and y∈{0,1} represents the label of the training video.

[0315] S7. Fuse the visual-text alignment map and the audio-text alignment map to obtain a multimodal fusion alignment map, calculate the event category probability map of the multimodal fusion alignment map in the feature dimension, and determine the event category based on the event category probability map to obtain the event category classification result.

[0316] For example, the visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map, an event category probability map of the multimodal fusion alignment map in the feature dimension is calculated, and the event category is determined based on the event category probability map to obtain an event category classification result. The following implementation steps can be used:

[0317] Combine the cross-modal enhanced features obtained from the visual branch and the audio branch and Splicing is performed on the feature dimension and input into the anomaly classifier consisting of a single-layer fully connected layer and a Sigmoid normalization function to obtain the binary anomaly score S classifier , and its formula is as follows:

[0318]

[0319] Concat represents the concatenation operation, FC c Represents a fully connected layer with an output dimension of 1.

[0320] (5) Multi-classification task branch

[0321] The visual-text alignment graph M obtained by the visual branch and the audio branch v With audio-text alignment graph M a Add element by element to get the multimodal fusion alignment map Using the Softmax normalization function in n dimensions, we can get the probability graph of n events corresponding to t moments. Multi-classification tasks can be achieved based on the highest event probability.

[0322] The calculation process of the multimodal fusion alignment map and the multi-classification probability map is as follows:

[0323] M m =M v +M a

[0324] P=Softmax(M m )

[0325] Furthermore, based on the multi-classification probability graph, the binary classification task can also be realized. Specifically, the probability sum of abnormal events is equivalent to the binary abnormal probability at that moment. Considering that the sum of the probabilities of all events after Softmax normalization is 1, the probability sum of abnormal events can be obtained by subtracting the probability of normal events from 1. Therefore, the binary abnormality score S of the multi-classification task branch is align The calculation formula is as follows:

[0326] S align =1-P normal

[0327] in Represents the probability of a normal event corresponding to time t.

[0328] For example, for the multi-classification task branch, the MIL-Align mechanism is adopted. Multi-classification probability map Represents the cross-modal similarity between vision, audio and all event text descriptions. Therefore, the largest K cross-modal similarities are selected in each row, and the average is taken. The Softmax function is used for normalization to obtain the alignment degree Z = {z1,...,z n}, and its formula is as follows:

[0329]

[0330] where Ω i,K represents the largest K cross-modal similarities in the i-th row.

[0331] The cross entropy loss is calculated based on the degree of alignment, and the loss function is calculated as follows:

[0332]

[0333] Contrastive loss is used to increase the distance between the text features of abnormal events and normal events. The distance between features is measured using cosine similarity. The contrastive loss function of the visual branch is calculated as follows:

[0334]

[0335] Among them, F t-v,1 Visual description text features representing normal events, F t-v,i ,i={2,..,n} represents the visual description text features of n-1 abnormal events, |·| represents the absolute value calculation, and ||·|| represents the L2 norm calculation. Similarly, the contrast loss of the audio branch can be obtained, and its loss function is calculated as follows:

[0336]

[0337] Combining all the above losses can get the final target optimization function The formula is as follows:

[0338]

[0339] Among them, λ is a hyperparameter.

[0340] like Figure 7 , which is a structural diagram of an electronic device for implementing a video anomaly detection method based on multimodal feature alignment provided by an embodiment of the present invention.

[0341] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a video anomaly detection method program based on multimodal feature alignment.

[0342] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the memory 11 (for example, executing a video anomaly detection method program based on multimodal feature alignment, etc.), as well as calling data stored in the memory 11, to execute various functions of the electronic device and process data.

[0343] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of a video anomaly detection method program based on multimodal feature alignment, but can also be used to temporarily store data that has been output or is about to be output.

[0344] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10.

[0345] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0346] Figure 7 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 7 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0347] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0348] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0349] The video anomaly detection method program based on multimodal feature alignment stored in the memory 11 of the electronic device is a combination of multiple instructions. When running in the processor 10, it can achieve the following:

[0350] Get the target video and extract the visual text category label of the target video;

[0351] Generate visual complete sentence prompts based on visual text category labels, and input the visual complete sentence prompts into a preset text encoder to obtain visual text features;

[0352] The visual features of the target video are extracted using a visual temporal network, and the visual-text features and visual features are input into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map. The visual-text alignment map and visual-text features are then used to construct cross-modal enhanced visual features.

[0353] Extract the audio file from the target video and use the preset audio text extractor to extract the audio text features of the audio file;

[0354] The audio time network is used to extract audio sequence features from audio files, and the audio text features and audio sequence features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph. The audio-text alignment graph and audio text features are then used to construct cross-modal enhanced audio features.

[0355] Performing a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain a cross-modal splicing feature, and inputting the cross-modal splicing feature into a pre-built anomaly classifier to obtain an anomaly classification result;

[0356] The visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map. The event category probability map of the multimodal fusion alignment map in the feature dimension is calculated, and the event category is determined based on the event category probability map to obtain the event category classification result.

[0357] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0358] Furthermore, if the modules / units integrated into the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0359] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, the computer program can implement:

[0360] Get the target video and extract the visual text category label of the target video;

[0361] Generate visual complete sentence prompts based on visual text category labels, and input the visual complete sentence prompts into a preset text encoder to obtain visual text features;

[0362] The visual features of the target video are extracted using a visual temporal network, and the visual-text features and visual features are input into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map. The visual-text alignment map and visual-text features are then used to construct cross-modal enhanced visual features.

[0363] Extract the audio file from the target video and use the preset audio text extractor to extract the audio text features of the audio file;

[0364] The audio time network is used to extract audio sequence features from audio files, and the audio text features and audio sequence features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph. The audio-text alignment graph and audio text features are then used to construct cross-modal enhanced audio features.

[0365] Performing a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain a cross-modal splicing feature, and inputting the cross-modal splicing feature into a pre-built anomaly classifier to obtain an anomaly classification result;

[0366] The visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map. The event category probability map of the multimodal fusion alignment map in the feature dimension is calculated, and the event category is determined based on the event category probability map to obtain the event category classification result.

[0367] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0368] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0369] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0370] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0371] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0372] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0373] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0374] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

[0375] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video anomaly detection method based on multimodal feature alignment, characterized in that: The method comprises: Obtain the target video and its visual text category label; Generate visual complete sentence prompts based on visual text category labels, and input the visual complete sentence prompts into a preset text encoder to obtain visual text features; The visual features of the target video are extracted using a visual temporal network, and the visual-text features and visual features are input into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map. The visual-text alignment map and visual-text features are then used to construct cross-modal enhanced visual features. Extract the audio file from the target video and use the preset audio text extractor to extract the audio text features of the audio file; The audio time network is used to extract audio sequence features from audio files, and the audio text features and audio sequence features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph. The audio-text alignment graph and audio text features are then used to construct cross-modal enhanced audio features. Performing a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain a cross-modal splicing feature, and inputting the cross-modal splicing feature into a pre-built anomaly classifier to obtain an anomaly classification result; The visual-text alignment map and the audio-text alignment map are fused to obtain a multimodal fusion alignment map. The event category probability map of the multimodal fusion alignment map in the feature dimension is calculated, and the event category is determined based on the event category probability map to obtain the event category classification result.

2. The video anomaly detection method based on multimodal feature alignment according to claim 1, wherein: Generating a visual complete sentence prompt according to the visual text category label includes: Use the word segmenter of the pre-built multimodal pre-trained model to convert visual text category labels into category tokens; Inserting category tags into a learnable context tag sequence yields visually complete sentence hints.

3. The video anomaly detection method based on multimodal feature alignment according to claim 1, wherein: The method of extracting visual features of a target video using a visual time network includes: The target video is processed into non-overlapping windows of preset length to obtain multi-window video segments, and the visual features of each window video segment are extracted using the image encoder in the pre-built multimodal pre-trained model. The visual features of each window video segment are spliced in the order of the windows to obtain a visual feature sequence; Position encoding is performed on multi-window video clips, and the position encoding is added to the visual feature sequence to obtain the position-embedded visual features; The position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features; Input the local visual features into the pre-built global graph convolutional network, and use the feature similarity network in the pre-built global graph convolutional network to calculate the feature similarity feature matrix, and use the relative distance network in the pre-built global graph convolutional network to calculate the relative distance feature matrix; The feature similarity feature matrix and the relative distance feature matrix are concatenated in the feature dimension to obtain the visual features of the target video.

4. The video anomaly detection method based on multimodal feature alignment according to claim 3, wherein: The position-embedded visual features and the pre-built attention mask matrix are input into the multi-head attention module to obtain local visual features, including: Divide the position-embedded visual features into multiple positions of the same number of attention heads as the multi-head attention module; The embedded visual features of each position are input into each attention head for calculation to obtain the query matrix, key matrix and value matrix of each attention head; The query matrix is dot-producted with the key matrix and scaled before being added to the pre-built attention mask matrix to obtain the attention map. After normalizing the attention map, perform dot product calculation with the value matrix to obtain the output visual features of each attention head; The output visual features of each attention head are concatenated to obtain local visual features.

5. The video anomaly detection method based on multimodal feature alignment according to claim 4, characterized in that: The output visual features of each attention head are concatenated to obtain local visual features, including: Local visual feature F′ v It is expressed using the following formula: F′ v =FC2(GELU(FC1(LN(F v-attn ))))+F v-embed Among them, FC1 is the input dimension f v The output dimension is 4f v The fully connected layer, FC2 is the input dimension of 4f v The output dimension is f v The fully connected layer, GELU is the nonlinear activation function GELU, F v-embed is the visual feature after position embedding, F v-attn is the output of the multi-head attention module.

6. The video anomaly detection method based on multimodal feature alignment according to claim 1, wherein: The step of inputting the visual-text features and the visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment graph includes: The nonlinear activation function in the multi-layer perceptron of the pre-built visual-text cross-modal attention module is used to activate the visual text features to obtain nonlinear text features; Use convolutional neural network to perform convolution processing on nonlinear text features to obtain text feature key matrix and text feature value matrix; Use convolutional neural network to convolve visual features to obtain visual feature query matrix; The text feature key matrix is transposed to obtain a transposed text feature key matrix, the similarity matrix between the transposed text feature key matrix and the visual feature query matrix is calculated, and the matrix is scaled according to the feature dimension of the similarity matrix to obtain a visual-text alignment graph.

7. The video anomaly detection method based on multimodal feature alignment according to claim 1 or 6, characterized in that: The method of constructing cross-modal enhanced visual features using the visual-text alignment graph and visual-text features includes: After performing dot product operations on the visual-text alignment map and the text feature value matrix, a convolutional neural network is used to perform convolution operations and batch normalization processing to obtain preliminary cross-modal enhanced visual features; The visual features of the target video are jump-connected with the preliminary cross-modal enhanced visual features to obtain the cross-modal enhanced visual features.

8. A video anomaly detection device based on multimodal feature alignment, characterized in that: The device implements the video anomaly detection method based on multimodal feature alignment according to any one of claims 1 to 7, and the device includes: The visual feature acquisition module is used to acquire the target video and extract the visual text category label of the target video; generate a visual complete sentence prompt based on the visual text category label, and input the visual complete sentence prompt into a preset text encoder to obtain visual text features; use the visual time network to extract the visual features of the target video, and input the visual text features and visual features into a pre-built visual-text cross-modal attention module to obtain a visual-text alignment map, and use the visual-text alignment map and visual text features to construct cross-modal enhanced visual features; The audio feature acquisition module is used to extract the audio file in the target video and use the preset audio-text extractor to extract the audio-text features of the audio file; the audio sequence features in the audio file are extracted using the audio time network, and the audio-text features and audio sequence features are input into the pre-built audio-text cross-modal attention module to obtain the audio-text alignment graph, and the audio-text alignment graph and audio-text features are used to construct cross-modal enhanced audio features; The video anomaly detection module is used to perform a splicing operation on the cross-modal enhanced visual features and the cross-modal enhanced audio features in the feature dimension to obtain cross-modal splicing features, and input the cross-modal splicing features into a pre-built anomaly classifier to obtain anomaly classification results; fuse the visual-text alignment map and the audio-text alignment map to obtain a multimodal fusion alignment map, calculate the event category probability map of the multimodal fusion alignment map in the feature dimension, and judge the event category based on the event category probability map to obtain the event category classification result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video anomaly detection method based on multimodal feature alignment as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video anomaly detection method based on multimodal feature alignment as described in any one of claims 1 to 7 is implemented.

Images