A multi-scene monitoring video anomaly detection method based on a cross-modal large model

Abstract

This invention provides a method for anomaly detection in multi-scene surveillance videos based on a cross-modal large model, relating to the technical field of anomaly detection. The method includes: using a large language model to generate normal and abnormal descriptions of the scene content of the surveillance video, creating a public text library; using a pre-trained CLIP model to obtain image feature vectors and text feature vectors respectively, and then constructing an initial similarity matrix by calculating the cosine similarity between the image feature vectors and text feature vectors; performing lightweight adaptive correction on the image feature vectors and text feature vectors respectively; to capture the temporal dependencies of the video sequence, using a simplified Transformer encoder to model the temporal features of the processed image feature vectors; using a simplified Transformer encoder for text reconstruction; introducing cross-modal semantic constraints; calculating anomaly scores for the image feature vectors and text feature vectors respectively, and integrating the two into a final score.

Description

Technical Field

[0001] This invention relates to the technical field of anomaly detection, and more particularly to a method for anomaly detection in multi-scene surveillance videos based on a cross-modal large model. Background Technology

[0002] With the widespread application of video surveillance systems in public safety, urban management, and intelligent transportation, the scale of video data generated by surveillance equipment is constantly growing. How to automatically identify abnormal events from massive amounts of surveillance video has become a crucial problem that urgently needs to be solved in the field of intelligent surveillance. Video anomaly detection technology, by analyzing surveillance video content and identifying events inconsistent with normal behavioral patterns, is of great significance for improving the intelligence level and practical application value of surveillance systems.

[0003] Existing video anomaly detection methods typically model based on visual features, often employing a learning approach on normal video samples to establish a consistent model of target appearance, motion patterns, or temporal characteristics. Deviations from this model are then classified as anomalous events. While these methods can achieve some success in specific monitoring scenarios, they generally rely on the consistency assumptions of low- or mid-level visual features, making it difficult to effectively express the high-level semantic information contained in anomalous events, thus limiting their ability to understand them. In practical monitoring applications, different monitoring scenarios exhibit significant differences in environmental structure, shooting angle, personnel density, and behavioral patterns, resulting in diverse and uncertain manifestations of anomalous events. Because existing methods heavily rely on the distribution of scene-related visual features, their detection performance tends to degrade when applied to multiple scenarios or unseen anomaly types, and their model structures and training processes are often quite complex. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this paper provides a multi-scenario surveillance video anomaly detection method based on a cross-modal large model. This method proposes a video anomaly detection approach that incorporates high-level semantic information, enhances the ability to express abnormal events, and is applicable to multi-scenario surveillance environments, thereby improving the generalization performance and practical application effectiveness of anomaly detection.

[0005] The technical means employed in this invention are as follows:

[0006] A method for anomaly detection in multi-scene surveillance videos based on a cross-modal large model utilizes cross-modal semantic priors to guide model learning and reduces model complexity and improves model generalization ability through public text libraries and image-text feature alignment design. The method is characterized by the following steps: Step 1: Use a large language model to generate normal and abnormal descriptions of the scene content in the surveillance video, and generate a public text library; Step 2: After obtaining the video frames and text database, the pre-trained CLIP model is used to obtain image feature vectors and text feature vectors respectively. Then, the cosine similarity between the image feature vectors and text feature vectors is calculated to construct an initial similarity matrix. ; Step 3: Perform lightweight adaptive correction on the image feature vector and the text feature vector respectively; Step 4: To capture the temporal dependencies of the video sequence, a simplified Transformer encoder is used to model the temporal features of the image feature vectors processed in Step 3. Step 5: Reconstruct the text using the simplified Transformer encoder described above; Step 6: To ensure that the temporal modeling process does not deviate from the cross-modal semantic alignment space established by CLIP, cross-modal semantic constraints are introduced; Step 7: Calculate anomaly scores for image feature vectors and text feature vectors respectively, and integrate the two into the final score.

[0007] Furthermore, in step 1, the label for normal video frames is set to normal, and the label for abnormal video frames is set to abnormal.

[0008] Furthermore, step 2 includes the following steps: Step 21: The CLIP image encoder divides the input video frame into fixed-size image blocks, extracts visual features through a multi-layer Transformer encoder, and then passes the linear projection layer... Mapping to the multimodal embedding space and performing L2 normalization yields the image feature vector. Meanwhile, the CLIP text encoder extracts text features through a multi-layer Transformer encoder and then through a linear projection layer. Mapping to the same multimodal embedding space and normalizing it yields the text feature vector. ; Step 22: Construct an initial similarity matrix reflecting the cross-modal semantic association strength between video frames and text descriptions by calculating the cosine similarity between image feature vectors and text feature vectors. .

[0009] Furthermore, the lightweight formulas for the image feature vector and text feature vector in step 3 are as follows: ; in, and These represent the weight matrix and the offset, respectively.

[0010] Furthermore, step 4 includes the following steps: Step 41: Generate position codes using sine / cosine functions. The calculation formula is: ; ; in, Indicates the position index. An index representing half the encoding dimension. Indicates the hidden layer dimension; Step 42: Add the position code to the image features extracted by CLIP to obtain: This allows each frame feature to carry its temporal location information; Step 43: Map the input features to the query matrix through a linear transformation. Key matrix Sum matrix ; Step 44: Calculate the self-attention weights for each attention head. The formula is: ; in, , and These represent the query, key, and value vectors obtained from the linear transformation, respectively. This represents the dimension of the key vector. Indicates the scaling factor; Step 45: Focus all attention on the output of the head. The multi-head attention output is obtained by concatenating the data and performing a linear transformation on the output weight matrix. The formula is: ; Step 46: The multi-head attention output is then fed into the feedforward neural network (FFN) for nonlinear transformation to obtain the feedforward neural network result. The calculation formula is: ; in, , Represents the weight matrix. , Indicates the bias term. Indicates the activation function; Step 47: Add residual connections and layer normalization operations after the multi-head attention layer and the feedforward network layer, respectively, to obtain the normalization result. The formula is: ; Where x is the multi-head attention output and the result of the feedforward neural network; Step 48: Normalize the results Perform a linear mapping to obtain video features i containing temporal information. trans The formula is expressed as: .

[0011] Furthermore, step 6 includes the following steps: Step 61: Calculate the similarity matrix between the temporal features extracted by the Transformer and the text features. ; Step 62: Calculate the initial similarity matrix using CLIP. As a supervisory signal, the cross-entropy loss between the two similarity matrices is calculated. .

[0012] Furthermore, the anomaly score of the image feature vector is calculated as follows: for each sample, select from the normal similarity set... The maximum value Then, the average of the selected similarities is taken to obtain the normal semantic similarity score of the sample. The formula is as follows: ; ; in, express The similarity matrix between the sample and the normal semantic prototype. Representing normal semantics parameter; For the set of anomalous semantic similarities, hard negative samples are used to obtain the anomalous semantic similarity score. The formula is as follows:

[0013] in, express Similarity matrix between the medium sample and the abnormal semantic prototype.

[0014] Define sample-level anomaly scoring as The calculation formula is as follows: .

[0015] Furthermore, the anomaly scoring process for the text feature vector is as follows: It measures the weighted semantic deviation between the encoded text features and the predefined set of text semantic features. Obtain a continuous abnormality score that reflects the degree of abnormality. ; Calculate the cosine distance for each sample to obtain the weighted semantic deviation. The calculation formula is as follows: ; Weighted aggregation of the distances to all text semantic prototypes yields sample-level semantic anomaly scores. ; ; in, Represents the weight matrix. This represents a very small constant.

[0016] Compared with the prior art, the present invention has the following advantages: (1) This invention introduces a general text library generated by a large language model, avoiding repeated calls to the large language model to generate text descriptions for each frame of image, thus greatly reducing the model training cost. At the same time, the method of using the text library for training makes full use of the high-level semantic information in the video frames, which significantly improves the understanding and expression of the essential meaning of abnormal events compared with the traditional method that only relies on visual features.

[0017] (2) This invention reduces the dependence on the distribution of visual features in a single scene by cross-modal semantic constraints. It can maintain stable detection performance in diverse scenarios such as different monitoring environments, shooting angles, and personnel densities, and has better generalization ability for unseen anomaly types.

[0018] (3) The present invention adopts an anomaly determination mechanism of text reconstruction rather than image reconstruction, and avoids the model from over-reconstructing abnormal samples by similarity weighting, which effectively reduces the false negative rate of abnormal video frames and improves the detection accuracy.

[0019] (4) This invention uses a pre-trained cross-modal large model as a feature extractor, which avoids the need to train complex deep models from scratch, reduces the dependence on large-scale labeled data and simplifies the model training process, and lowers the technical threshold and cost of actual deployment. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the overall process of the multi-scenario surveillance video anomaly detection method based on a cross-modal large model in this embodiment of the invention.

[0022] Figure 2 This is a schematic diagram of the cross-modal feature encoding and fusion module based on CLIP in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the module structure of the improved Transformer in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] like Figure 1-3 As shown, this invention provides a multi-scene surveillance video anomaly detection method based on a cross-modal large model. It utilizes cross-modal semantic priors to guide model learning and reduces model complexity and improves model generalization ability through a public text library and image-text feature alignment design. The method includes the following steps: Step 1: Use a large language model to generate normal and abnormal descriptions for the scene content of the surveillance video, and generate a public text library. Label normal video frames as normal and abnormal video frames as abnormal.

[0027] Step 2: After obtaining the video frames and text database, the pre-trained CLIP model is used to obtain image feature vectors and text feature vectors respectively. Then, the cosine similarity between the image feature vectors and text feature vectors is calculated to construct an initial similarity matrix. Step 2 includes the following steps: Step 21: The CLIP image encoder divides the input video frame into fixed-size image blocks, extracts visual features through a multi-layer Transformer encoder, and then passes the linear projection layer... Mapping to the multimodal embedding space and performing L2 normalization yields the image feature vector. Meanwhile, the CLIP text encoder extracts text features through a multi-layer Transformer encoder and then through a linear projection layer. Mapping to the same multimodal embedding space and normalizing it yields the text feature vector. ; Step 22: Construct an initial similarity matrix reflecting the cross-modal semantic association strength between video frames and text descriptions by calculating the cosine similarity between image feature vectors and text feature vectors. .

[0028] Step 3: Perform lightweight adaptive correction on the image feature vector and text feature vector respectively; the cross-modal feature adaptation module, Dual Adapter, is used to perform symmetrical lightweight fine-tuning of visual and text features to achieve alignment across modal spaces while avoiding large-scale updates to the pre-trained model parameters. To balance cross-modal shared semantics and modality-specific features, this invention adopts a semi-shared hybrid adaptation structure, including a shared adapter, a visual-only adapter, and a text-only adapter. Each part has the same structure, and the formulas are as follows:

[0029] in As input features, This is the module weight matrix. These are learnable parameters used to control the adaptation strength. Let... For shared adapters, For visual adapters, As a text adapter, the final module output is as follows: ; .

[0030] The lightweight formulas for the image feature vector and text feature vector in step 3 are as follows: ; in, and These represent the weight matrix and the offset, respectively.

[0031] Step 4: To capture the temporal dependencies of the video sequence, a simplified Transformer encoder is used to model the temporal features of the image feature vectors processed in Step 3. Step 4 includes the following steps: Step 41: Generate position codes using sine / cosine functions. The calculation formula is: ; ; in, Indicates the position index. An index representing half the encoding dimension. Indicates the hidden layer dimension; Step 42: Add the position code to the image features extracted by CLIP to obtain: This allows each frame feature to carry its temporal location information; Step 43: Map the input features to the query matrix through a linear transformation. Key matrix Sum matrix ; Step 44: Calculate the self-attention weights for each attention head. The formula is: ; in, , and These represent the query, key, and value vectors obtained from the linear transformation, respectively. This represents the dimension of the key vector. Indicates the scaling factor; Step 45: Focus all attention on the output of the head. The multi-head attention output is obtained by concatenating the data and performing a linear transformation on the output weight matrix. The formula is: ; Step 46: The multi-head attention output is then fed into the feedforward neural network (FFN) for nonlinear transformation to obtain the feedforward neural network result. The calculation formula is: ; in, , Represents the weight matrix. , Indicates the bias term. Indicates the activation function; Step 47: Add residual connections and layer normalization operations after the multi-head attention layer and the feedforward network layer, respectively, to obtain the normalization result. The formula is: ; Where x is the multi-head attention output and the result of the feedforward neural network; Step 48: Normalize the results Perform a linear mapping to obtain video features i containing temporal information. trans The formula is expressed as: .

[0032] Step 5: Reconstruct the text using the simplified Transformer encoder described above; Step 6: To ensure that the temporal modeling process does not deviate from the cross-modal semantic alignment space established by CLIP, cross-modal semantic constraints are introduced; Step 6 includes the following steps: Step 61: Calculate the similarity matrix between the temporal features extracted by the Transformer and the text features. ; Step 62: Calculate the initial similarity matrix using CLIP. As a supervisory signal, the cross-entropy loss between the two similarity matrices is calculated. .

[0033] Step 7: Calculate anomaly scores for both image feature vectors and text feature vectors, and then integrate them into the final score. The anomaly score calculation for the image feature vectors is as follows: for each sample, select from the normal similarity set... The maximum value Then, the average of the selected similarities is taken to obtain the normal semantic similarity score of the sample. The formula is as follows: ; ; in, express The similarity matrix between the sample and the normal semantic prototype. Representing normal semantics parameter; For the set of anomalous semantic similarities, hard negative samples are used to obtain the anomalous semantic similarity score. The formula is as follows:

[0034] in, express Similarity matrix between the medium sample and the abnormal semantic prototype.

[0035] Define sample-level anomaly scoring as The calculation formula is as follows: .

[0036] The anomaly score calculation process for the text feature vector is as follows: it measures the weighted semantic deviation between the encoded text features and the predefined set of text semantic features. Obtain a continuous abnormality score that reflects the degree of abnormality. ; Calculate the cosine distance for each sample to obtain the weighted semantic deviation. The calculation formula is as follows: ; Weighted aggregation of the distances to all text semantic prototypes yields sample-level semantic anomaly scores. ; ; in, Represents the weight matrix. This represents a very small constant.

[0037] Example 1 like Figure 1 The diagram shown is the overall flowchart of the method. This embodiment provides a complete video anomaly detection process, mainly including a general text library generation module, a cross-modal feature extraction module, a cross-modal feature lightweight correction module, a temporal modeling module, a text reconstruction module, and an anomaly score calculation module.

[0038] In the general text library generation stage: First, a large language model (e.g., Chat GPT) is used to generate normal and abnormal descriptions of the scene content in the surveillance video. To ensure the pre-trained CLIP encoder is not affected by redundant semantics and to achieve the best text encoding results, a prompt is designed to standardize the text content generated by the large model, achieving uniform formatting and concise, clear semantics. The large language model is then used to label the generated text library, setting labels and abnormal subspace types. Normal video frames are labeled as "normal," while abnormal video frame descriptions, in addition to abnormal labeling, also need to classify the abnormality type, such as abnormal behavior, abnormal objects, or abnormal interactions.

[0039] The modules include cross-modal feature extraction, lightweight cross-modal feature correction, temporal modeling, text reconstruction, and anomaly scoring calculation. Figure 2 As shown.

[0040] Cross-modal feature extraction stage: After obtaining the video frames and text database, a pre-trained CLIP model is used for cross-modal feature encoding. The CLIP image encoder adopts a Vision Transformer (ViT) architecture, which segments the input video frames into fixed-size image blocks and extracts visual features through a multi-layer Transformer encoder. The encoded image features are then processed through a linear projection layer. Mapping to the multimodal embedding space and performing L2 normalization yields the image feature vector. Meanwhile, the CLIP text encoder encodes the generated text description based on the Transformer architecture, and the encoded text features are then passed through a linear projection layer. Mapping to the same multimodal embedding space and normalizing it yields the text feature vector. An initial similarity matrix is ​​constructed by calculating the cosine similarity between image features and text features. This similarity matrix reflects the strength of cross-modal semantic association between video frames and text descriptions.

[0041] Cross-modal feature lightweight correction stage: After acquiring the initial features of the image and text, the initial features are lightweighted and adaptively corrected. This module includes a visual feature correction submodule and a cross-modal adaptation submodule, achieving rapid adaptation and semantic alignment for downstream tasks (such as anomaly detection) without destroying the original feature space structure. VisualCorrection is used to normalize and correct the linear / nonlinear mapping of the input visual features to alleviate the distribution offset problem of the pre-trained model in the target scene and improve the stability and transferability of the features. The formula is as follows:

[0042] in and These are the weight matrix and offset of the module, respectively.

[0043] The Dual Adapter module, a cross-modal feature adaptation module, performs symmetrical, lightweight fine-tuning of visual and textual features to achieve cross-modal alignment while avoiding large-scale updates to pre-trained model parameters. To balance cross-modal semantic sharing with modality-specific features, this invention employs a semi-shared hybrid adaptation structure, including a SharedAdapter, a Visual-only Adapter, and a Text-only Adapter. Each part has the same structure, and the formulas are as follows:

[0044] in As input features, This is the module weight matrix. These are learnable parameters used to control the adaptation strength. Let... For shared adapters, For visual adapters, As a text adapter, the final module output is as follows:

[0045]

[0046] Temporal feature modeling stage: such as Figure 3 The diagram shows the modified Transformer encoder structure. To capture the temporal dependencies of video sequences, this invention specifically adapts the Transformer model, removing the decoder and retaining only the encoder structure for temporal feature extraction. The specific process is as follows: Since the self-attention mechanism itself is permutationally equivariant and cannot distinguish sequence order, absolute positional information is first injected into the features of each video frame through a position encoder. A sine / cosine function is used to generate the positional code, calculated using the following formula:

[0047]

[0048] in For location index, It is an index that is half the encoding dimension. This is the hidden layer dimension of the model. It is obtained by adding the positional encoding to the image features extracted by CLIP. This allows each frame feature to carry its temporal location information.

[0049] like Figure 3 As shown, the encoded feature sequence enters a multi-layer attention mechanism module for processing. In each layer, the input features are first mapped to a query matrix through a linear transformation. Key matrix Sum matrix Then, for each attention head h, the self-attention weight is calculated using the following formula:

[0050] in, , and These are the query, key, and value vectors obtained through linear transformation. It is the dimension of the key vector. It's the scaling factor. Divide by As a scaling factor, it prevents the gradient of the softmax function from vanishing due to excessively large dot product values. The multi-head attention mechanism captures diverse dependencies between different positions in the input sequence using H parallel attention heads. The outputs of all attention heads are concatenated and linearly transformed using the output weight matrix to obtain:

[0051] The output of the multi-head attention is then fed into a feedforward neural network (FFN) for nonlinear transformation, calculated as follows:

[0052] in , This is the weight matrix. , For bias terms, The activation function is used. Residual connections and layer normalization operations are added after each sub-layer (multi-head attention layer and feedforward network layer), as shown in the formula:

[0053] This structural design contributes to gradient propagation and model training stability. For example... Figure 3 As shown, by stacking N Transformer encoder layers, temporal dependency features of the video sequence are extracted and fused step by step, and the final video feature representation containing temporal information is as follows:

[0054] Text Reconstruction Stage: The text reconstruction module of this invention also utilizes the Transformer structure described above. To prevent different text orders from affecting the semantic features of the text, a priori order is introduced. The text reconstruction module removes positional encoding, and the reconstructed text features are as follows:

[0055] Cross-modal semantic constraint training: To ensure that the temporal modeling process does not deviate from the cross-modal semantic alignment space established by CLIP, this invention designs a cross-modal semantic constraint loss. First, the similarity matrix between the temporal features extracted by the Transformer and the text features is calculated. Then, the initial similarity matrix calculated by CLIP is used... As a supervisory signal, the cross-entropy loss between the two similarity matrices is calculated using the following formula:

[0056]

[0057] This loss function calculates the weighted cross-entropy over all matrix elements, enabling the Transformer module to maintain consistency with the CLIP cross-modal semantic space during the extraction of temporal features. This enhances the ability to model video temporal dependencies and utilizes the rich semantic prior knowledge contained in the CLIP pre-trained model.

[0058] Anomaly Score Calculation Stage: This invention employs two anomaly scoring methods. The Score1 module is used to construct a sample-level anomaly score based on the comparison between the normal semantic similarity set and the anomaly semantic similarity set. First, for each sample, a sample is selected from the normal similarity set... The maximum value is calculated, and then the average of the selected similarities is taken to obtain the normal semantic similarity score of the sample.

[0059]

[0060] in for The similarity matrix between the sample and the normal semantic prototype. Normal semantics Parameters. For the abnormal semantic similarity set, hard negative samples are used, as shown in the following formula:

[0061] in for The similarity matrix between the sample and the abnormal semantic prototype. The final Score1 is based on the difference between positive and negative semantic similarity; a larger value indicates that the sample is closer to abnormal semantics and farther from normal semantics, and vice versa. The sample-level anomaly score is defined as:

[0062] The Score2 module is used to calculate sample-level anomaly scores based on the text semantic space. It measures the weighted semantic deviation between the encoded text features and a predefined set of text semantic features to obtain a continuous anomaly score reflecting the degree of anomaly. To ensure the numerical stability of the similarity calculation, the features are first... Normalization: ,

[0063] Calculate the cosine distance for each sample:

[0064] We perform weighted aggregation of the distances to all text semantic prototypes to obtain sample-level semantic anomaly scores: ; in, This is the weight matrix. This is a very small constant used to avoid numerical instability. This score reflects the overall deviation of a sample from the semantic prototype set in the text semantic space, and can be directly used for anomaly detection decisions or fused with other modality anomaly scores.

[0065] Example 2 This embodiment illustrates how to quickly adapt a trained model to a new monitoring scenario. For a new monitoring scenario, a small number of normal video samples in that scenario are first collected as reference data; typically, 100-200 video clips are sufficient. A large language model is used to generate text descriptions for the keyframes of these normal samples, constructing a scenario-specific normal text feature library. Because this invention is based on the cross-modal alignment capability and zero-shot generalization characteristics of the CLIP pre-trained model, there is no need to retrain or fine-tune the entire model; only updating the text feature library is required to achieve scenario transfer. The model can leverage the general semantic understanding capabilities obtained by CLIP pre-training on massive image-text pairs to identify novel abnormal patterns not found in the training data, thereby maintaining stable detection performance and strong generalization ability in different monitoring environments such as office buildings, shopping malls, subway stations, and parking lots.

[0066] Example 3 System hardware configuration and performance optimization This embodiment provides recommended system configurations and performance optimization schemes. For hardware configuration, an NVIDIA RTX 3090 or A4000 GPU with performance of 24GB or more of dedicated video memory is recommended, capable of processing high-definition video streams at a resolution of 1920×1080 and a frame rate of 30fps in real time. To further improve inference efficiency and reduce deployment costs, INT8 quantization compression can be performed on the Transformer encoder and MLP decoder. The quantized model can improve inference speed by approximately 1.8-2.2 times while maintaining anomaly detection accuracy loss of less than 2%. At the data processing level, 16 consecutive frames in the video sequence are processed in parallel as a batch, fully leveraging the parallel computing advantages of the GPU. For monitoring scenarios with fixed cameras, CLIP encoding results of normal text feature libraries can be pre-calculated and cached offline to avoid repetitive encoding calculations during online inference, further reducing system latency. Through these optimization measures, the system can simultaneously process 4-6 channels of high-definition video streams for real-time anomaly detection on a single GPU.

[0067] This invention effectively addresses the technical problems of traditional video anomaly detection methods, such as over-reliance on single visual features and poor generalization ability, by introducing cross-modal semantic constraints and text feature reconstruction mechanisms. Utilizing the powerful cross-modal alignment capability and zero-shot generalization characteristics of the CLIP pre-trained model, combined with the temporal modeling capabilities of the adapted Transformer and the anomaly scoring mechanism based on text feature reconstruction, high-precision anomaly detection is achieved in multi-scene monitoring environments. The above embodiments are merely preferred implementations of this invention and are not intended to limit the scope of protection of this invention. Conventional changes and substitutions made by those skilled in the art based on the technical solutions of this invention should be included within the scope of protection of this invention.

[0068] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for anomaly detection in multi-scene surveillance videos based on a cross-modal large model, which utilizes cross-modal semantic priors to guide model learning, and reduces model complexity and improves model generalization ability through a public text library and image-text feature alignment design, characterized in that... Includes the following steps: Step 1: Use a large language model to generate normal and abnormal descriptions of the scene content in the surveillance video, and generate a public text library; Step 2: After obtaining the video frames and text database, the pre-trained CLIP model is used to obtain image feature vectors and text feature vectors respectively. Then, the cosine similarity between the image feature vectors and text feature vectors is calculated to construct an initial similarity matrix. ; Step 3: Perform lightweight adaptive correction on the image feature vector and the text feature vector respectively; Step 4: To capture the temporal dependencies of the video sequence, a simplified Transformer encoder is used to model the temporal features of the image feature vectors processed in Step 3. Step 5: Reconstruct the text using the simplified Transformer encoder described above; Step 6: To ensure that the temporal modeling process does not deviate from the cross-modal semantic alignment space established by CLIP, cross-modal semantic constraints are introduced; Step 7: Calculate anomaly scores for image feature vectors and text feature vectors respectively, and integrate the two into the final score.

2. The method for detecting anomalies in multi-scene surveillance videos based on a cross-modal large model according to claim 1, wherein in step 1, normal video frames are labeled as normal and abnormal video frames are labeled as abnormal.

3. The method for anomaly detection in multi-scene surveillance video based on a cross-modal large model according to claim 1, wherein step 2 includes the following steps: Step 21: The CLIP image encoder divides the input video frame into fixed-size image blocks, extracts visual features through a multi-layer Transformer encoder, and then passes the linear projection layer... Mapping to the multimodal embedding space and performing L2 normalization yields the image feature vector. Meanwhile, the CLIP text encoder extracts text features through a multi-layer Transformer encoder and then through a linear projection layer. Mapping to the same multimodal embedding space and normalizing it yields the text feature vector. ; Step 22: Construct an initial similarity matrix reflecting the cross-modal semantic association strength between video frames and text descriptions by calculating the cosine similarity between image feature vectors and text feature vectors. .

4. The multi-scene surveillance video anomaly detection method based on a cross-modal large model according to claim 1, wherein the lightweight formulas for the image feature vector and text feature vector in step 3 are as follows: ； in, and These represent the weight matrix and the offset, respectively.

5. The method for anomaly detection in multi-scene surveillance video based on a cross-modal large model according to claim 1, wherein step 4 includes the following steps: Step 41: Generate position codes using sine / cosine functions. The calculation formula is: ； ； in, Indicates the position index. An index representing half the encoding dimension. Indicates the hidden layer dimension; Step 42: Add the position code to the image features extracted by CLIP to obtain: This allows each frame feature to carry its temporal location information; Step 43: Map the input features to the query matrix through a linear transformation. Key matrix Sum matrix ; Step 44: Calculate the self-attention weights for each attention head. The formula is: ； in, , and These represent the query, key, and value vectors obtained from the linear transformation, respectively. This represents the dimension of the key vector. Indicates the scaling factor; Step 45: Focus all attention on the output of the head. The multi-head attention output is obtained by concatenating the data and performing a linear transformation on the output weight matrix. The formula is: ； Step 46: The multi-head attention output is then fed into the feedforward neural network (FFN) for nonlinear transformation to obtain the feedforward neural network result. The calculation formula is: ； in, , Represents the weight matrix. , Indicates the bias term. Indicates the activation function; Step 47: Add residual connections and layer normalization operations after the multi-head attention layer and the feedforward network layer, respectively, to obtain the normalization result. The formula is: ； Where x is the multi-head attention output and the result of the feedforward neural network; Step 48: Normalize the results Perform a linear mapping to obtain video features i containing temporal information. trans The formula is expressed as: 。 6. The method for anomaly detection in multi-scene surveillance video based on a cross-modal large model according to claim 1, wherein step 6 includes the following steps: Step 61: Calculate the similarity matrix between the temporal features extracted by the Transformer and the text features. ; Step 62: Calculate the initial similarity matrix using CLIP. As a supervisory signal, the cross-entropy loss between the two similarity matrices is calculated. .

7. The multi-scene surveillance video anomaly detection method based on a cross-modal large model according to claim 1, wherein the anomaly score of the image feature vector is calculated as follows: for each sample, select from the normal similarity set... The maximum value Then, the average of the selected similarities is taken to obtain the normal semantic similarity score of the sample. The formula is as follows: ； ； in, express The similarity matrix between the sample and the normal semantic prototype. Representing normal semantics parameter; For the set of anomalous semantic similarities, hard negative samples are used to obtain the anomalous semantic similarity score. The formula is as follows: in, express Similarity matrix between the medium sample and the abnormal semantic prototype. Define sample-level anomaly scoring as The calculation formula is as follows: 。 8. The method for anomaly detection in multi-scene surveillance video based on a cross-modal large model according to claim 1, characterized in that, The anomaly score calculation process for the text feature vector is as follows: it measures the weighted semantic deviation between the encoded text features and the predefined set of text semantic features. Obtain a continuous abnormality score that reflects the degree of abnormality. ; Calculate the cosine distance for each sample to obtain the weighted semantic deviation. The calculation formula is as follows: ； Weighted aggregation of the distances to all text semantic prototypes yields sample-level semantic anomaly scores. ; ； in, Represents the weight matrix. This represents a very small constant.

Images