A Video Anomaly Detection Method and Device Based on Domain Knowledge-Guided Large Model

Abstract

This application relates to a method and apparatus for video anomaly detection based on a domain knowledge-guided large-scale model. The method includes: constructing a set of guiding questions; obtaining a final set of questions through multiple rounds of iterative optimization using an employee and a leading model; converting these questions into affirmative statement phrases; embedding the phrases and the video to be detected using a CLIP encoder to obtain text and visual representation sets; calculating the semantic and context-related scores for each frame; sampling keyframes; constructing a set containing multiple sub-questions for each sampled frame; inputting the set into a pre-trained multimodal large-scale model to obtain an inference answer; generating an anomaly score; completing the anomaly score of the video to be detected through weighted interpolation; and combining the inference answer to achieve video anomaly detection based on a domain knowledge-guided large-scale model. This method achieves low data dependency, high efficiency, strong interpretability, and comprehensive video anomaly detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information computing, and in particular, to a video anomaly detection method and device based on domain knowledge guided large model. BACKGROUND

[0002] Video anomaly detection (VAD) aims to automatically identify abnormal behaviors or events in video sequences, and is a key component in specific application scenarios such as autonomous driving, campus security monitoring, and industrial quality detection. By detecting situations that deviate from normal operating states early, VAD systems help prevent accidents, ensure public safety, and maintain the stability of industrial processes. As these systems are increasingly deployed in complex real-world environments, the demand for accurate, reliable, and interpretable anomaly detection has grown significantly. However, due to the high safety requirements of such scenarios, abnormal situations are usually rare and difficult to collect in large quantities.

[0003] Recent advances in visual language models (VLMs) and multi-modal large language models (MLLMs) provide a promising approach to interpretable video anomaly detection by enabling joint visual understanding and natural language reasoning. Two main paradigms have emerged: (i) zero-shot methods without training, such as LAVAD, which utilize pre-trained models for direct inference without fine-tuning, offering instant deployability and inherent interpretability, but are limited by poor domain specialization; and (ii) instruction-finetuning methods like Holmes-VAD and VADor, which achieve stronger domain alignment by fine-tuning on large annotated datasets, but at the cost of high annotation burden, making them impractical in data-sensitive or resource-limited scenarios.

[0004] Despite these advances, two fundamental challenges remain unresolved. First, due to the high safety requirements of real-world applications, abnormal events are inherently rare and difficult to collect in large quantities, which severely limits the feasibility of supervised or fine-tuning-based methods. Second, anomalies are often highly context-dependent, exhibiting unique patterns that vary greatly across different domains, posing significant challenges to generalization and deep semantic understanding in specific scenarios. SUMMARY

[0005] Therefore, it is necessary to provide a video anomaly detection method and device based on domain knowledge guided large model, which can generalize to specific scenarios while improving the accuracy of anomaly detection.

[0006] A video anomaly detection method based on domain knowledge guided large model, the method comprising:

[0007] The process involves acquiring the set of videos to be detected, a small subset of sample videos from the training phase, domain knowledge documents to provide anomaly categories and causal patterns, and real label sequences with fine-grained labels for the corresponding video frames.

[0008] A set of guiding questions is constructed, and the employee model and the dominant model perform multiple rounds of iterative optimization on the set of guiding questions based on the small subset of sample videos, the domain knowledge documents, and the real label sequence to obtain the final set of guiding questions.

[0009] The final set of guiding questions is transformed into a set of affirmative statement phrases, and the set of affirmative statement phrases and the set of videos to be detected are embedded respectively. Based on the embedded text representation set and visual representation set, the semantic and context-related scores of each frame of the image to be detected in the set of videos to be detected are calculated. Then, keyframe sampling is performed from the set of videos to be detected to obtain a set of sampled frames.

[0010] Based on the final set of guiding questions, a question set including multiple sub-questions is constructed for each sampled frame image in the sampled frame set. The sampled frame images and the corresponding question set are input into a pre-trained multimodal large model to obtain the reasoning answer for each question in the question set, and an anomaly score is generated based on the reasoning answer.

[0011] The complete anomaly score of the video set to be detected is obtained based on the anomaly score of the sampled frame images in the sampled frame set. Video anomaly detection is achieved based on the complete anomaly score and the inference answer of each sampled frame image in the sampled frame set.

[0012] In one embodiment, when updating the set of guiding questions:

[0013] Using the employee model, anomaly detection is performed on each frame of the small sample video subset based on the currently updated set of guiding questions to obtain a binary prediction result;

[0014] Using the dominant model, frames with prediction errors are filtered out based on the binary prediction results and the true label sequence;

[0015] By integrating the guidance question set, binary prediction results, prediction error frames, and corresponding ground truth labels, along with domain knowledge documents, the reasons for prediction failure are analyzed, and knowledge dimensions in the domain knowledge documents that are not reflected in the guidance question set are identified, and the guidance question set is updated accordingly.

[0016] After multiple rounds of iterative optimization of the employee model and the dominant model, the final set of guiding questions is obtained.

[0017] In one embodiment, the guiding question set is a set of learnable natural language parameters, which are updated under the guidance of the domain knowledge document.

[0018] In one embodiment, when calculating the semantic and context-related scores of each frame of the image to be detected in the video set based on the embedded text representation set and the visual representation set:

[0019] The visual representation set includes the visual features of each frame of the image to be detected in the video set to be detected, and the text representation set includes the text features of each guiding question phrase in the guiding question set;

[0020] For each frame of the image to be detected, the maximum cosine similarity between the corresponding visual features and all text features in the text representation set is calculated as the semantic relevance score;

[0021] For each frame of the image to be detected, the similarity between the image and the two adjacent frames of the image to be detected is calculated based on the visual features and used as the context-related score.

[0022] In one embodiment, based on the semantic relevance score and context relevance score of the image to be detected, a set of sampled frames is obtained by sampling keyframes from the set of videos to be detected, including:

[0023] Based on the semantic relevance score and the context relevance score, calculate the overall score for each frame of the image to be detected;

[0024] A cumulative distribution function is calculated based on the total score of all images to be detected, and a predetermined number of points are uniformly sampled along the cumulative axis based on the cumulative distribution function.

[0025] The sampled points are mapped back to the original time axis to obtain multiple frames of images to be detected, which are the sampled frames.

[0026] In one embodiment, when constructing a question set including multiple sub-questions for each sampled frame image in the sampled frame set based on the final guiding question set:

[0027] In the final set of guiding questions, the most relevant guiding questions are selected for each sampled frame image;

[0028] Each selected guiding question is further divided into multiple sub-questions, and a question set corresponding to the sampled frame image is constructed based on the multiple sub-questions obtained from the division.

[0029] In one embodiment, the three most relevant guiding questions are selected for each sampled frame image in the final set of guiding questions.

[0030] In one embodiment, generating anomaly scores for the sampled frame images based on the inference answer includes:

[0031] An anomaly score is calculated using a scoring prompt template based on the set of questions in the sampled frame image, the reasoned answer to each question in the corresponding set of questions, and the sampled frame image itself.

[0032] In one embodiment, when obtaining the complete anomaly score of the video set to be detected based on the anomaly scores of the sampled frame images in the sampled frame set, a weighted interpolation completion method is used, specifically including:

[0033] For a given non-sampled frame image, calculate the context similarity between the non-sampled frame image and each sampled frame image;

[0034] The weights between the non-sampled frame images and each sampled frame image are determined based on the context similarity.

[0035] The anomaly score of the unsampled frame image is calculated based on the contextual similarity and weight between the unsampled frame image and each sampled frame image.

[0036] The complete anomaly score of the video set to be detected is obtained based on the anomaly scores of all unsampled frame images and all sampled frame images.

[0037] This application also provides a video anomaly detection device based on a domain knowledge-guided large model, the device comprising:

[0038] The data acquisition module is used to acquire the set of videos to be detected, a small subset of sample videos from the training phase, domain knowledge documents to provide anomaly categories and causal patterns, and real label sequences with fine-grained labels for corresponding video frames.

[0039] The final guidance question set acquisition module is used to construct the guidance question set. The employee model and the dominant model perform multiple rounds of iterative optimization on the guidance question set based on the small subset of sample videos, the domain knowledge documents, and the real label sequence to obtain the final guidance question set.

[0040] The keyframe sampling module is used to transform the final set of guiding questions into a set of affirmative statement phrases, and embed the set of affirmative statement phrases and the set of videos to be detected respectively. Based on the embedding, the text representation set and visual representation set are obtained, and the semantic and context-related scores of each frame of the image to be detected in the set of videos to be detected are calculated. Then, keyframe sampling is performed from the set of videos to be detected to obtain a set of sampled frames.

[0041] The anomaly score calculation module is used to construct a question set including multiple sub-questions for each sampled frame image in the sampled frame set based on the final guiding question set, input the sampled frame images and the corresponding question set into a pre-trained multimodal large model, obtain the reasoning answer for each question in the question set, and generate an anomaly score based on the reasoning answer;

[0042] The video anomaly detection module is used to obtain the complete anomaly score of the video set to be detected based on the anomaly score of the sampled frame images in the sampled frame set, and to realize video anomaly detection based on the complete anomaly score and the reasoning answer of each sampled frame image in the sampled frame set.

[0043] The aforementioned video anomaly detection method and apparatus based on a domain knowledge-guided large model constructs a set of guiding questions. The employee model and the dominant model iteratively optimize this set multiple times using a small subset of sample videos from the training phase, domain knowledge documents providing anomaly categories and causal patterns, and the real label sequences of corresponding video frame fine-grained labels. This results in the final set of guiding questions, which is then transformed into a set of affirmative statement phrases. CLIP is then used to perform the final set of guiding questions. The encoder embeds a set of affirmative statement phrases and a set of videos to be detected, respectively, to obtain a text representation set and a visual representation set. Based on the text representation set and the visual representation set, the semantic relevance score and context relevance score of each frame of the video to be detected are calculated. Then, keyframe sampling is performed from the video to be detected to obtain a set of sampled frames. According to the final set of guiding questions, a question set including multiple sub-questions is constructed for each sampled frame image in the sampled frame set. The sampled frame images and the corresponding question set are then input into a pre-trained multimodal large model to obtain the reasoning answer for each question in the question set. Anomaly score of the sampled frame image is generated based on the reasoning answer. Based on the anomaly score of the sampled frame images in the sampled frame set, the complete anomaly score of the video to be detected is obtained through weighted interpolation. Based on the complete anomaly score and the reasoning answer of each sampled frame image in the sampled frame set, video anomaly detection based on domain knowledge guided by the large model is realized. This method combines few-sample videos and domain knowledge to iteratively optimize the guided questions, uses CLIP to accurately sample keyframes, and then uses a multimodal large model to generate anomaly scores through fine-grained inference and interpolation to complete the full-frame scores, achieving video anomaly detection with low data dependence, high efficiency, strong interpretability, and complete coverage. Attached Figure Description

[0044] Figure 1 This is an application environment diagram of a video anomaly detection method based on a domain knowledge-guided large model in one embodiment.

[0045] Figure 2 This is a schematic diagram of the framework proposed in this method in one embodiment;

[0046] Figure 3 For an experiment, different sampling methods were used on the UT dataset. Comparison diagram;

[0047] Figure 4 This is a schematic diagram comparing AP under different sampling methods on the UT dataset in an experiment;

[0048] Figure 5 This is a structural block diagram of a video anomaly detection device based on a domain knowledge-guided large model in one embodiment.

[0049] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In view of the shortcomings of the existing technology, this application, such as Figure 1 As shown, a video anomaly detection method based on a domain knowledge-guided large model is provided, which specifically includes the following steps:

[0052] Step S100: Obtain the set of videos to be detected, a small subset of sample videos from the training phase, a domain knowledge document to provide anomaly categories and causal patterns, and the real label sequence of fine-grained labels for the corresponding video frames.

[0053] Step S110: Construct a guiding question set. The employee model and the dominant model perform multiple rounds of iterative optimization on the guiding question set based on a small subset of sample videos, the domain knowledge documents, and the real label sequences to obtain the final guiding question set.

[0054] Step S120: The final set of guiding questions is transformed into a set of affirmative statement phrases. The set of affirmative statement phrases and the set of videos to be detected are embedded respectively. Based on the embedded text representation set and visual representation set, the semantic and context-related scores of each frame of the image to be detected in the set of videos to be detected are calculated. Then, keyframe sampling is performed from the set of videos to be detected to obtain a set of sampled frames.

[0055] Step S130: Based on the final set of guiding questions, construct a question set including multiple sub-questions for each sampled frame image in the sampled frame set. Input the sampled frame images and the corresponding question set into the pre-trained multimodal large model to obtain the reasoning answer for each question in the question set, and generate an anomaly score based on the reasoning answer.

[0056] Step S140: Obtain the complete anomaly score of the video set to be detected based on the anomaly scores of the sampled frame images in the sampled frame set, and realize video anomaly detection based on the complete anomaly score and the reasoning answer of each sampled frame image in the sampled frame set.

[0057] This application proposes a domain knowledge-guided, interpretable video anomaly detection method based on a multimodal large language model, comprising: a domain knowledge-guided question learning part (step S110), an adaptive anomaly-aware frame sampling part (step S120), and fine-grained anomaly reasoning and detection parts (steps S130 and S140), the process of which is as follows: Figure 2 As shown in the diagram. The domain knowledge-guided question learning component employs a linguistic learning paradigm, leveraging the iterative interaction between the employee model (which uses the current question set to detect a small number of video frames) and the dominant model (which combines domain knowledge to optimize the question set corresponding to error cases) to generate the final guiding question set with the highest verification accuracy. The adaptive anomaly-aware frame sampling section will... The text is converted into semantic phrases, and after passing through the CLIP encoder to obtain text and visual embeddings, semantic and context relevance scores are calculated. Then, a set of sampling frames with abnormally high relevance is generated through density-aware sampling. Finally, the fine-grained anomaly inference and detection part is used for each... The relevant questions are screened and decomposed into sub-questions. The employee model (i.e., the multimodal large model in step S130) generates inference results and anomaly scores for sampled frames. Then, weighted interpolation (combining temporal distance, contextual similarity, and other factors to calculate weights) expands the sampled frame scores to all video frames, ultimately yielding anomaly scores for all frames. These three modules, logically connected by domain knowledge-guided questioning, intelligent sampling focusing on key frames, and fine-grained inference detection, enable efficient and interpretable video anomaly detection.

[0058] In step S100, the video set to be detected is a set containing Frame video , represented as The goal is to achieve this by doing so for each frame. ( Predict an outlier score To test Abnormal events in the training phase. And a small subset of sample videos... The real label sequence corresponding to the fine-grained labels of the video frames. ,in Domain knowledge documents It includes anomaly categories and causal patterns related to the content of the video set to be detected. Specifically, during the training phase, based on a dual-agent framework, the guidance question is iteratively updated, and domain knowledge text is incorporated to improve the guidance effect. Not part of the video set to be tested However, these are videos belonging to the same field.

[0059] In this embodiment, in the domain knowledge-guided question learning section, domain knowledge documents are used. And some guiding questions are generated from some videos selected from the training set. Use these guiding questions in the anomaly reasoning and detection module. To detect anomalies in the test set videos.

[0060] In step S110, when updating the guidance question set: using the employee model, anomaly detection is performed on each frame of the small sample video subset based on the updated guidance question set to obtain binary prediction results. Then, using the dominant model, frames with prediction errors are filtered out based on the binary prediction results and the true label sequence. By integrating the guidance question set, binary prediction results, frames with prediction errors, corresponding true labels, and domain knowledge documents, the reasons for prediction failure are analyzed, and knowledge dimensions in the domain knowledge documents that are not reflected in the guidance question set are identified, thus updating the guidance question set. After multiple rounds of iterative optimization by the employee model and the dominant model, the final guidance question set is obtained.

[0061] In this embodiment, an initial set of bootstrapping questions is constructed. The initial natural language parameters for subsequent iterations are based on general anomaly queries. Then, the core of the dual-agent iterative optimization process begins, with each iteration starting with the employee model. from Medium-sampled frame subset And in conjunction with the current round of guiding questions Perform anomaly detection and output binary classification results. (Where 1 represents abnormal and 0 represents normal), then the dominant model Filter out One classification error frame, integrated , Error frame true label Error sampling frame and Analyze and predict the reasons for failure and identify China not in The knowledge dimensions reflected will then generate the next round of optimization problem sets. Through a cycle of multiple rounds of employee model evaluation and leading model optimization, the process gradually allows... Align with domain knowledge and visual context. Finally, after all iterations are completed, compare the results of each round. Based on the detection accuracy on the validation set, the one with the highest accuracy was selected. As the final set of guiding questions This is used for subsequent anomaly reasoning and detection.

[0062] In this embodiment, the core innovation of the domain knowledge-guided question learning section lies in the adoption of a language-based learning paradigm. In this paradigm, the model's knowledge and decision boundaries are explicitly represented and optimized in natural language. This method extracts domain knowledge into a set of guiding questions, represented as... ,in The number of questions is indicated. Multi-round language-based optimization is utilized between two agents based on a multimodal large language model (MLLM): an employee model and a leader model. Language-based learning does not simply rely on gradient-based parameter updates, but rather enables these agents to iteratively optimize the guided questions through natural language interactions. Q This enables scene-adaptive anomaly detection without requiring extensive manual annotation.

[0063] Specifically, the iterative problem-solving learning system consists of two multimodal large language models: one is the employee model (represented as...). ) and a dominant model (represented as The employee model is responsible for using the current set of bootstrapping questions. For a small subset of the video set (represented as...) Anomaly detection is performed. Then, the dominant model uses the detection results from the employee model and the domain knowledge document. Update and optimize Within this framework, Defined as a set of learnable natural language parameters, optimized to express and detect anomalies in a specific scene. During training, a limited set of video samples provides the perceptual basis, while domain knowledge documents... This provides high-level descriptions of anomaly categories and causal patterns. Through iterative updates involving only a small subset of questions in each round, the system progressively aligns domain knowledge with the visual context. After training, the final guiding question... Select from the rounds with the highest verification accuracy.

[0064] Furthermore, through To conduct a problem assessment of few-sample videos, specifically, to evaluate the few-sample video set... The subset of small sample frames obtained by mid-sampling Perform anomaly detection. Given the current problem set. Staff refers to each frame. Output binary classification results (Where 1 represents an anomaly and 0 represents a normal state). Initial problem set. This could be a generic exception query. Staff's reasoning task can be formalized as:

[0065] ;

[0066] In the above formula, Indicates the first The optimized set of guiding questions, It is the first frame, yes based on The corresponding test results generated.

[0067] Next, by The domain guidance problem optimization was carried out, from Select the output of the current round Frames that are misclassified are identified in order to provide targeted feedback.

[0068] Specifically, The current problem set , Test results Corresponding real labels Few sampled frames Domain knowledge documents As input, it then performs a detailed analysis of misclassified cases, summarizing the reasons for prediction failures by integrating domain knowledge, and identifying... China has not yet The Leader's output is a new set of questions, reflecting the relevant aspects and proposing further detailed updates. Formalized as:

[0069] ;

[0070] In the above formula, It is the set of optimization guidance questions for the next iteration.

[0071] In step S120, intelligent selection of video frames concentrates computational resources on the most informative frames while reducing overall overhead. First, based on the final guidance question set... Process these questions by converting them into affirmative statements. (Contains l decomposed semantic phrases). Next, the CLIP encoder is used to process these semantic phrases and video frames respectively. Embedding is performed to obtain the text representation. and visual representation In terms of score calculation, the module obtains the semantic relevance score by calculating the maximum cosine similarity between each frame and all question embeddings. Simultaneously, dynamic changes are captured by measuring the similarity between adjacent frames, resulting in a context relevance score. .

[0072] In this embodiment, by calculating weighted semantic and contextual similarity scores, frames with high correlation to anomalies are prioritized. Then, density-aware sampling is applied to concentrate computational resources on the most informative video frames while reducing overall overhead.

[0073] In this embodiment, when calculating the semantic relevance score and context relevance score of each frame of the video to be detected based on the text representation set and the visual representation set: the visual representation set includes the visual features of each frame of the video to be detected, and the text representation set includes the text features of each guiding question phrase in the guiding question set. For each frame of the video to be detected, the maximum cosine similarity between the corresponding visual features and all text features in the text representation set is calculated as the semantic relevance score. Simultaneously, the similarity between the visual features and two adjacent frames of the video to be detected is also calculated as the context relevance score.

[0074] Specifically, the semantic relevance score is calculated using the following formula:

[0075] .

[0076] Specifically, the context-related score is calculated using the following formula:

[0077] .

[0078] In this embodiment, based on the semantic relevance score and context relevance score of the image to be detected, a set of sampled frames is obtained by sampling keyframes from the set of videos to be detected. This includes: calculating the comprehensive total score of each frame of the image to be detected based on the semantic relevance score and context relevance score; calculating a cumulative distribution function based on the comprehensive total score of all images to be detected; uniformly sampling a preset number of points along the cumulative axis based on the cumulative distribution function; and then mapping the sampled points back to the original time axis to obtain multiple frames of images to be detected, which are the sampled frames.

[0079] Specifically, to achieve non-uniform sampling, a density-aware adaptive sampler inspired by Holmes-UAV is introduced. First, the cumulative distribution function (CDF) over semantic and context scores is computed:

[0080] ;

[0081] In the above formula, It is a small positive constant used to prevent the gradient from being zero. It is a weighted coefficient used to balance semantic and contextual relevance, while Used to adjust the sharpness of the sampling distribution.

[0082] Furthermore, uniform sampling along the accumulation axis These points are then mapped back to the original timeline, resulting in a non-uniform frame index on the timeline. This produces the final set of sampled frames. ,in It's a video. A subset of medium frames, This specifies the number of sampling frames.

[0083] In steps S130 and S140, the model's ability to perform fine-grained anomaly analysis beyond general inference is enhanced, generating interpretable and evidence-based anomaly scores for video frames and extending them to the complete set of video frames through interpolation.

[0084] In this embodiment, when constructing a question set including multiple sub-questions for each sampled frame image in the sampled frame set based on the final guiding question set, the most relevant guiding questions are selected for each sampled frame image in the final guiding question set, and each selected guiding question is further divided into multiple sub-questions. The question set for the corresponding sampled frame image is constructed based on the multiple sub-questions obtained from the division.

[0085] Preferably, in the final set of guiding questions, the three most relevant guiding questions are selected for each sampled frame image.

[0086] Specifically, regarding videos Each sampling frame uses a frame image. The module starts from the final set of bootstrap questions. The three questions most relevant to the current frame were selected and combined to form... These problems are then decomposed into a set of fine-grained subproblems. The sub-problem will be based on the frame The specific content will be adjusted accordingly.

[0087] In this embodiment, each sub-problem As a fine-grained cue, with the sampled frame The data are input into a multimodal large model for processing to obtain the corresponding anomaly inference results. All reasoning results constitute interpretable reasoning output. .

[0088] In this embodiment, the modal model used in step S130 can be the employee model used in step S110. Its abnormal reasoning process is represented as follows:

[0089]

[0090] Since the set of questions contains multiple sub-questions, all answers for a specific captured frame image are... Constitute interpretable reasoning output .

[0091] In this embodiment, generating the anomaly score of the sampled frame image based on the reasoning answer includes: calculating the anomaly score using a scoring prompt template based on the question set of the sampled frame image, the reasoning answer for each question in the corresponding question set, and the sampled frame image itself.

[0092] Specifically, based on the reasoning context, by using a set of sub-problems Reasoning results The frame itself is input into the scoring prompt template. In the process, the acquired frame image is calculated. abnormal scores The following formula is used:

[0093]

[0094] In this embodiment, by giving Anomaly score per sample frame They are extended using weighted interpolation. The weighted interpolation, which considers temporal proximity, contextual continuity, and semantic consistency, is used to obtain a complete anomaly score for the entire video to be detected, based on a complete set of video frames. The scoring prompt template is included. A simple example is as follows: "Based on the reasoning above, from {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}, if any abnormality or violation of safety rules is detected, a relatively high score is assigned to reflect the potential risk. Lower scores indicate normal behavior, and higher scores indicate abnormal behavior. The strict format is: Answer: [Your brief reasoning], Output Abnormality Score: [A value from the list above]."

[0095] Specifically, based on the anomaly scores of sampled frame images in the sampled frame set, the complete anomaly score of the video set to be detected is obtained through weighted interpolation. This includes: for a certain non-sampled frame image, calculating the contextual similarity between the non-sampled frame image and each sampled frame image; determining the weight between the non-sampled frame image and each sampled frame image based on the contextual similarity; calculating the anomaly score of the non-sampled frame image based on the contextual similarity and weights; and finally, obtaining the complete anomaly score of the video set to be detected based on the anomaly scores of all non-sampled frame images and all sampled frame images.

[0096] Specifically, define non-sampled frames. and sampling frames Contextual similarity between The following formula is used:

[0097] ;

[0098] In the above formula, It is the context similarity of consecutive frames.

[0099] Furthermore, weight It is given by the following formula:

[0100] ;

[0101] In the above formula, , and The effects of temporal distance, contextual similarity, and joint semantic strength are controlled separately.

[0102] Finally, for non-sampled frames Interpolation anomaly score for:

[0103] ;

[0104] In this paper, we also demonstrate through experiments that our method combines domain knowledge with visual evidence using only a small number of video examples and domain-specific text input, and performs video anomaly detection in a way that is essentially interpretable to humans.

[0105] The experiments were conducted on two specific scenario datasets: the UT dataset, a scenario-specific dataset collected from real-world urban integrated utility tunnels, containing 480 videos and various infrastructure-related anomalies. Domain knowledge was derived from the "2018 Compilation of Operation and Management Regulations" (details provided in the appendix); and the TAD dataset, a traffic anomaly dataset containing 500 real-world video clips covering events such as accidents and illegal U-turns. Domain knowledge from road traffic safety laws was also incorporated into the experiments.

[0106] For the UT dataset, the area under the frame-level ROC curve on the overall test set was calculated. This is standard practice in video anomaly detection. Additionally, this example calculates Mean Precision (AP) as a supplementary metric to provide a more comprehensive evaluation. For the TAD dataset, this example also uses the following... And AUC only on anomalous videos ( ). Completely normal videos (all with 0 labels) were excluded, and videos containing both normal and abnormal segments were focused on, thus providing a more informative metric for localization performance.

[0107] In the experiments, baseline methods were divided into two categories: uninterpretable and interpretable methods. The uninterpretable method category contains methods that only generate anomaly scores without providing a natural language explanation for video anomaly detection. In contrast, interpretable methods elucidate the reasoning behind anomaly detection by generating easily understandable explanations, thereby improving the model's transparency.

[0108] In the experiments, the performance of our proposed method (DEVA) was systematically compared with state-of-the-art video anomaly detection methods on the UT dataset (Table 1) and the TAD dataset (Table 2), covering both traditional non-interpretable methods and the latest interpretable multimodal methods. On the UT dataset, DEVA achieved a baseline AUC of 75.57% and an AP of 64.54%, significantly outperforming all other mainstream methods. In particular, DEVA achieved the highest scores among interpretable methods on both metrics, further validating its effectiveness and robustness in complex real-world scenarios. On the TAD dataset, DEVA's baseline AUC reached 71.11%, significantly higher than all other interpretable methods, demonstrating the superior ability of our invention to accurately detect video anomalies. Furthermore, DEVA's baseline AUC of 87.14% represents the best performance among interpretable video anomaly detection methods. In conclusion, our invention not only maintains competitiveness compared to traditional non-interpretable methods but also sets a new state-of-the-art level among methods based on interpretable multimodal large models.

[0109] In the experiments, ablation experiments were also conducted on different combinations of components in this method to evaluate their individual and collective contributions, as shown in Table 3. Applying DKQL alone (row 1) yielded 65.15%. This indicates that domain-guided question generation provides meaningful semantic priors, which are beneficial for downstream reasoning. Using FARD alone (line 3) yielded 69.49% This achieves the highest single-module performance, highlighting the effectiveness of structured, multi-step anomaly reasoning. Combining DKQL with AAFS (line 4) improves the result to 71.19%, indicating that semantically guided queries help AAFS focus on information-rich frames. Further combining DKQL with FARD (line 5) improves the score to 71.41%, demonstrating the strong synergy between domain knowledge and fine-grained reasoning. The complete combination of all three modules (line 7) achieves the best performance, reaching 75.57%. And 64.54% AP.

[0110] Figure 3 We compared the AUC_O performance of three frame sampling strategies (uniform sampling, Top-K sampling, and AAFS sampling) at different frame numbers (M= 8, 12, and 16) on the UT dataset. It can be seen that the AUC_O of the three methods is improved as the number of sampled frames increases. Among them, AAFS is consistently better than the other two methods and reaches the highest value of 75.57 when M=16, indicating that this method can capture key information in the video more stably and effectively in the anomaly detection task.

[0111] Figure 4 The AP (Average Performance) metrics of the three methods under the same conditions are presented. The results show that AP increases with the number of sampling frames, with AAFS demonstrating the best performance across all sampling frame counts, especially reaching 64.54 at M=16, significantly higher than uniform sampling and Top-K sampling methods. This indicates that AAFS has a stronger advantage in improving anomaly detection performance.

[0112] Table 4 illustrates the effects of using different guiding questions during inference. The default settings (Row 1) use generic prompts such as "Are there any anomalous events in the video frame?" and "Describe the video and indicate if there is an anomaly?", which lack scene specificity and have limited performance. In contrast, DKQL generates domain-knowledge-based questions such as "Are there any people present in the restricted area, especially near obstacles or equipment, indicating possible unauthorized entry or tampering?" and "Are there any signs of environmental changes, such as wet spots or water stains, which could indicate a leak or spill, suggesting a structural or operational anomaly?". Optimal performance is achieved using 10 DKQL questions (Row 3), demonstrating that more questions can encode more domain knowledge and provide greater diversity, which is beneficial for anomaly detection.

[0113] Taking a sample from the TAD dataset (Case ID: 01_Accident_019) as an example, such as Figure 3 and 4 The following diagram illustrates the interpretable reasoning results of this invention (DEVA) compared to LAVAD and AnomalyRuler. In this case, our method selects scene-related guiding questions, such as "Are there visible debris, rollovers, or other signs of accidents disrupting normal traffic flow?". Based on the video content, the model further generates fine-grained sub-questions, such as "Are there signs of debris on the road indicating an accident?" and "Did the second vehicle rollover or visibly deviate from its lane?" Compared to the original guiding questions, these sub-questions focus more on the visual content of the scene and provide more specific details for anomaly identification. DEVA performs fine-grained reasoning through these sub-questions and accurately identifies a rear-end collision between two vehicles, with debris scattered on the road and severe traffic obstruction. In contrast, LAVAD and AnomalyRuler only capture partial scene information, such as the presence of vehicles and the surrounding road environment, but fail to accurately identify the core anomaly—the collision.

[0114] Table 1 Performance comparison on the UT dataset

[0115]

[0116] Table 2 Performance comparison on the TAD dataset

[0117]

[0118] Table 3. Impact of each module in the DEVA framework in ablation studies

[0119]

[0120] Table 4. Performance comparison under different bootstrap problem settings on the UT dataset.

[0121]

[0122] In the aforementioned video anomaly detection method based on domain knowledge-guided large-scale models, a question learning unit based on a linguistic learning paradigm aligns domain-specific textual knowledge with a small number of video examples to generate guiding questions for specific scenarios. Then, an adaptive frame sampling unit is introduced to select frames semantically and contextually relevant to potential anomalies. Finally, a fine-grained anomaly reasoning and detection unit refines these questions into frame-level queries and performs multi-step reasoning on the sampled frames to achieve accurate and interpretable anomaly detection. This method is the first to utilize domain-specific textual knowledge to guide video anomaly detection and reasoning. This approach significantly reduces the reliance on large-scale video data and anomaly annotations, providing a new paradigm that differs from both zero-shot reasoning and large-scale model instruction adjustments. Furthermore, this method has been successfully deployed in a real tunnel monitoring system in a certain region, demonstrating strong practical value.

[0123] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0124] In one embodiment, such as Figure 5 As shown, a video anomaly detection device based on a domain knowledge-guided large model is provided, comprising: a data acquisition module 200, a final guidance question set acquisition module 210, a keyframe sampling module 220, an anomaly score calculation module 230, and a video anomaly detection module 240, wherein:

[0125] The data acquisition module 200 is used to acquire the set of videos to be detected, a small subset of sample videos from the training phase, domain knowledge documents to provide anomaly categories and causal patterns, and real label sequences with fine-grained labels for corresponding video frames.

[0126] The final guidance question set is obtained by module 210, which is used to construct the guidance question set. The employee model and the dominant model perform multiple rounds of iterative optimization on the guidance question set based on the small subset of sample videos, the domain knowledge documents, and the real label sequence to obtain the final guidance question set.

[0127] The keyframe sampling module 220 is used to convert the final set of guiding questions into a set of affirmative statement phrases, and embed the set of affirmative statement phrases and the set of videos to be detected respectively. Based on the embedded text representation set and visual representation set, the semantic and context-related scores of each frame of the image to be detected in the set of videos to be detected are calculated, and then keyframe sampling is performed from the set of videos to be detected to obtain a set of sampled frames.

[0128] The anomaly score calculation module 230 is used to construct a question set including multiple sub-questions for each sampled frame image in the sampled frame set according to the final guiding question set, and then input the sampled frame images and the corresponding question set into a pre-trained multimodal large model to obtain the reasoning answer for each question in the question set, and generate an anomaly score based on the reasoning answer.

[0129] The video anomaly detection module 240 is used to obtain the complete anomaly score of the video set to be detected based on the anomaly score of the sampled frame images in the sampled frame set, and to perform video anomaly detection based on the complete anomaly score and the reasoning answer of each sampled frame image in the sampled frame set.

[0130] Specific limitations regarding the video anomaly detection device based on domain knowledge-guided large-scale models can be found in the limitations of the video anomaly detection method based on domain knowledge-guided large-scale models described above, and will not be repeated here. Each module in the aforementioned video anomaly detection device based on domain knowledge-guided large-scale models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0131] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video anomaly detection method based on a domain knowledge-guided large model. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0132] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0134] The process involves acquiring the set of videos to be detected, a small subset of sample videos from the training phase, domain knowledge documents to provide anomaly categories and causal patterns, and real label sequences with fine-grained labels for the corresponding video frames.

[0135] A set of guiding questions is constructed, and the employee model and the dominant model perform multiple rounds of iterative optimization on the set of guiding questions based on the small subset of sample videos, the domain knowledge documents, and the real label sequence to obtain the final set of guiding questions.

[0136] The final set of guiding questions is transformed into a set of affirmative statement phrases, and the set of affirmative statement phrases and the set of videos to be detected are embedded respectively. Based on the embedded text representation set and visual representation set, the semantic and context-related scores of each frame of the image to be detected in the set of videos to be detected are calculated. Then, keyframe sampling is performed from the set of videos to be detected to obtain a set of sampled frames.

[0137] Based on the final set of guiding questions, a question set including multiple sub-questions is constructed for each sampled frame image in the sampled frame set. The sampled frame images and the corresponding question set are input into a pre-trained multimodal large model to obtain the reasoning answer for each question in the question set, and an anomaly score is generated based on the reasoning answer.

[0138] The complete anomaly score of the video set to be detected is obtained based on the anomaly score of the sampled frame images in the sampled frame set. Video anomaly detection is achieved based on the complete anomaly score and the inference answer of each sampled frame image in the sampled frame set.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0140] The process involves acquiring the set of videos to be detected, a small subset of sample videos from the training phase, domain knowledge documents to provide anomaly categories and causal patterns, and real label sequences with fine-grained labels for the corresponding video frames.

[0141] A set of guiding questions is constructed, and the employee model and the dominant model perform multiple rounds of iterative optimization on the set of guiding questions based on the small subset of sample videos, the domain knowledge documents, and the real label sequence to obtain the final set of guiding questions.

[0142] The final set of guiding questions is transformed into a set of affirmative statement phrases, and the set of affirmative statement phrases and the set of videos to be detected are embedded respectively. Based on the embedded text representation set and visual representation set, the semantic and context-related scores of each frame of the image to be detected in the set of videos to be detected are calculated. Then, keyframe sampling is performed from the set of videos to be detected to obtain a set of sampled frames.

[0143] Based on the final set of guiding questions, a question set including multiple sub-questions is constructed for each sampled frame image in the sampled frame set. The sampled frame images and the corresponding question set are input into a pre-trained multimodal large model to obtain the reasoning answer for each question in the question set, and an anomaly score is generated based on the reasoning answer.

[0144] The complete anomaly score of the video set to be detected is obtained based on the anomaly score of the sampled frame images in the sampled frame set. Video anomaly detection is achieved based on the complete anomaly score and the inference answer of each sampled frame image in the sampled frame set.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for video anomaly detection based on domain knowledge guided large model, characterized in that, The method comprises: acquiring a video set to be detected, a small sample video subset in a training stage, a domain knowledge document for providing an abnormality category and a causal pattern, and a real label sequence corresponding to a video frame fine-grained label; constructing a guide question set, iteratively optimizing the guide question set by an employee model and a leading model according to the small sample video subset, the domain knowledge document, and the real label sequence, and obtaining a final guide question set, wherein when updating the guide question set: using the employee model to perform abnormality detection on each frame image in the small sample video subset according to the current updated guide question set, obtaining a binary prediction result, and using the leading model to screen out a prediction error frame according to the binary prediction result and the real label sequence; analyzing the prediction failure reason by integrating the guide question set, the binary prediction result, the prediction error frame, and the corresponding real label, and the domain knowledge document, identifying a knowledge dimension not embodied in the guide question set in the domain knowledge document, and updating the guide question set, obtaining the final guide question set through multiple iterations of the employee model and the leading model, wherein the employee model detects the small sample video frame with the current question set, and the leading model optimizes the question set corresponding to the error case in combination with the domain knowledge; converting the final guide question set into a set of affirmative statement phrases, embedding the set of affirmative statement phrases and the video set to be detected respectively, calculating a text representation set and a visual representation set based on the embedding to obtain a semantic and context-related score of each frame to-be-detected image in the video set to be detected, and then sampling key frames from the video set to be detected to obtain a sample frame set; constructing a question set comprising multiple sub-questions for each sample frame image in the sample frame set according to the final guide question set, inputting the sample frame image and the corresponding question set into a pre-trained multi-modal large model to obtain a reasoning answer of each question in the question set, and generating an abnormality score according to the reasoning answer; obtaining a complete abnormality score of the video set to be detected according to the abnormality score of the sample frame image in the sample frame set, and realizing video abnormality detection according to the complete abnormality score and the reasoning answer of each sample frame image in the sample frame set.

2. The method of claim 1, wherein the domain knowledge guided large model-based video anomaly detection method is characterized by, The guide question set is a set of learnable natural language parameters and is updated under the guidance of the domain knowledge document.

3. The method of claim 2, wherein the domain knowledge guided large model-based video anomaly detection method is characterized by, When calculating the semantic and context-related score of each frame to-be-detected image in the video set to be detected based on the embedding to obtain a text representation set and a visual representation set: the visual representation set comprises visual features of each frame to-be-detected image in the video set to be detected, and the text representation set comprises text features of each guide question phrase in the guide question set; for each frame to-be-detected image, the maximum cosine similarity between the corresponding visual feature and all text features in the text representation set is calculated as the semantic-related score; For each frame of the to-be-detected image, a similarity between the to-be-detected image and adjacent two frames of to-be-detected images is calculated according to the visual feature as the context-related score.

4. The method of claim 3, wherein the domain knowledge guided large model-based video anomaly detection method is characterized by, According to the semantic-related score and the context-related score of the to-be-detected image, key frame sampling is performed on the to-be-detected video set to obtain a set of sampled frames, including: According to the semantic-related score and the context-related score, a comprehensive total score of each frame of the to-be-detected image is calculated; An accumulated distribution function is calculated according to the comprehensive total scores of all to-be-detected images, and a preset number of points are uniformly sampled along an accumulated axis based on the accumulated distribution function; The sampled points are mapped back to the original time axis to obtain a plurality of to-be-detected images, i.e., the sampled frames.

5. The method of claim 4, wherein the domain knowledge guided large model based video anomaly detection method is characterized by, When constructing a question set including a plurality of sub-questions for each sampled frame image in the set of sampled frames according to the final set of guide questions: In the final set of guide questions, the most relevant guide questions are selected for each sampled frame image; Each selected guide question is further divided into a plurality of sub-questions, and a question set for the corresponding sampled frame image is constructed according to the plurality of sub-questions obtained by the division.

6. The method of claim 5, wherein the domain knowledge guided large model-based video anomaly detection method is characterized by, In the final set of guide questions, the most relevant 3 guide questions are selected for each sampled frame image.

7. The method of claim 5, wherein the domain knowledge guided large model-based video anomaly detection method is characterized by, When generating an anomaly score for the sampled frame image according to the reasoning answer, it includes: According to the question set of the sampled frame image, the reasoning answer of each question in the corresponding question set, and the sampled frame image itself, the anomaly score is calculated by using a scoring prompt template.

8. The method of claim 7, wherein the domain knowledge guided large model-based video anomaly detection method is characterized by, When obtaining the complete anomaly score of the to-be-detected video set according to the anomaly scores of the sampled frame images in the set of sampled frames, a weighted interpolation completion method is used, specifically including: For a non-sampled frame image, the context similarity between the non-sampled frame image and each sampled frame image is calculated; According to the context similarity, the weight between the non-sampled frame image and each sampled frame image is determined; According to the context similarity and the weight between the non-sampled frame image and each sampled frame image, the anomaly score of the non-sampled frame image is calculated; According to the anomaly scores of all non-sampled frame images and all sampled frame images, the complete anomaly score of the to-be-detected video set is obtained.

9. An apparatus for video anomaly detection based on domain knowledge guiding a large model, characterized in that, The device implements the video anomaly detection method based on the domain knowledge guided large model according to any one of the above claims 1-8, and the device includes: A data acquisition module is configured to acquire a to-be-detected video set, a small sample video subset in a training phase, a domain knowledge document for providing an abnormal category and a causal pattern, and a real label sequence of a corresponding video frame fine-grained label; A final set of guide questions obtaining module is configured to construct a guide question set, and to perform multi-round iteration optimization on the guide question set by an employee model and a leading model according to the small sample video subset, the domain knowledge document, and the real label sequence to obtain a final set of guide questions. a key frame sampling module configured to convert the final set of guide questions into a set of affirmative statement phrases, and respectively embed the set of affirmative statement phrases and the set of videos to be detected, calculate a set of text representations and a set of visual representations based on the embeddings, and obtain semantic and context-related scores of each frame of images to be detected in the set of videos to be detected, and sample key frames from the set of videos to be detected to obtain a set of sampled frames; an abnormal score calculation module configured to construct, according to the final set of guide questions, a set of questions including a plurality of sub-questions for each sampled frame image in the set of sampled frames, input the sampled frame image and the corresponding set of questions into a pre-trained multi-modal large model to obtain an inference answer for each question in the set of questions, and generate an abnormal score according to the inference answer; a video anomaly detection module configured to obtain a complete abnormal score of the set of videos to be detected according to the abnormal scores of the sampled frame images in the set of sampled frames, and realize video anomaly detection according to the complete abnormal score and the inference answer of each sampled frame image in the set of sampled frames.

Images