A method and system for video anomaly detection based on adaptive routing size model cooperation
By employing a collaborative approach combining adaptive routing and inference anchor score correction, the contradiction between inference efficiency and accuracy in video anomaly detection is resolved, achieving efficient and accurate video anomaly detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-08
- Publication Date
- 2026-06-26
Smart Images
Figure CN122290022A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and multimodal deep learning, specifically relating to a method and system for video anomaly detection based on adaptive routing and a collaborative model of size. Background Technology
[0002] With the development of multimodal large models, video understanding technology has shifted from simple pattern recognition to complex semantic reasoning. Existing multimodal large models have demonstrated excellent capabilities in interpreting visual context and answering open-ended questions. However, when applied to fine-grained tasks such as video anomaly detection, they face a fundamental contradiction between "inference depth" and "inference efficiency."
[0003] Traditional video anomaly detection methods primarily rely on feature-based expert models. These models are similar to the human "System 1" (rapid perception), exhibiting high inference efficiency, but operating within a latent feature space and lacking explicit semantic localization. Therefore, they struggle to handle semantic ambiguity; for example, they cannot distinguish between "playing around" and "violent fighting" based solely on motion patterns. In contrast, large multimodal models represent "System 2" (deep understanding), possessing the cognitive depth to resolve such ambiguities, but their massive parameter count leads to excessively high computational costs for dense frame-level processing. Furthermore, large multimodal models suffer from a "heterogeneity gap": their discrete text output is difficult to align with the continuous precision scores required for fine-grained analysis, and they are prone to producing illusions when visual evidence is weak.
[0004] Therefore, there is an urgent need for a video anomaly detection method that can dynamically combine the efficiency of lightweight expert models with the reasoning capabilities of multimodal large models, and solve the problem of modal heterogeneity. Summary of the Invention
[0005] The problem this invention aims to solve is that existing technologies struggle to balance inference efficiency and semantic accuracy in video anomaly detection, and that there is a heterogeneous gap between the output of multimodal large models and continuous anomaly scores. To address this, this invention proposes a large-model collaborative video anomaly detection method and system based on adaptive routing. It dynamically schedules samples through a context-aware adaptive router and aligns heterogeneous outputs using an inference anchoring score correction mechanism. This method is suitable for surveillance video analysis, fine-grained action recognition, and anomaly detection in complex scenarios.
[0006] The present invention adopts the following technical solution: In a first aspect, this invention proposes a collaborative video anomaly detection method based on an adaptive routing size model, comprising the following steps: S1, acquire video segments of the video to be detected, extract temporal visual features of the video segments using a pre-trained lightweight visual language expert model, and output the expert prediction probability distribution. S2, obtain statistical information on the probability distribution of expert predictions, perform routing mapping on the statistical information and concatenate it with temporal visual features to form a multi-dimensional state vector; the statistical information includes prediction confidence, semantic ambiguity and temporal consistency; S3. Input the multidimensional state vector into the context-aware adaptive router and output the routing probability. If the routing probability is greater than the preset threshold, determine that the current video segment is a difficult sample and proceed to step S4. Otherwise, determine that the current video segment is a simple sample, directly calculate the anomaly score of the video segment based on the expert prediction probability distribution, and proceed to step S6. S4. For difficult samples, input them into a domain-aligned multimodal large model to generate a text response describing the anomaly category of the difficult sample. S5, using the inference anchoring score correction mechanism, calculate the semantic grounding score between the text embedding corresponding to the text response and the temporal visual features, and calculate the anomaly score of the video segment based on the semantic grounding score; S6, combining the anomaly scores of all video segments, yields the final anomaly detection result for the video to be detected.
[0007] Further, in S2, the prediction confidence is obtained by calculating the information entropy of the expert prediction probability distribution; the semantic ambiguity is obtained by calculating the difference between the Top-1 probability and the Top-2 probability in the expert prediction probability distribution; and the temporal consistency is obtained by calculating the Jensen-Shannon divergence between the expert prediction probability distribution of the current video segment and the expert prediction probability distribution of the previous video segment relative to the average distribution.
[0008] Furthermore, the training process of the context-aware adaptive router includes: Video samples are acquired and each video segment is labeled as normal or abnormal. The method in step S1 is used to obtain the expert prediction probability distribution of each video segment. The cross-entropy loss of the expert prediction probability distribution is calculated based on the labeling results. At the same time, each video segment is input into a multimodal large model that has been aligned with the domain to obtain the discrete error function value output by the multimodal large model. Wherein, when the abnormal category label predicted by the multimodal large model is the same as the labeling result, the discrete error function value is 0; otherwise, the discrete error function value is a preset penalty constant of 1. Construct an optimal route label, wherein the optimal route label is 1 if and only if the difference between the cross-entropy loss of the expert prediction probability distribution and the discrete error function value output by the multimodal large model is greater than a preset computational cost tolerance, and is used to indicate the activation of the multimodal large model; otherwise, the optimal route label is 0. The multidimensional state vectors of each video segment are obtained using the method in step S2. These multidimensional state vectors are used as the input to the context-aware adaptive router, and the optimal route label is used as the supervision signal. The binary cross-entropy loss function is then used to train the context-aware adaptive router.
[0009] Furthermore, in S4, the training process of the domain-aligned multimodal large model includes two stages: In the first stage, the basic multimodal large model is fine-tuned under supervision using a video anomaly detection instruction dataset, injecting domain knowledge; the samples in the video anomaly detection instruction dataset consist of video clips and question-and-answer text; the questions in the question-and-answer text are used to instruct the multimodal large model to analyze whether the video clips are anomalies; In the second stage, a reinforcement learning algorithm based on group sequence policy optimization is used, and a composite reward function including format compliance reward and label correctness reward is used to optimize the supervised fine-tuning model.
[0010] Furthermore, in S5, the formula for calculating the semantic grounding score is: ; in, Let be the temporal visual features of the t-th video segment. The text response generated by the multimodal large model describes the anomaly category of the difficult sample. This is a text encoder; the superscript T indicates transpose. For semantic grounding score.
[0011] Furthermore, if the text response generated by the multimodal large model describing the anomaly category of the difficult sample is "normal", then the semantic grounding score will be... Set to a preset negative value.
[0012] Furthermore, the formula for calculating the abnormal score is as follows: ; in, Let t be the anomaly score of the t-th video segment. Let the t-th video segment be a simple sample. The t-th video segment is considered a hard sample. To predict the probability that the t-th video segment is normal using a lightweight visual language expert model. For the sigmoid function, and For scaling parameters, For semantic grounding score.
[0013] Furthermore, the lightweight visual language expert model includes a VadCLIP visual encoder, a temporal modeling module, and a post-processing module; the VadCLIP visual encoder encodes the input video segment, and then the temporal modeling module generates temporal visual features of the video segment; the temporal visual features are then processed by the post-processing module to generate a category-level similarity matrix and converted into an expert prediction probability distribution output.
[0014] Furthermore, the context-aware adaptive router employs a multilayer perceptron.
[0015] Secondly, this invention proposes a size model collaborative video anomaly detection system based on adaptive routing, which is used to implement the above-mentioned size model collaborative video anomaly detection method based on adaptive routing.
[0016] The beneficial effects of this invention are: (1) This invention proposes a large-scale model collaborative framework, which effectively balances the high efficiency of expert models with the strong reasoning ability of multimodal large models, and solves the efficiency-reasoning dilemma in fine-grained video understanding.
[0017] (2) The designed context-aware adaptive router utilizes multidimensional uncertainty statistics and only calls the large model when there is semantic ambiguity, which significantly reduces the computational cost while maintaining high accuracy.
[0018] (3) The proposed reasoning anchoring score correction mechanism solves the modal heterogeneity problem by anchoring the discrete reasoning results of the large model to the continuous visual space and naturally suppresses hallucinations when visual evidence is lacking. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the overall framework of a video anomaly detection method based on an adaptive routing size model. Figure 2 This is a flowchart illustrating a collaborative video anomaly detection method based on an adaptive routing size model. Figure 3 This is a schematic diagram of a video anomaly detection system based on an adaptive routing size model. Detailed Implementation
[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.
[0021] like Figure 1 As shown, this invention proposes a video anomaly detection method based on adaptive routing and a large-scale model collaboration. First, a lightweight visual language expert model is used to extract temporal visual features of video segments and output a predicted probability distribution. A multi-dimensional state vector is constructed and input into a context-aware adaptive router to divide the samples into easy and difficult types. For easy samples, the anomaly score is calculated directly based on the expert prediction results. For difficult samples, the input is used to generate a text response through a domain-aligned multimodal large model, and the semantic grounding score between the text embedding and visual features is calculated using an inference anchoring score correction mechanism to obtain the anomaly score.
[0022] like Figure 2 As shown, the main steps include: Step 1: Build a domain instruction dataset, train a multimodal large model, and deploy a lightweight visual language expert model.
[0023] This step aims to prepare the basic data and initial model capabilities for the collaborative system.
[0024] First, a video anomaly detection instruction dataset is constructed. This dataset is a paired dataset consisting of instructions / questions, video clips, and standard answers. It is used to train a general multimodal large-scale model to understand and follow specialized instructions in the field of video anomaly detection, generating compliant and reliable text responses. Specifically, when constructing the dataset, normal video clips are classified as simple samples. For anomalous videos, a converged expert model is used for prediction. Clips with extremely low anomaly prediction scores are treated as normal samples, while ambiguous clips with higher scores are assigned pseudo-labels by the multimodal large-scale model. These labeled samples constitute both the instruction dataset for supervised fine-tuning of the multimodal large-scale model and the actual labels used to calculate the router's cross-entropy loss and discrete error function values, thereby completing the router's offline training.
[0025] The instruction / question is a video analysis task described in natural language. For example: "Analyze this surveillance video clip to determine if there are any abnormal events, and if so, specify their exact categories." The video clip is a short video segment extracted from a real-world application scenario (such as surveillance video); The standard answer is a standardized text output that meets the task requirements, clearly stating "normal" or "abnormal." If it is abnormal, a specific label from a predefined anomaly category system (e.g., "fighting," "breaking in," "theft") must also be provided. Next, domain-specific supervised fine-tuning is performed on the large model. The multimodal large model is fine-tuned using the constructed instruction dataset, injecting domain knowledge of video anomaly detection to enable it to understand the semantic descriptions of abnormal events, serving as the basis for subsequent collaborative reasoning. During this training phase, the model parameters are adjusted to accurately fit the mapping relationship from "video clip - instruction / question" to "standard answer" in the dataset.
[0026] Finally, based on domain-supervised fine-tuning, reinforcement learning based on group sequence policy optimization is further employed. A composite reward function is used, incorporating both format compliance and label correctness. The format compliance reward ensures that the large model's text output strictly adheres to a pre-defined structured format (e.g., "Conclusion: Abnormal / Normal; Category: A"). The label correctness reward directly measures whether the model's anomaly judgment of video segments is consistent with the true labels. Guided by this reward mechanism, the model's reasoning ability when handling difficult samples is strengthened, thereby improving the robustness and accuracy of the entire collaborative system.
[0027] Through the two-stage training described above, the multimodal large model learns from a massive number of "{instruction / question, video clip, standard answer}" pairing samples and combines them with a reinforcement learning paradigm based on group sequence policy optimization. This enables the model to establish a strong association between the visual patterns of surveillance videos and the semantic descriptions of anomaly analysis, thereby mastering domain-specific knowledge and reasoning patterns.
[0028] Simultaneously, a lightweight visual language expert model is deployed. This model, serving as "System 1" (fast perception), is responsible for processing all input video segments and extracting their temporal visual features. The lightweight visual language expert model outputs an expert prediction probability distribution, providing foundational data for subsequent state vector construction. In this embodiment, the lightweight visual language expert model includes a VadCLIP visual encoder, a temporal modeling module, and a post-processing module. Specifically, firstly, the VadCLIP visual encoder with frozen parameters extracts frame-level features from the input video segment; then, the frame-level features are input into the temporal modeling module, which includes positional encoding and a multi-layer Transformer encoder, to extract temporal visual features capturing time dependencies; finally, the post-processing module dynamically generates visual conditional text prompts based on the temporal visual features using a multi-layer perceptron and cross-attention mechanism, and calculates the cosine similarity between the temporal visual features and the generated text prompts to generate a category-level similarity matrix, which is ultimately converted into an expert prediction probability distribution output.
[0029] In this embodiment, the lightweight visual language expert model is obtained through pre-training. This training process is existing technology in the field and will not be described in detail here.
[0030] Step 2: Constructing a multidimensional state vector and extracting statistical features This step aims to quantify the model's cognitive state regarding the current video segment, such as... Figure 1 As shown, the specific content includes: Based on the expert prediction probability distribution output in step one, expert prediction statistics are calculated. These statistics are then routed and concatenated with temporal visual features to construct a multi-dimensional state vector. The expert prediction statistics include the following three key dimensions: Prediction confidence ( : Used to measure the uncertainty of a model, obtained by calculating the information entropy of the probability distribution predicted by experts.
[0031] Semantic ambiguity ( ): Used to capture the degree of class confusion, obtained by calculating the difference between the Top-1 probability and the Top-2 probability in the expert prediction probability distribution.
[0032] Timing consistency ( ): Used to quantify the flickering degree of the prediction, obtained by calculating the Jensen-Shannon divergence of the current prediction distribution and the previous prediction distribution relative to the mean distribution.
[0033] In this embodiment, the routing mapping is implemented using a multilayer perceptron, which is used to fuse and increase the dimensionality of the three statistical scalars mentioned above, transforming the three-dimensional statistical feature vector... The input is mapped to a high-dimensional statistical feature embedding vector, which is then concatenated with the temporal visual features extracted in step one. The resulting multidimensional state vector contains visual context and the model's own cognitive metadata, which is used to guide subsequent routing decisions.
[0034] Step 3: Perform context-aware adaptive routing scheduling To balance inference efficiency and depth, the system dynamically schedules samples using a context-aware adaptive router.
[0035] The multidimensional state vector generated in step two is input into the context-aware adaptive router, which is implemented using a lightweight multilayer perceptron (MLP). This router projects and maps the multidimensional state vector to values between 0 and 1, and then outputs the final routing probability after passing it through a sigmoid activation function. The routing probability is then determined based on a preset threshold. If the routing probability is less than or equal to the threshold (set to 0.5), the current segment is determined to be a "simple sample". At this point, the system considers the judgment of the lightweight visual language expert model to be reliable enough, and directly calculates the anomaly score based on the expert prediction probability distribution, without needing to call the multimodal large model.
[0036] If the routing probability is greater than the threshold, the current segment is determined to be a "hard sample". 1) At this point, the system activates "System 2" (Deep Understanding), inputs the sample into a domain-aligned multimodal big model, and performs deep reasoning based on the original video clips to generate a text response describing the anomaly category of the difficult sample.
[0037] Before the system is actually deployed for inference, the context-aware adaptive router needs to be trained offline. The dataset used in its training process is from the same source as the dataset used for fine-tuning the multimodal large model in step one. First, the discrete error function value of the multimodal large model output is defined (0 when the model's predicted class is the same as the true label, otherwise a penalty constant of 1). Next, the optimal route label is constructed: the optimal route label is determined if and only if the difference between the cross-entropy loss of the expert prediction probability distribution and the discrete error function value of the multimodal large model is greater than a preset computational cost tolerance. When the condition is met, it indicates that calling the large model can bring significant performance benefits, and the optimal route label is set to 1 (indicating activation of the large model); otherwise, it is set to 0. This process is specifically represented as follows:
[0038] in, This means that the value is 1 if the expression is true, and 0 otherwise. These are pseudo tags annotated by Qwen3VL-Max. For expert model prediction results, For cross-entropy loss, This represents the discrete error function value output by the multimodal large-scale model. When the multimodal large-scale model predicts incorrectly, the optimal route label is always 0; when the expert model predicts correctly with extremely high confidence, the optimal route label is 0; and when the multimodal large-scale model predicts correctly and the expert model's cross-entropy loss is greater than the set computational cost tolerance constant... When routing is performed, the optimal route label is 1.
[0039] Finally, the multidimensional state vectors of the training set fragments are extracted as input to the router, the optimal route label is used as the supervision signal, and the binary cross-entropy loss function is used to update the router's model weights.
[0040] Step 4: Calculate the inference anchoring score and semantic grounding. For video segments identified as difficult samples, this invention no longer relies solely on discrete text labels, but instead solves the modal heterogeneity problem through an inference anchor score correction mechanism.
[0041] The text response generated by the multimodal large model describes the anomaly category of the difficult sample. Input the text encoder and extract its text embedding. Calculate the relationship between this text embedding and the temporal visual features extracted in step one. Semantic grounding score between The specific calculation formula is as follows:
[0042] in, For text encoders, this embodiment uses the VadCLIP text encoder. If the text response generated by the multimodal large model describing the anomaly category of the difficult sample is "normal", then it is directly... The default negative value is set, and in this embodiment, the negative value is set to negative infinity.
[0043] This step anchors the discrete inference results of the large model back to the continuous visual feature space, achieving semantic alignment and effectively suppressing model illusions when visual evidence is weak.
[0044] Step 5: Calculate the final outlier score The final anomaly score is calculated based on the routing results. :
[0045] in, This represents the probability that a lightweight visual language expert model predicts something as normal. For the sigmoid function, and These are scaling parameters, and the two parameters are empirical values obtained through testing on the validation set. Set to 10, Setting it to -2 is used to align and smooth the distribution range of grounding fractions with the normal or abnormal probability interval (between 0 and 1).
[0046] The anomaly scores of all video segments are combined to obtain the final anomaly detection result for the video to be detected.
[0047] This invention proposes a video anomaly detection method that utilizes a multimodal large model in collaboration with an expert model. By employing a context-aware adaptive routing module, it addresses the excessive computational overhead of frame-by-frame detection in long video understanding. Furthermore, an inference anchoring score module enables feedback from the multimodal large model to the expert model. This significantly improves the efficiency of using a multimodal large model for video anomaly detection without sacrificing performance.
[0048] To verify the effectiveness of the proposed adaptive routing-based size model collaborative video anomaly detection method, detailed experimental verification was conducted on two widely used video anomaly detection benchmark datasets, UCF-Crime and XD-Violence.
[0049] This experiment uses the following two datasets for performance evaluation: (1) UCF-Crime dataset: This is a large-scale weakly supervised dataset containing 1900 unpruned surveillance videos, totaling 128 hours in length. The dataset covers 13 categories of real-world anomalies (such as explosions, traffic accidents, fights, robberies, etc.) as well as normal activities. It is trained using video-level labels and tested using frame-level annotations, following standard protocols.
[0050] (2) XD-Violence dataset: This is a multimodal violence detection dataset containing 4,754 videos collected from movies and outdoor scenes, with a total duration of 217 hours. The dataset includes 6 anomaly categories, including abuse, car accidents, explosions, fights, riots, and shootings, and emphasizes joint detection using audio and visual cues.
[0051] The collaborative system of this invention comprises two parts: a lightweight expert model and a multimodal large model. Lightweight visual language expert model: It adopts VadCLIP as the backbone network, the temporal encoder contains 2 Transformer layers, and the hidden layer dimension is 512.
[0052] Multimodal Large Model (MLLM): InternVL3.5-8B is used as the main inference kernel (Qwen3-VL-8B-Instruct was also tested as a variant). When the large model is activated by the router, the context window is set to sample 16 frames uniformly from the target segment.
[0053] Training strategy: The multimodal large model is first supervised fine-tuning (SFT) on a constructed video anomaly detection instruction dataset, followed by alignment using reinforcement learning based on group sequence policy optimization (GSPO). Training is performed on 8 NVIDIA A100 GPUs.
[0054] Context-Aware Adaptive Routers: Computational Cost Tolerance Set it to 0.15, and set the initial activation threshold to 0.5.
[0055] The evaluation metrics commonly used in the field of video anomaly detection are adopted: (1) Area under the ROC curve (AUC): Used to evaluate the frame-level detection performance of the UCF-Crime dataset. The higher the AUC value, the stronger the model's ability to distinguish between abnormal frames and normal frames.
[0056] (2) Average precision (AP): used to evaluate the detection performance of the XD-Violence dataset, taking into account the precision under different recall rates.
[0057] Table 1 shows the comparison results between the method of this invention and existing mainstream methods on the UCF-Crime and XD-Violence datasets. The comparison methods include weakly supervised methods (such as VadCLIP, CLIP-TSA) and multimodal zero-shot / few-shot methods (such as HeadHunt-VAD, LLAVA-1.5).
[0058] As can be seen from Table 1, the CollaVAD method proposed in this invention achieves optimal performance on both datasets.
[0059] On the UCF-Crime dataset, CollaVAD, based on InternVL3.5, achieved an AUC of 89.12%, a 1.1% improvement over the top-performing weakly supervised benchmark VadCLIP (88.02%), and a 2.09% improvement over similar multimodal methods like HeadHunt-VAD. On the XD-Violence dataset, this method achieved an AP of 86.35%, significantly outperforming VadCLIP (84.51%), with an improvement of 1.84%. This demonstrates that by introducing the inference capabilities of a large model, this invention can effectively handle complex violence scenarios and fine-grained semantic ambiguity.
[0060] Table 1. Performance comparison of different methods on the UCF-Crime and XD-Violence datasets.
[0061] To verify the effectiveness of each core component in this invention, Table 2 presents the ablation experiment results on the UCF-Crime dataset. The effects of using only the expert model, introducing full routing, introducing an adaptive router, and introducing an inference anchor score correction mechanism were evaluated. Full routing refers to a strategy that directly sends all samples with uncertain predictions from the expert model (i.e., samples with anomaly scores greater than 0.01 and less than 0.99 predicted by the expert model) to the multimodal large model for processing without intelligent filtering by the adaptive router. This control experiment aims to test the performance ceiling when the large model is fully involved; however, the lack of intelligent scheduling leads to a significant increase in system computational costs.
[0062] Table 2 Ablation experiments of key components (Dataset: UCF-Crime)
[0063] Using only a lightweight expert model, the AUC is 85.50%, which, while fast inference, has limited ability to handle complex semantics. Introducing a context-aware adaptive router allows the model to intelligently identify "hard samples" and invoke a larger model, improving the AUC to 87.39%. This result demonstrates the effectiveness of constructing multidimensional state vectors using prediction confidence, semantic ambiguity, and temporal consistency. The role of the inference anchoring score correction mechanism: adding a score correction mechanism to the routing further boosts performance to 89.12%. In summary, the proposed large-scale model collaborative video anomaly detection method balances efficiency and depth through an adaptive routing strategy and achieves modality alignment through a score correction mechanism, achieving significant performance improvements on multiple benchmark datasets.
[0064] Based on the same inventive concept, this invention also discloses a size model-based collaborative video anomaly detection system using adaptive routing, such as... Figure 3 As shown, it includes: The feature extraction module is used to acquire video segments of the video to be detected, extract the temporal visual features of the segments using a lightweight visual language expert model, and output the expert prediction probability distribution. A state construction module is used to construct a multi-dimensional state vector; the multi-dimensional state vector is formed by concatenating the temporal visual features with expert prediction statistics; the expert prediction statistics include prediction confidence, semantic ambiguity, and temporal consistency. The routing and scheduling module is used to input multi-dimensional state vectors into the context-aware adaptive router and output routing probabilities. If the routing probability is greater than a preset threshold, the current segment is determined to be a hard sample; otherwise, it is determined to be a simple sample, and the anomaly score is directly calculated based on the expert prediction probability distribution. The segments determined to be hard samples are input into the domain-aligned multimodal large model to generate a text response describing the anomaly category. The score correction module is used to calculate the semantic grounding score between the text embedding corresponding to the text response generated by the multimodal large model and the temporal visual features using the inference anchoring score correction mechanism, and to calculate the anomaly score of the video segment based on the semantic grounding score. The results output module is used to combine the anomaly scores of all video segments and output the final anomaly detection result of the video to be detected.
[0065] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0066] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.
[0067] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.
Claims
1. A video anomaly detection method based on an adaptive routing-based size model collaborative method, characterized in that, Includes the following steps: S1, acquire video segments of the video to be detected, extract temporal visual features of the video segments using a pre-trained lightweight visual language expert model, and output the expert prediction probability distribution. S2, obtain statistical information on the probability distribution of expert predictions, perform routing mapping on the statistical information and concatenate it with temporal visual features to form a multi-dimensional state vector; the statistical information includes prediction confidence, semantic ambiguity and temporal consistency; S3: Input the multi-dimensional state vector into the context-aware adaptive router and output the routing probability; If the routing probability is greater than the preset threshold, the current video segment is determined to be a difficult sample and proceeds to step S4; otherwise, the current video segment is determined to be a simple sample, and the anomaly score of the video segment is directly calculated based on the expert prediction probability distribution and proceeds to step S6. S4. For difficult samples, input them into a domain-aligned multimodal large model to generate a text response describing the anomaly category of the difficult sample. S5, using the inference anchoring score correction mechanism, calculate the semantic grounding score between the text embedding corresponding to the text response and the temporal visual features, and calculate the anomaly score of the video segment based on the semantic grounding score; S6, combining the anomaly scores of all video segments, yields the final anomaly detection result for the video to be detected.
2. The video anomaly detection method based on adaptive routing and a size model collaborative method according to claim 1, characterized in that, In S2, the prediction confidence is obtained by calculating the information entropy of the expert prediction probability distribution; the semantic ambiguity is obtained by calculating the difference between the Top-1 probability and the Top-2 probability in the expert prediction probability distribution; and the temporal consistency is obtained by calculating the Jensen-Shannon divergence between the expert prediction probability distribution of the current video segment and the expert prediction probability distribution of the previous video segment relative to the average distribution.
3. The video anomaly detection method based on adaptive routing and a size model collaborative method according to claim 1, characterized in that, The training process of the context-aware adaptive router includes: Video samples are acquired and each video segment is labeled as normal or abnormal. The method in step S1 is used to obtain the expert prediction probability distribution of each video segment. The cross-entropy loss of the expert prediction probability distribution is calculated based on the labeling results. At the same time, each video segment is input into a multimodal large model that has been aligned with the domain to obtain the discrete error function value output by the multimodal large model. Wherein, when the abnormal category label predicted by the multimodal large model is the same as the labeling result, the discrete error function value is 0; otherwise, the discrete error function value is a preset penalty constant of 1. Construct an optimal route label, wherein the optimal route label is 1 if and only if the difference between the cross-entropy loss of the expert prediction probability distribution and the discrete error function value output by the multimodal large model is greater than a preset computational cost tolerance, and is used to indicate the activation of the multimodal large model; otherwise, the optimal route label is 0. The multidimensional state vectors of each video segment are obtained using the method in step S2. These multidimensional state vectors are used as the input to the context-aware adaptive router, and the optimal route label is used as the supervision signal. The binary cross-entropy loss function is then used to train the context-aware adaptive router.
4. The video anomaly detection method based on adaptive routing and a size model collaborative method according to claim 1, characterized in that, In S4, the training process of the domain-aligned multimodal large model includes two stages: In the first stage, the basic multimodal large model is fine-tuned under supervision using a video anomaly detection instruction dataset, and domain knowledge is injected. The samples in the video anomaly detection instruction dataset consist of video clips and question-and-answer text; the questions in the question-and-answer text are used to instruct the multimodal large model to analyze whether the video clips are anomalies. In the second stage, a reinforcement learning algorithm based on group sequence policy optimization is used, and a composite reward function including format compliance reward and label correctness reward is used to optimize the supervised fine-tuning model.
5. The method for collaborative video anomaly detection based on an adaptive routing size model according to claim 1, characterized in that, In S5, the formula for calculating the semantic grounding score is: ； in, Let be the temporal visual features of the t-th video segment. The text response generated by the multimodal large model describes the anomaly category of the difficult sample. This is a text encoder; the superscript T indicates transpose. For semantic grounding score.
6. The method for collaborative video anomaly detection based on size model of adaptive routing according to claim 5, characterized in that, If the text response generated by the multimodal large model describing the anomaly category of the difficult sample is "normal", then the semantic grounding score will be... Set to a preset negative value.
7. The method for collaborative video anomaly detection based on an adaptive routing size model according to claim 1, characterized in that, The formula for calculating the abnormal score is: ； in, Let t be the anomaly score of the t-th video segment. Let the t-th video segment be a simple sample. The t-th video segment is considered a hard sample. To predict the probability that the t-th video segment is normal using a lightweight visual language expert model. For the sigmoid function, and For scaling parameters, For semantic grounding score.
8. The method for collaborative video anomaly detection based on an adaptive routing size model according to claim 1, characterized in that, The lightweight visual language expert model includes a VadCLIP visual encoder, a temporal modeling module, and a post-processing module. The VadCLIP visual encoder encodes the input video segment, and the temporal modeling module generates temporal visual features of the video segment. The temporal visual features are then processed by the post-processing module to generate a category-level similarity matrix and converted into an expert prediction probability distribution output.
9. The method for collaborative video anomaly detection based on an adaptive routing size model according to claim 1, characterized in that, The context-aware adaptive router employs a multilayer perceptron.
10. A size-model collaborative video anomaly detection system based on adaptive routing, used to implement the size-model collaborative video anomaly detection method based on adaptive routing as described in claim 1, characterized in that, The system includes: The feature extraction module is used to acquire video segments of the video to be detected, extract the temporal visual features of the segments using a lightweight visual language expert model, and output the expert prediction probability distribution. A state construction module is used to construct a multi-dimensional state vector; the multi-dimensional state vector is formed by concatenating the temporal visual features with expert prediction statistics; the expert prediction statistics include prediction confidence, semantic ambiguity, and temporal consistency. The routing and scheduling module is used to input multi-dimensional state vectors into the context-aware adaptive router and output routing probabilities. If the routing probability is greater than a preset threshold, the current segment is determined to be a hard sample; otherwise, it is determined to be a simple sample, and the anomaly score is directly calculated based on the expert prediction probability distribution. The segments determined to be hard samples are input into the domain-aligned multimodal large model to generate a text response describing the anomaly category. The score correction module is used to calculate the semantic grounding score between the text embedding corresponding to the text response generated by the multimodal large model and the temporal visual features using the inference anchoring score correction mechanism, and to calculate the anomaly score of the video segment based on the semantic grounding score. The results output module is used to combine the anomaly scores of all video segments and output the final anomaly detection result of the video to be detected.