A video anomaly detection method and system based on semantic modulation and knowledge distillation

Abstract

This invention belongs to the field of computer vision technology and provides a video anomaly detection method and system based on semantic modulation and knowledge distillation. The method involves acquiring a video stream to be detected, extracting image frames from the video stream, generating text descriptions corresponding to the image content for each frame, extracting visual features and textual features from each frame, using the visual features as input, and employing a pre-trained student network to perform temporal modeling of the visual features, generating implicit conditional features, generating modulation parameters and feature modulation amounts, enhancing and fusing the visual features, and then passing the fused features through a classifier to obtain frame-level anomaly scores. During pre-training, the student network undergoes multi-objective distillation training with the teacher network, thus improving training stability and cross-scene generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a video anomaly detection method and system based on semantic modulation and knowledge distillation. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Video anomaly detection aims to automatically identify abnormal events that deviate from normal behavioral patterns in long-term videos from scenarios such as surveillance, security, and industrial inspection. It is characterized by strong real-time performance, large data scale, and significant scene differences. Existing technologies typically use visual information as the primary input, employing convolutional networks, temporal modeling networks, or graph neural networks to learn representations of video frame or segment features and output frame-level or segment-level anomaly scores. However, in real-world applications, anomaly events are diverse and have blurred boundaries. Videos often contain occlusion, lighting changes, viewpoint changes, and background interference, making models relying solely on visual features prone to false positives and false negatives. On one hand, some normal behaviors that resemble abnormal appearances are misjudged as anomalies; on the other hand, some semantic-level anomalies (such as "illegal entry" or "dangerous actions") are not significantly different in low-level visual features, making them difficult to reliably distinguish.

[0004] Due to the scarcity of anomalous samples and the high cost of annotation, anomaly detection tasks often employ weakly supervised or unsupervised learning paradigms, such as providing only video-level labels or training reconstruction / prediction models using normal data. While these methods reduce annotation costs, the supervision signal is weak, the model relies more on statistical correlation, and it is prone to overfitting to scene-specific noise, resulting in performance degradation when generalizing across scenes. Furthermore, existing weakly supervised methods generally employ aggregation strategies such as multi-instance learning to obtain video-level predictions by aggregating segment-level scores; however, multi-instance learning is prone to getting stuck in local optima in the early stages of training, with the model tending to focus on a small number of high-response segments, making it difficult to learn stable temporal structures and semantic boundaries, thus affecting the accuracy of anomaly localization.

[0005] To enhance semantic discrimination capabilities, some studies have introduced semantic information from text descriptions, action category priors, or large language models, achieving semantic enhancement through cross-modal alignment or attention mechanisms. These multimodal methods can leverage linguistic information to provide higher-level semantic constraints during training or inference, thereby improving anomaly recognition in complex scenarios. However, existing multimodal solutions often require continuous text input or invocation of cross-modal encoders during the inference phase, leading to additional computational overhead and system deployment complexity. Furthermore, in real-world scenarios such as surveillance, text descriptions are often unavailable or unstable, making it difficult to directly deploy multimodal models.

[0006] Knowledge distillation, a common technique for model compression and transfer, typically aligns the outputs or intermediate features of teacher and student models, allowing students to approximate teacher performance at a lower cost. However, for anomaly detection problems involving multimodal semantic enhancement, traditional distillation primarily focuses on the consistency of classification outputs or global features, making it difficult to effectively transfer the "dynamic modulation ability of language on visual representations." Especially when the teacher model uses text to conditionally modulate or enhance visual features at the interval level, if only the final score is distilled, the student model struggles to reproduce the teacher's semantic guidance behavior in temporal local regions, resulting in limited transfer effectiveness. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes a video anomaly detection method and system based on semantic modulation and knowledge distillation. During the training phase, this invention fully utilizes semantic information such as text to enhance visual representations. Simultaneously, through distillation transfer, it effectively transfers the semantic enhancement capabilities of the teacher network to the student network, which relies solely on visual input. This enables the student network to achieve anomaly detection and localization performance close to that of a multimodal teacher network without the need for text input during the inference phase, while also improving training stability and cross-scene generalization capabilities.

[0008] According to some embodiments, the present invention adopts the following technical solution: A video anomaly detection method based on semantic modulation and knowledge distillation includes the following steps: Acquire the video stream to be detected, extract the image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame; Visual features and textual features of each frame are extracted separately. Using the visual features as input, a pre-trained student network is used to perform temporal modeling on the visual features, and then implicit conditional features are generated. In turn, modulation parameters and feature modulation amounts are generated to enhance and fuse the visual features. The fused features are then passed through a classifier to obtain frame-level anomaly scores. During the pre-training process, the student network is pre-trained by the teacher network through multi-objective distillation. The teacher network takes visual features and text features as input, performs temporal modeling on the visual features, calculates anomaly scores on the text features, and conditionally modulates the visual features under text guidance to form semantically enhanced fusion features. The fusion features are then processed by a classifier to obtain frame-level anomaly scores. The text adaptation features, feature modulation amounts, and prediction results in the teacher network are used as distillation supervision signals in the multi-objective distillation training.

[0009] As an alternative implementation, the process of extracting image frames from the video stream to be detected includes acquiring the video stream in the scene to be detected, parsing it frame by frame, extracting image frames, and using a pre-trained multimodal large language model to generate a text description corresponding to the content of each image frame.

[0010] As an alternative implementation, the process of extracting the visual features of each frame of image and the text features of the text description includes: inputting the image frame and its corresponding text description into a pre-trained visual-language multimodal pre-trained model, and extracting the visual features of each frame of image and the text features of the corresponding text description.

[0011] As an alternative implementation, the teacher network includes: The temporal modeling module is used to perform temporal modeling on the extracted visual features; The text anomaly scoring module is used to calculate anomaly scores for text features to characterize the degree of correlation between text semantics and anomalous events. Text adapters are used to perform dimensional transformation and feature mapping on text features to complete the task-oriented adaptation and processing of the original text features. The modulation parameter generation module generates parameters and gating coefficients for conditional modulation based on the adapted features, so as to achieve semantically guided modulation of visual features. The feature fusion module is used to inject feature modulation amounts into visual features under the guidance of text semantics to form a semantically enhanced fused feature representation; A frame-level classifier is used to classify and predict fused features, and output frame-level anomaly scores.

[0012] As a further defined implementation, in the teacher network, the process by which the modulation parameter generation module generates parameters for conditional modulation and gating coefficients based on the adapted features includes: scaling factor Generate as: Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and U is the trainable parameter matrix; offset factor Generate as: Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and V is the trainable parameter matrix; The basic gating weight g is obtained by transforming the low-dimensional text adaptation features, and the generation formula is: Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and gate is a trainable linear transformation layer used to generate gate scores, which are then activated by an activation function to obtain the basic gate weights. The gating activation coefficient w driven by anomaly confidence is generated by the following formula:

[0013] Where p is the text anomaly confidence score. It's the set threshold. It is a constant less than the set value; the final gating coefficient used for modulation is obtained by multiplying the content-driven basic gating by the text anomaly confidence-driven activation coefficient.

[0014] As an alternative implementation, the student network includes a temporal modeling module, an implicit conditional generation network module, a modulation parameter generation module, a feature fusion module, and a frame-level classifier module. The student network only receives visual features as input. The temporal modeling module performs temporal modeling on the visual features and then inputs them into the implicit conditional generation network module to generate implicit conditional features. After the implicit conditional features pass through two learnable projection matrices, the modulation parameter generation module generates scaling factors and offset factors. The gate weights are obtained by passing through a fully connected layer and an activation function.

[0015] As an optional implementation, the multi-objective distillation training utilizes a distillation loss function, specifically including feature distillation, feature modulation amount distillation, and frame-level score distillation, wherein: Feature distillation selects low-dimensional text adaptation features output by the text adapter in the teacher network and implicit conditional features output by the implicit conditional generation network module in the student network as distillation alignment objects. Feature modulation amount distillation selects the feature modulation amounts in the feature fusion module of the teacher network and the feature modulation amounts in the feature fusion module of the student network as distillation alignment objects; Frame-level score distillation uses the frame-level anomaly scores output by the teacher network as soft labels, ensuring that the distribution of the frame-level anomaly scores output by the student network is consistent with that of the teacher, provided that the student network does not use text input.

[0016] As a further defined implementation, in frame-level score distillation, the teacher network outputs anomaly scores for each frame, introduces a temperature coefficient to soften them, and converts them into frame-level soft label probabilities through an activation function; the student network also uses a temperature coefficient to soften the anomaly scores for each frame when there is no text input. A binary cross-entropy loss is used to measure the difference between student output and teacher soft label, and the square of the temperature coefficient is used to compensate for the reduction in gradient magnitude caused by temperature scaling.

[0017] As an alternative implementation, the teacher network employs a three-stage training strategy. The first stage focuses on visual branch training, optimizing the visual encoding and temporal modeling modules to enable the model to fully learn the spatiotemporal representation capabilities of video sequences. The second stage focuses on text branch training, optimizing the text encoding and text anomaly scoring modules to improve the model's ability to model text semantics and its anomaly correlations. The third stage, based on the training in the first two stages, focuses on training cross-modal modulation-related parameters, enabling the model to conditionally modulate and effectively fuse visual features under the guidance of text semantics, forming semantically enhanced fusion features and improving the final anomaly detection and localization performance.

[0018] As an alternative implementation, the student network adopts a two-stage training strategy. The first stage focuses on learning modulation behavior by freezing the classification head and updating only the modulation-related module parameters, enabling the student network to quickly and stably fit the modulation pattern of the teacher network. The second stage unfreezes the data based on the first stage and jointly optimizes the task objective and the distillation objective.

[0019] A video anomaly detection system based on semantic modulation and knowledge distillation includes: The data acquisition module is configured to acquire the video stream to be detected, extract image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame. The anomaly detection module is configured to extract the visual features and textual features of each frame of image respectively. Using the visual features as input, a pre-trained student network is used to perform temporal modeling on the visual features, then generate implicit conditional features, and then generate modulation parameters and feature modulation amounts to enhance and fuse the visual features. The fused features are then passed through a classifier to obtain frame-level anomaly scores. The multi-objective distillation training module is configured such that the student network undergoes multi-objective distillation training in advance through the teacher network during the pre-training process. The teacher network takes visual features and text features as input, performs temporal modeling on the visual features, calculates anomaly scores on the text features, and conditionally modulates the visual features under text guidance to form semantically enhanced fusion features. The fusion features are passed through a classifier to obtain frame-level anomaly scores. The text adaptation features, feature modulation amount, and prediction results in the teacher network are used as distillation supervision signals in the multi-objective distillation training.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention presents a video anomaly detection method based on semantic modulation and knowledge distillation. By constructing a teacher-student network architecture, textual semantic information is introduced during the training phase to guide visual feature modulation, thereby enhancing the anomaly detection capability of the teacher network. However, during the inference phase, the student network no longer relies on text input, completing anomaly detection solely based on visual features. This structure effectively avoids the overhead problem of existing multimodal methods requiring real-time text, language models, or cross-modal modules during the deployment phase, reducing system complexity and resource consumption.

[0021] The video anomaly detection method based on semantic modulation and knowledge distillation of this invention jointly designs feature modulation amount distillation, feature distillation, and result distillation. This enables the student network to not only effectively reproduce the feature modulation behavior of the teacher network, but also to inherit the semantic fusion representation of the teacher network at the feature level, thereby achieving more complete and finer-grained semantic consistency alignment. With the help of this distillation mechanism, the student model's ability to express and distinguish complex anomalies is significantly improved, especially in anomaly segments with semantic ambiguity and unclear temporal boundaries, achieving more accurate detection results and higher localization accuracy.

[0022] The video anomaly detection method based on semantic modulation and knowledge distillation of this invention drives the student network to learn adaptive enhancement methods from visual features through modulation behavior distillation, thereby improving the model's expressive power and adaptability to new scenarios. This mechanism weakens the dependence on the teacher structure and enhances the flexibility of student network structure design, which helps to freely adjust the network complexity under different task scales or platform requirements, thereby improving the system's versatility and scalability.

[0023] This invention presents a video anomaly detection method based on semantic modulation and knowledge distillation. To avoid modulation degradation (such as gating all doors closed or all doors open) in the early training phase of the student network, this invention employs a two-stage training strategy. The first stage focuses on learning modulation behavior by freezing the classification head and training only modulation-related modules, enabling them to quickly approximate the teacher's modulation behavior. The second stage jointly trains the task objective and the distillation objective, comprehensively improving detection performance. This strategy effectively separates the modulation learning and discriminative learning stages, enhances training stability, avoids optimization conflicts, and significantly improves the convergence speed and performance of the final model under weak supervision.

[0024] This invention presents a video anomaly detection method based on semantic modulation and knowledge distillation. While significantly improving the intelligence and practical performance of video anomaly detection systems, it also greatly enhances the generalization ability to identify various types of complex anomalies. This method balances inference efficiency and operational stability while meeting real-time response requirements, achieving efficient and accurate analysis and alerting of surveillance videos. Based on its excellent engineering deployability and scalability, this invention can be widely applied in various scenarios such as smart city construction, intelligent traffic management, industrial production safety monitoring, and comprehensive park management, possessing high application value and broad market prospects.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a flowchart illustrating the video anomaly detection method based on semantic modulation and knowledge distillation according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the timing modeling module structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the text anomaly scoring module structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the text adapter structure according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the teacher network modulation parameter generation module in an embodiment of the present invention; Figure 6 This is a schematic diagram of the feature fusion module structure according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the frame-level classifier structure according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the implicit conditional generation network module structure according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the distillation alignment path according to an embodiment of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0032] To ensure that the content provided in the embodiments is clear to those skilled in the art, the following explanations of terminology are provided first.

[0033] Video surveillance anomaly detection is an intelligent recognition technology based on computer vision and machine learning. Through real-time or offline analysis of video sequences, it can promptly and accurately detect and locate abnormal events or potential risks that deviate from normal behavioral patterns from massive, continuous surveillance footage. This technology can replace or assist manual inspections, achieving automated alarms and efficient early warnings, reducing labor costs and the probability of missed detections, thereby improving the level of intelligence in security management and risk prevention.

[0034] BLIP2 Model: BLIP2 (Bootstrapping Language-Image Pre-training 2) is a highly efficient vision-language pre-training model with excellent performance in visual semantic understanding and language generation, capable of providing semantic descriptions and explanations for video content. This invention uses the BLIP2 model to process video frames and automatically generate natural language text descriptions corresponding to the frame content.

[0035] CLIP Model: CLIP (Contrastive Language-Image Pretraining) is a vision-language multimodal pretraining model proposed by OpenAI. It achieves joint modeling and alignment of image content and text semantics through comparative learning on large-scale image-text pairs. This invention uses the CLIP model to extract visual features and corresponding textual description features from video frames to construct feature representations for cross-modal semantic alignment.

[0036] Graph Convolutional Networks (GCNs) are deep learning neural networks specifically designed for processing graph data. Their core capabilities are: mining the relationships between nodes in a graph, aggregating features of neighboring nodes, and achieving feature learning and information extraction from graph data.

[0037] Multilayer Perceptron (MLP): A multilayer perceptron (MLP) is a fundamental feedforward artificial neural network. It consists of at least three layers: an input layer, one or more hidden layers, and an output layer. Unlike simple linear models, the core feature of the MLP is the introduction of nonlinear activation functions in the hidden layers. This allows it to learn and simulate complex nonlinear relationships, solving classification and regression problems, and is a crucial cornerstone of deep learning technology.

[0038] Teacher-Student Networks: Teacher-Student Networks represent an advanced paradigm for training machine learning models. This paradigm aims to guide the learning of a student model—which has a more streamlined structure and lower computational cost—using a pre-trained and high-performance "teacher model." Its core strategy lies in employing knowledge distillation techniques to effectively transfer the deep knowledge inherent in the teacher model—such as the output class probability distribution or intermediate feature representations—to the student model. Through this process, the student model can approximate or even surpass the performance of the teacher model while maintaining its lightweight architecture. This invention introduces a teacher-student network architecture, which demonstrates significant advantages in model compression, inference acceleration, and efficient deployment in resource-constrained environments, possessing promising application prospects and widespread application value.

[0039] Example 1 This embodiment provides a video anomaly detection method based on semantic modulation and knowledge distillation. It utilizes the multimodal pre-trained model CLIP to extract visual features from video frames and corresponding textual features from the text descriptions. Knowledge transfer is achieved under weak supervision through a teacher-student network structure. While maintaining low annotation costs, this method significantly improves detection accuracy, processing efficiency, and result interpretability in complex monitoring environments. The method steps include: Step 1: Acquire the video stream to be detected, extract the image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame; Step 2: Extract the visual features and textual features of each frame image respectively. Input the visual features and textual features into the teacher network. First, perform temporal modeling on the visual features, calculate the anomaly score on the textual features, and conditionally modulate the visual features under the guidance of the text to form semantically enhanced fusion features. The fusion features are then processed by a classifier to obtain frame-level anomaly scores. Step 3: Input only the visual features into the student network, perform temporal modeling on the visual features, and then input them into the implicit conditional generation network module to generate implicit conditional features, thereby generating modulation parameters and feature modulation amounts, enhancing and fusing the visual features, and the fused features are passed through a classifier to obtain frame-level anomaly scores. Step 4: Construct a distillation loss function to obtain low-dimensional text adaptation features, feature modulation amounts, and prediction results in the teacher network, and use these as distillation supervision signals to perform multi-objective distillation training on the student network. Step 5: The teacher network employs a three-stage training strategy: the first stage focuses on visual branch training, optimizing visual encoding and temporal modeling modules to enable the model to fully learn the spatiotemporal representation capabilities of video sequences; the second stage focuses on text branch training, optimizing text encoding and text anomaly scoring modules to improve the model's ability to model text semantics and its anomaly correlations; the third stage, based on the first two stages, focuses on training cross-modal modulation-related parameters (including text adapters, modulation parameter generation modules, etc.), enabling the model to conditionally modulate and effectively fuse visual features under the guidance of text semantics, thereby forming semantically enhanced fusion features and improving the final anomaly detection and localization performance.

[0040] Step Six: The student network adopts a two-stage training strategy: The first stage focuses on modulation behavior learning. By freezing the classification head and only updating the modulation-related module parameters, the student network can quickly and stably fit the modulation pattern of the teacher network. The second stage unfreezes the data and jointly optimizes the task objective and distillation objective, further improving the anomaly detection and localization performance while maintaining the modulation ability transfer effect.

[0041] Example 2 This embodiment provides a video anomaly detection method based on semantic modulation and knowledge distillation. It constructs a teacher-student network architecture and achieves knowledge transfer under weak supervision through this architecture. Figure 1 As shown, the specific training and implementation process is as follows: Step 1: Obtain the video stream to be detected, extract the image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame; Specifically, the video stream in the scene to be detected is obtained and parsed frame by frame to extract image frames. The pre-trained multimodal large language model BLIP2 is used to generate text descriptions corresponding to the content of each image frame. The generated text description is manually fine-tuned to ensure that the description is consistent with the image content; Step 2: Extract the visual features and textual features of each frame image. Input the visual and textual features into the teacher network. First, perform temporal modeling on the visual features, calculate anomaly scores for the textual features, and then conditionally modulate the visual features under textual guidance to form semantically enhanced fused features. The fused features are then processed by a classifier to obtain frame-level anomaly scores, as follows: The image frames and their corresponding text descriptions are input into the CLIP model to extract the visual features of each image frame and the text features of the corresponding text description. The teacher network of this invention includes six functional modules, specifically: (1) Temporal modeling module, used to perform temporal modeling on visual features extracted by CLIP model; (2) Text anomaly scoring module, used to calculate anomaly scores for text features to characterize the degree of correlation between text semantics and anomalous events; (3) Text adapter, used to perform dimensional transformation and feature mapping on text features to complete the task-oriented adaptation and processing of the original text features; (4) Modulation parameter generation module, which generates parameters (such as scaling factor and offset factor) and gating coefficients for conditional modulation based on the adapted features, so as to realize semantic-guided modulation of visual features. (5) Feature fusion module, used to inject feature modulation amount into visual features under the guidance of text semantics to form semantically enhanced fused feature representation; (6) Frame-level classifier, used to classify and predict fused features and output frame-level anomaly scores.

[0042] As one example, the specific data processing procedure in the teacher network is as follows: Visual features are input to the temporal modeling module for time modeling. The structure of the temporal modeling module is as follows: Figure 2 As shown: First, a corresponding temporal position code is generated for the frame feature sequence. Then, the visual features are fused with the temporal position code, so that the visual features simultaneously contain frame content information and temporal position information. A Transformer temporal encoder is used to capture the local temporal dependencies of the video frame sequence. To further capture global temporal dependencies, a graph convolutional network is introduced to model global temporal dependencies from the perspectives of feature similarity and relative distance. The formula is as follows:

[0043] in, and It's an adjacency matrix; Softmax normalization is used to ensure... and The sum of each row is equal to 1. These are frame-level visual features output by the Transformer temporal encoder. It is the only learnable weight used to transform the feature space.

[0044] The similarity matrix is ​​calculated using cosine similarity. The details are as follows:

[0045] The distance adjacency matrix is ​​calculated based on the positional distance between every two frames. The details are as follows:

[0046] The proximity relationship between the i-th frame and the j-th frame is determined solely by their relative temporal positions. σ is a hyperparameter used to control the range of influence of the distance relationship.

[0047] Text features are input into the text anomaly scoring module to obtain an anomaly score for the text. The structure of the text anomaly scoring module is as follows: Figure 3 As shown: First, the input text features are linearly mapped to learn the correlation between each dimension while keeping the feature dimensions unchanged, so as to enhance the feature expressive power. Then, the features are normalized and nonlinear transformations are introduced to improve the discriminativeness of the representation. Finally, the processed high-dimensional features are mapped to a single scalar output to obtain the text's anomaly score.

[0048] Text features are input in parallel to a text adapter to obtain low-dimensional adapted features. The structure of the text adapter is as follows: Figure 4 As shown: First, the input text features are linearly transformed, and nonlinear activation is introduced to enhance feature representation. Then, the features are dimensionally compressed and mapped to the target output dimension through dimensionality reduction projection, resulting in an adapted low-dimensional feature representation. Since the text features encoded by the CLIP model have a dimension of 512, this embodiment sets the target output dimension to 32, thereby significantly reducing computational overhead and parameter size while preserving key semantic information and improving the efficiency of subsequent modulation parameter generation and cross-modal fusion.

[0049] The modulation parameter generation module utilizes anomaly scores and low-dimensional text adaptation features to obtain modulation parameters (scaling factor, offset factor, and gating weights). A schematic diagram of the teacher network modulation parameter generation module is shown below. Figure 5 As shown: The input frame-level text features are used to calculate the corresponding text anomaly score by the text anomaly scoring module; at the same time, the low-dimensional text adaptation features output by the text adapter are multiplied by two learnable projection matrices to generate scaling and offset factors for conditional modulation.

[0050] scaling factor The formula for generating the formula is as follows:

[0051] Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and U is the trainable parameter matrix.

[0052] offset factor The formula for generating the formula is as follows:

[0053] Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and V is the trainable parameter matrix.

[0054] The basic gating weight g is obtained by transforming the low-dimensional text adaptation features. It is the "basic gating strength" learned from the text content, and the generation formula is as follows:

[0055] Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and gate is a trainable linear transformation layer used to generate gating scores, which are then activated by an activation function to obtain the basic gating weights.

[0056] The gating activation coefficient w driven by anomaly confidence is generated by the following formula:

[0057] Where p is the text anomaly confidence score. It's the set threshold. It is a very small constant.

[0058] The final gating coefficient used for modulation is obtained by multiplying the content-driven basic gating by the text anomaly confidence-driven enable coefficient.

[0059] The feature modulation amount is obtained by using scaling factor, offset factor, and gating weight. The feature modulation amount is then added to the temporally modeled visual features to obtain the fused features. The structure of the feature fusion module is as follows: Figure 6 As shown: The formula for generating the characteristic modulation amount is as follows:

[0060] Where 'a' represents the final gating coefficient. These are normalized visual features. It is a scaling factor. It is the offset factor. This indicates element-wise multiplication.

[0061] The formula for generating fused features is as follows:

[0062] Where X represents the visual features after temporal modeling. Indicates the characteristic modulation amount.

[0063] The fused features are input into the frame-level classifier to obtain frame-level anomaly scores. A schematic diagram of the frame-level classifier is shown below. Figure 7 As shown: The input fusion features are first subjected to nonlinear transformation by a multilayer perceptron to enhance feature representation and extract more discriminative representations. At the same time, a residual structure is introduced to add and fuse the original input features with the output of the multilayer perceptron to preserve the original information and improve training stability. Finally, the fused features are linearly mapped to output the anomaly score for each frame.

[0064] Step 3: Input only visual features into the student network. First, perform temporal modeling on the visual features, then input them into the implicit conditional generation network module to generate implicit conditional features, and then generate modulation parameters and feature modulation amounts to enhance and fuse the visual features. The fused features are then passed through a classifier to obtain frame-level anomaly scores. As one embodiment, the student network includes five functional modules: a temporal modeling module, an implicit conditional generation network module, a modulation parameter generation module, a feature fusion module, and a frame-level classifier module.

[0065] The student network only accepts visual features as input. After performing temporal modeling on the visual features, the data is fed into the implicit conditional generation network module to generate implicit conditional features. The structure of the implicit conditional generation network module is as follows: Figure 8 As shown, apart from the input, the rest of the structure is completely identical to the text adapter in the teacher network. Similar to the teacher network's process, implicit conditional features are processed through two learnable projection matrices to generate scaling and offset factors, and the gated weights are obtained by passing through a fully connected layer and an activation function. The feature fusion module and frame-level classifier are consistent with those in the teacher network.

[0066] Step 4: Construct a distillation loss function to obtain low-dimensional text adaptation features, feature modulation amounts, and prediction results in the teacher network, and use these as distillation supervision signals to perform multi-objective distillation training on the student network. Specifically, such as Figure 9 As shown, in order to further improve the learning effect of student networks under weak supervision, this embodiment designs a multi-path knowledge distillation mechanism, including feature distillation, feature modulation amount distillation and frame-level score distillation.

[0067] Feature distillation: Low-dimensional text adaptation features output by the text adapter in the teacher network and implicit conditional features output by the implicit conditional generation network module in the student network are selected as distillation alignment objects. This allows the student network to learn to generate a conditional feature "equivalent to text semantic driving" without text input, thereby replicating the teacher's semantic enhancement and modulation capabilities. The mean squared error is calculated as the distillation loss, as shown in the following formula:

[0068] in, These are the implicit conditional features output by the implicit conditional generation network module of the student network. Low-dimensional text adaptation features output by the teacher network text adapter.

[0069] Feature modulation distillation: Feature modulation values ​​from the feature fusion module of the teacher network and the feature fusion module of the student network are selected as distillation alignment objects. The "text-driven feature enhancement strategy" of the teacher network is transferred to the student network, enabling the student network to reproduce the same feature modulation behavior even without text. The mean squared error is calculated as the distillation loss, as shown in the following formula:

[0070] in, It is the feature modulation quantity in the student network feature fusion module. It is the feature modulation quantity in the feature fusion module of the teacher network.

[0071] Frame-level score distillation: Using the frame-level anomaly scores output by the teacher network as soft labels, the distribution of the frame-level anomaly scores output by the student network is made consistent with that of the teacher network, without the student network using text input.

[0072] Specifically, the teacher network outputs anomaly scores for each frame. To obtain a smoother supervision signal, a temperature coefficient T is introduced to soften it, and then converted into frame-level soft label probabilities through an activation function, as shown in the following formula:

[0073] The student network outputs anomaly scores for each frame without any text input. Similarly, softening is achieved using a temperature coefficient, as shown in the following formula:

[0074] The binary cross-entropy loss is used to measure the difference between student output and teacher soft labels, as shown in the following formula:

[0075] Among them, multiplied by This is used to compensate for the reduction in gradient magnitude caused by temperature scaling, making distillation training more stable.

[0076] Step 5: The teacher network adopts a three-stage training strategy: The first stage focuses on visual branch training, optimizing the visual encoding and temporal modeling modules to enable the model to fully learn the spatiotemporal representation capabilities of video sequences; the second stage focuses on text branch training, optimizing the text encoding and text anomaly scoring modules to improve the model's ability to model text semantics and its anomaly correlations; the third stage, based on the training in the first two stages, focuses on training cross-modal modulation-related parameters (including text adapters, modulation parameter generation modules, etc.), enabling the model to conditionally modulate and effectively fuse visual features under the guidance of text semantics, thereby forming semantically enhanced fusion features and improving the final anomaly detection and localization performance.

[0077] Step 6: The student network adopts a two-stage training strategy: The first stage focuses on modulation behavior learning. By freezing the classification head and only updating the modulation-related module parameters, the student network can quickly and stably fit the modulation pattern of the teacher network. The second stage unfreezes the data and jointly optimizes the task objective and distillation objective, further improving the anomaly detection and localization performance while maintaining the modulation ability transfer effect.

[0078] Finally, the student network that has been fully trained is used as the anomaly detection model in the real-world scene detection task.

[0079] The video anomaly detection method based on semantic modulation and knowledge distillation proposed in this embodiment has good versatility and scalability, and can be widely applied to anomaly detection tasks in various video surveillance and behavior analysis scenarios, including but not limited to public safety, smart cities, campus and park management, traffic monitoring, and other complex dynamic environments for anomaly event identification.

[0080] Example 3 A video anomaly detection system based on semantic modulation and knowledge distillation includes: The data acquisition module is configured to acquire the video stream to be detected, extract image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame. The anomaly detection module is configured to extract the visual features and textual features of each frame of image respectively. Using the visual features as input, a pre-trained student network is used to perform temporal modeling on the visual features, then generate implicit conditional features, and then generate modulation parameters and feature modulation amounts to enhance and fuse the visual features. The fused features are then passed through a classifier to obtain frame-level anomaly scores. The multi-objective distillation training module is configured such that the student network undergoes multi-objective distillation training in advance through the teacher network during the pre-training process. The teacher network takes visual features and text features as input, performs temporal modeling on the visual features, calculates anomaly scores on the text features, and conditionally modulates the visual features under text guidance to form semantically enhanced fusion features. The fusion features are passed through a classifier to obtain frame-level anomaly scores. The text adaptation features, feature modulation amount, and prediction results in the teacher network are used as distillation supervision signals in the multi-objective distillation training.

[0081] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0082] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video anomaly detection method based on semantic modulation and knowledge distillation, characterized in that, Includes the following steps: Acquire the video stream to be detected, extract the image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame; Visual features and textual features of each frame are extracted separately. Using the visual features as input, a pre-trained student network is used to perform temporal modeling on the visual features, and then implicit conditional features are generated. In turn, modulation parameters and feature modulation amounts are generated to enhance and fuse the visual features. The fused features are then passed through a classifier to obtain frame-level anomaly scores. During the pre-training process, the student network is pre-trained by the teacher network through multi-objective distillation. The teacher network takes visual features and text features as input, performs temporal modeling on the visual features, calculates anomaly scores on the text features, and conditionally modulates the visual features under text guidance to form semantically enhanced fusion features. The fusion features are then processed by a classifier to obtain frame-level anomaly scores. The text adaptation features, feature modulation amounts, and prediction results in the teacher network are used as distillation supervision signals in the multi-objective distillation training.

2. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 1, characterized in that, The process of extracting image frames from the video stream to be detected includes acquiring the video stream of the scene to be detected, parsing it frame by frame, extracting image frames, and using a pre-trained multimodal large language model to generate text descriptions corresponding to the content of each image frame. The process of extracting the visual features of each frame of the image and the text features of the text description includes: inputting the image frame and its corresponding text description into the pre-trained visual-language multimodal pre-trained model, and extracting the visual features of each frame of the image and the text features of the corresponding text description.

3. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 1, characterized in that, The teacher network includes: The temporal modeling module is used to perform temporal modeling on the extracted visual features; The text anomaly scoring module is used to calculate anomaly scores for text features to characterize the degree of correlation between text semantics and anomalous events. Text adapters are used to perform dimensional transformation and feature mapping on text features to complete the task-oriented adaptation and processing of the original text features. The modulation parameter generation module generates parameters and gating coefficients for conditional modulation based on the adapted features, so as to achieve semantically guided modulation of visual features. The feature fusion module is used to inject feature modulation amounts into visual features under the guidance of text semantics to form a semantically enhanced fused feature representation; A frame-level classifier is used to classify and predict fused features, and output frame-level anomaly scores.

4. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 3, characterized in that, In the teacher network, the process by which the modulation parameter generation module generates parameters for conditional modulation and gating coefficients based on the adapted features includes: scaling factor Generate as: Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and U is the trainable parameter matrix; offset factor Generate as: Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and V is the trainable parameter matrix; The basic gating weight g is obtained by transforming the low-dimensional text adaptation features, and the generation formula is: Where C is the low-dimensional text adaptation feature matrix output by the text adapter, and gate is a trainable linear transformation layer used to generate gate scores, which are then activated by an activation function to obtain the basic gate weights. The gating activation coefficient w driven by anomaly confidence is generated by the following formula: Where p is the text anomaly confidence score. It's the set threshold. It is a constant less than the set value; the final gating coefficient used for modulation is obtained by multiplying the content-driven basic gating by the text anomaly confidence-driven activation coefficient.

5. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 1, characterized in that, The student network includes a temporal modeling module, an implicit conditional generation network module, a modulation parameter generation module, a feature fusion module, and a frame-level classifier module. The student network only receives visual features as input. The temporal modeling module performs temporal modeling on the visual features and then inputs them into the implicit conditional generation network module to generate implicit conditional features. After the implicit conditional features pass through two learnable projection matrices, the modulation parameter generation module generates scaling factors and offset factors. The gate weights are obtained by passing through a fully connected layer and an activation function.

6. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 1, characterized in that, The multi-objective distillation training utilizes a distillation loss function, specifically including feature distillation, feature modulation amount distillation, and frame-level score distillation, wherein: Feature distillation selects low-dimensional text adaptation features output by the text adapter in the teacher network and implicit conditional features output by the implicit conditional generation network module in the student network as distillation alignment objects. Feature modulation amount distillation selects the feature modulation amounts in the feature fusion module of the teacher network and the feature modulation amounts in the feature fusion module of the student network as distillation alignment objects; Frame-level score distillation uses the frame-level anomaly scores output by the teacher network as soft labels, ensuring that the distribution of the frame-level anomaly scores output by the student network is consistent with that of the teacher, provided that the student network does not use text input.

7. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 6, characterized in that, In frame-level score distillation, the teacher network outputs anomaly scores for each frame, softens them with a temperature coefficient, and converts them into frame-level soft label probabilities through an activation function; the student network outputs anomaly scores for each frame without text input, and also softens them with a temperature coefficient. A binary cross-entropy loss is used to measure the difference between student output and teacher soft label, and the square of the temperature coefficient is used to compensate for the reduction in gradient magnitude caused by temperature scaling.

8. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 1, characterized in that, The teacher network adopts a three-stage training strategy. The first stage focuses on visual branch training, which optimizes the visual encoding and temporal modeling modules to enable the model to fully learn the spatiotemporal representation of video sequences. The second stage focuses on text branch training, improving the model's ability to model text semantics and its anomaly correlations by optimizing text encoding and text anomaly scoring modules. The third stage, based on the training in the first two stages, focuses on training cross-modal modulation parameters, enabling the model to conditionally modulate and effectively fuse visual features under the guidance of text semantics, forming semantically enhanced fusion features and improving the final anomaly detection and localization performance.

9. The video anomaly detection method based on semantic modulation and knowledge distillation as described in claim 1, characterized in that, The student network adopts a two-stage training strategy. The first stage focuses on learning modulation behavior by freezing the classification head and updating only the modulation-related module parameters, enabling the student network to quickly and stably fit the modulation pattern of the teacher network. The second stage unfreezes the data based on the first stage and jointly optimizes the task objective and the distillation objective.

10. A video anomaly detection system based on semantic modulation and knowledge distillation, characterized in that, include: The data acquisition module is configured to acquire the video stream to be detected, extract image frames from the video stream to be detected, and generate a text description corresponding to the image content based on each image frame. The anomaly detection module is configured to extract the visual features and textual features of each frame of image respectively. Using the visual features as input, a pre-trained student network is used to perform temporal modeling on the visual features, then generate implicit conditional features, and then generate modulation parameters and feature modulation amounts to enhance and fuse the visual features. The fused features are then passed through a classifier to obtain frame-level anomaly scores. The multi-objective distillation training module is configured such that the student network undergoes multi-objective distillation training in advance through the teacher network during the pre-training process. The teacher network takes visual features and text features as input, performs temporal modeling on the visual features, calculates anomaly scores on the text features, and conditionally modulates the visual features under text guidance to form semantically enhanced fusion features. The fusion features are passed through a classifier to obtain frame-level anomaly scores. The text adaptation features, feature modulation amount, and prediction results in the teacher network are used as distillation supervision signals in the multi-objective distillation training.

Images