A video detection method and system based on prompt calibration and continuous learning

Abstract

The application discloses a video detection method and system based on prompt calibration and continuous learning, and relates to the technical fields of multimedia content security, artificial intelligence and computer vision. A non-replay continuous learning framework is constructed with a prompt pool and a dynamic category prototype as the core, the historical knowledge of the illegal video is continuously retained and updated, so that the storage cost is effectively reduced, and the privacy and compliance risks caused by the retention of sensitive data are reduced. Through the synergistic effect of the prompt calibration mechanism and the dynamic prototype constraint mechanism, the model can learn the characteristics of new illegal video variants while maintaining the stable recognition ability of the historical category boundary, effectively alleviating the catastrophic forgetting problem in the incremental training scene. An uncertainty weighting fusion mechanism based on information entropy is introduced, which can adaptively suppress the adverse effects of low-confidence modalities on the final determination result in complex scenes such as multi-modal information inconsistency, thereby improving the overall robustness and practical deployment applicability of the illegal video detection system.

Description

Technical Field

[0001] This invention relates to the fields of multimedia content security, artificial intelligence and computer vision, and to automated review and analysis of illegal videos. In particular, it proposes a video detection method and system based on prompt calibration (PC) and continuous learning (CL) to address the evolution bottleneck of deep learning systems under incremental data streams. Background Technology

[0002] In recent years, with the rapid development of short video and streaming media platforms, video content has become an important carrier of internet information dissemination. Existing rule-based or static deep learning-based automated moderation systems typically rely on centralized training with fixed-size datasets and then deploy them as online detection models after training.

[0003] However, in the complex and ever-changing real-world network environment, the forms of illegal videos continuously evolve with changing social trends, updated camouflage methods, and evolving evasion strategies, exhibiting a clear concept drift characteristic. When faced with these new variants of violations that constantly emerge in the form of incremental data streams, traditional static detection models are prone to declining recognition capabilities and increasing false positive rates. While directly using newly added illegal video data to fine-tune the original classification model can improve the model's ability to recognize the current new variants to some extent, it also disrupts the original parameter distribution, causing the model to gradually lose its ability to distinguish historical violation patterns and existing classification boundaries, thus resulting in a serious "catastrophic forgetting" problem.

[0004] To address the aforementioned issues, existing Continuous Learning (CL) methods typically employ the construction of historical sample libraries or feature caches, using Experience Replay to retain and retrain old knowledge. However, in practical applications involving the review of massive amounts of video content, long-term storage of original illegal videos containing sensitive information or their high-dimensional features not only incurs significant storage and computational burdens but may also raise risks related to privacy protection and data compliance. Furthermore, existing methods generally lack the ability to dynamically adjust and stably transfer historical experience when faced with continuously evolving illegal video patterns. Therefore, how to achieve continuous modeling and stable identification of the evolutionary patterns of illegal videos without heavily relying on traditional replay mechanisms has become a critical technical problem urgently needing to be solved in the field of multimedia content security. Summary of the Invention

[0005] The purpose of this invention is to overcome the "catastrophic forgetting" problem faced by existing illegal video detection methods in incremental data stream scenarios, as well as the storage, computation, and compliance burdens brought about by historical sample playback mechanisms. It provides an illegal video detection method and system based on cue calibration and continuous learning. This method constructs a cue pool based on historical evolutionary experience and combines it with a continuous update mechanism of dynamic category prototypes. Without heavily relying on the storage of historical original video samples, it achieves continuous modeling of the evolutionary patterns of illegal videos, preservation of historical knowledge, and stable identification of new variant content.

[0006] The core idea of ​​this invention is as follows: First, the multimodal information of the input violation video, including visual, audio, and textual data, is mapped to a unified latent representation space to form an initial video representation. Then, a cue pool is constructed in conjunction with the backbone feature extraction network to store cue information that can represent different historical stages and different violation variants. A query mechanism is used to retrieve the most relevant historical experience from the cue pool to calibrate the current video features. Based on this, dynamic prototypes corresponding to each category are further constructed. The category center representation is continuously maintained through momentum updates, and prototype consistency constraints are used to enhance the model's ability to preserve historical category boundaries. Finally, an uncertainty assessment mechanism based on information entropy is introduced in the classification and determination stage to adaptively weight and fuse the outputs of different modalities, thereby improving the system's detection robustness and continuous recognition capability in complex scenarios.

[0007] Based on the above inventive concept, this invention proposes a video detection method based on cue calibration and continuous learning, which includes the following steps:

[0008] Step S1: For the input target video, extract its text modality, visual modality and audio modality respectively, and obtain the corresponding modal feature representation by the pre-trained encoder of each modality. Then, project the different modal features into a unified latent representation space through the modality mapping layer. Finally, use a preset fusion function to fuse the modal representations to obtain the initial video representation.

[0009] Step S2: Construct a learnable cue pool to store abstract cue information of inappropriate videos at different time stages and under different evolutionary modes; based on the cue pool, perform query mapping on the initial video representation to obtain a query vector for cue retrieval; calculate the matching degree between the query vector and each cue key in the cue pool, and select the top few cue items most relevant to the current input based on the matching results; aggregate the cue representations corresponding to the selected cue items to form a historical cue representation adapted to the current video features; use an attention mechanism to interact with the initial video representation and the historical cue representation to generate sample cue after cue calibration.

[0010] Step S3: For each category in the violation video detection task, maintain the corresponding dynamic category prototype in the unified latent representation space; during continuous learning, based on the sample representation at the current stage, incrementally adjust the prototype of each category using the momentum update method; introduce prototype consistency constraint loss during training to encourage the sample representation to move closer to its category prototype and maintain distinction from other category prototypes.

[0011] Step S4: Based on the classification outputs of text modality, visual modality, and audio modality, calculate their category probability distributions and corresponding information entropy respectively; and generate adaptive fusion weights based on uncertainty to perform weighted fusion of the feature representations of each modality, and then output the final detection result of the target video through the classification layer.

[0012] Step S2 is as follows:

[0013] Build a learnable cue pool It stores abstract prompts for illegal videos at different time stages and under different evolutionary patterns, and its structure is defined as follows:

[0014]

[0015] in, Indicates the total number of prompts. Indicates the first A prompt button, This represents the sequence of prompt values ​​corresponding to the prompt key;

[0016] Initial video representation Perform query mapping to obtain query vectors for suggestive retrieval. ,in Indicates the query mapping function;

[0017] Then, the query vector is obtained using cosine similarity calculation. With all the prompt keys The correlation coefficient scores are used to select the top-ranked correlation coefficient scores. Each prompt item is identified, and its prompt values ​​are aggregated to obtain a historical prompt representation. ;

[0018] Subsequently, an attention mechanism is used to interact with the initial video representation and the historical cue representation to generate a cue-enhanced representation that includes historical experience calibration information. And generate a calibrated sample representation using residuals:

[0019]

[0020] in, This represents the final generated sample.

[0021] Step S3 is as follows:

[0022] For each category in the illegal video detection task, a corresponding dynamic category prototype is maintained in the unified latent representation space. C represents the number of categories; for the current batch, those belonging to category... sample set The calibrated sample represents the result as indicated by its prompts. Calculate the within-class mean and incrementally adjust the class prototype using momentum updates:

[0023]

[0024] in, The momentum coefficient;

[0025] Introducing prototype consistency constraint loss during training Its definition is:

[0026]

[0027] in, Indicates sample The true category label, Indicates sample Belongs to the real category The corresponding category prototype vector, This refers to the temperature parameter.

[0028] Step S4 is as follows:

[0029] Let the mode The classification output logic value is ,in, , For modal indexing, These represent visual, audio, and text modalities, respectively. Let m be the classification weight matrix corresponding to mode m. The two are bias vectors, and together they constitute the mode. The classification layer parameters are used to represent the samples. Mapping to classification logic The corresponding category probability distribution is The uncertainty of this mode is then expressed using Shannon information entropy. This is represented; then, modal fusion weights are generated based on the uncertainty:

[0030]

[0031] The modal feature representations are weighted and fused based on modal fusion weights to obtain the final fused representation. Finally, the detection results of the target video are output through the classification layer. :

[0032]

[0033] in, and These represent the weight parameters and bias parameters of the classification layer, respectively.

[0034] Furthermore, this invention also discloses a video detection system based on cue calibration and continuous learning for performing the above-described method, comprising:

[0035] Multimodal feature extraction module: used to receive target video data from video platform, and encode, extract and uniformly represent text, audio and visual information to generate initial video representation;

[0036] The prompt calibration interaction module is used to build and maintain the prompt pool, generate query vectors based on input samples, and complete prompt retrieval, prompt aggregation, and prompt enhancement interaction to realize the prompt calibration of current features based on historical experience.

[0037] Dynamic Prototype Update Module: This module is used to maintain the dynamic category prototypes corresponding to each violation category in the latent representation space, and to perform momentum updates on the category prototypes based on the representation results of the samples at the current stage during continuous learning, while applying prototype consistency constraints.

[0038] Uncertainty fusion judgment module: used to calculate the uncertainty of the output results of each mode, generate mode fusion weights, and output the final violation video detection result based on the fused sample representation.

[0039] Compared with the prior art, the present invention has at least the following beneficial effects:

[0040] (1) This invention constructs a continuous learning framework without replay, with a prompt pool and dynamic category prototype as the core. It can continuously retain and update the historical knowledge of illegal videos without long-term storage of a large number of original samples or high-dimensional features of historical illegal videos, thereby effectively reducing storage overhead and reducing privacy and compliance risks caused by the retention of sensitive data.

[0041] (2) By combining the prompting calibration mechanism and the dynamic prototype constraint mechanism, the present invention enables the model to maintain a stable ability to identify historical category boundaries while learning the features of new illegal video variants, thereby effectively alleviating the catastrophic forgetting problem in incremental training scenarios and improving the stability and reliability of continuous detection.

[0042] (3) The present invention further introduces an uncertainty-weighted fusion mechanism based on information entropy, which can adaptively suppress the adverse effects of low confidence modalities on the final judgment result in complex scenarios such as missing text, audio interference by noise, visual content disguise, or inconsistency of multimodal information, thereby improving the overall robustness and practical deployment applicability of the illegal video detection system. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the overall process framework of a violation video detection method and system based on prompt calibration and continuous learning proposed in this invention. Detailed Implementation

[0044] This section will combine specific steps and relevant mathematical theories to provide a detailed analysis and in-depth explanation of the application process and structure of this invention.

[0045] Explanation of terms used in this invention:

[0046] Multimodal Feature Fusion (MFE) refers to a theory that, when faced with composite file content containing text, images (visual spatial images), and audio, overcomes the limitations of a single modal perspective by mapping, cross-aligning, or fusion and weighting information from multiple modalities through a unified architecture, thereby achieving more comprehensive and accurate recognition.

[0047] Concept drift refers to the phenomenon where the relationship between the input data distribution and the target concept changes over time in data stream or temporal learning scenarios. In illegal video detection tasks, concept drift typically manifests as the continuous evolution of the representation, camouflage methods, and propagation patterns of illegal content, leading to a decline in the performance of the original static model.

[0048] Catastrophic forgetting refers to a phenomenon in artificial neural networks where, during the learning process of a model on new task data with new feature distributions, the weights of the original layers of the neural network parameters are modified and overwritten on a large scale, causing a sharp collapse and decline in the detection accuracy that had been acquired and solidified in previous old model tasks.

[0049] Continual Learning: A branch of the concept in machine learning and neural networks; it refers to a model development paradigm that aims to enable systems to continuously and incrementally acquire new skills and experience as they are constantly fed with various new tasks (data streams with different distributions of time series, new judgment benchmarks, etc.), and strives to minimize the sharp loss of performance of the ability to adapt to the early knowledge patterns due to the adaptation to "new knowledge" (i.e., the aforementioned "catastrophic forgetting" phenomenon).

[0050] Prompt-Calibrated: This term refers to a technical approach where a set of abstract "instructions" or "learning experience feature embeddings" (i.e., prompt embeddings) stored in the model are preset. The system dynamically extracts historical alignment content that conforms to the rules by perceiving the correlation between the current input target features and the features in various historical experience pools, and uses this information to correct the current feature state representation, thereby guiding and adjusting the network to adapt to learning.

[0051] The prompt pool is a storage structure consisting of multiple learnable prompts. Each prompt typically contains a prompt key for matching the query and a corresponding prompt value. The system maps the current input to a query vector and calculates its similarity with the prompt keys in the prompt pool, retrieving the prompt most relevant to the current sample to dynamically invoke historical experience and calibrate features.

[0052] Attention mechanism: This refers to a modeling mechanism that assigns different weights to different information units by calculating the correlation between input representations, thereby highlighting the parts more relevant to the current task. In this invention, the attention mechanism is used to interactively model the retrieved historical cue representations and the current sample representation to generate enhanced features after cue calibration.

[0053] Dynamic Class Prototypes: These refer to maintaining a representative central representation parameter for each class in the feature space, rather than storing all historical samples of that class. This central representation can be continuously adjusted through mean updates or momentum updates as new samples arrive, thus characterizing the stable semantic center of the class during its temporal evolution.

[0054] Shannon entropy is a fundamental information theory quantity used to measure the uncertainty of random variables. In classification tasks, the information entropy can be calculated using the class probability distribution of the model output to measure the degree of uncertainty of the corresponding modality's prediction result; the larger the entropy value, the more uncertain the modality's output. In this invention, Shannon entropy is used to generate modality fusion weights to suppress the interference of low-confidence modalities on the final judgment result.

[0055] Example 1

[0056] This embodiment provides a specific application process of a violation video detection method based on cue calibration and continuous learning in the identification of hate speech videos. The method is applicable to violation video detection tasks that include multimodal information such as text, vision, and audio. This embodiment includes the following four steps:

[0057] S1 Multimodal Feature Extraction and Initial Representation Construction: For the input target video, its visual frame sequence, audio signal, and text content are extracted respectively, and the corresponding modal feature representations are obtained using pre-trained visual encoders, audio encoders, and text encoders. Further, different modal features are projected onto a unified latent representation space through a modality mapping layer to obtain each modal representation. Subsequently, a preset fusion function is used to fuse the modal representations to obtain the initial video representation. In this embodiment, the initial video representation is used to characterize the basic semantic information of the current video sample and serves as input for subsequent cue calibration and continuous learning processing.

[0058] Specifically, let the first The text, visual, and audio inputs for each video sample are as follows: , and The visual encoder, audio encoder, and text encoder are respectively denoted as... , and Then, the modal features can be expressed as:

[0059]

[0060] in, , , Let these represent visual modal features, audio modal features, and text modal features, respectively. A modality mapping layer maps these features to a latent representation space of a unified dimension, resulting in:

[0061]

[0062] in, and These represent the mapping parameters for the corresponding modes. For modal indexing, These represent visual, audio, and text modalities, respectively. Then, a fusion function is used. The modal representations are fused to obtain the initial video representation vector:

[0063]

[0064] The fusion function This can be achieved using a weighted average summation.

[0065] S2 Enhanced Cue Retrieval and Cue Calibration: Based on the pre-set cue pool in the system, the initial video representation is improved. Perform query mapping to obtain query vectors for suggestive retrieval. ,in The query mapping function, which can be a linear transformation or projection operation, provides a basis for matching subsequent historical prompts. The matching degree between the query vector and each prompt key in the prompt pool is calculated, and the top few prompts most relevant to the current input are selected based on the matching results. The prompt representations corresponding to the selected prompts are aggregated to form historical prompt representations adapted to the current video features.

[0066] Specifically, a learnable prompt pool is constructed to store abstract prompt information from violation videos at different time stages and under different evolutionary patterns. Its structure is defined as follows:

[0067]

[0068] in, Indicates the total number of prompts. Indicates the first A prompt key. This represents the sequence of prompt values ​​corresponding to the prompt key. This represents the number of suggestion vectors contained in a single suggestion item. Then, the query vector is obtained using cosine similarity calculation. With all the prompt keys correlation coefficient score Evaluate whether the historical dynamic logic included in the prompt should be correct:

[0069]

[0070] in, This represents taking the modulus of the vector. The top-ranked vectors with the highest correlation coefficient scores are selected. Each prompt item is identified, and its prompt values ​​are aggregated to obtain a historical prompt representation. :

[0071]

[0072] Subsequently, an attention mechanism is used to interact the initial video representation with the historical cue representation, allowing the retrieved historical cue information to guide and compensate for the current input features, generating a cue enhancement representation that includes historical experience calibration information. , ,here This refers to a multi-head attention computation method that uses video representation features as the query and cue embeddings as the key and value; and generates cue-calibrated sample representations through residual methods.

[0073]

[0074] Through the above processing, the current input sample can retain its own semantic information while combining historical evolution experience to complete feature calibration, thereby enhancing the model's adaptability to new illegal variants.

[0075] S3 Dynamic Category Prototype Update and Continuous Learning Constraints: For each category in the violation video detection task, a corresponding dynamic category prototype is maintained in a unified latent representation space. C represents the number of categories. For the current batch, those belonging to category... sample set The calibrated sample represents the result as indicated by its prompts. Calculate the within-class mean and use momentum updates to incrementally adjust the class prototypes to maintain the continuous evolution of class centers without replaying historical original samples:

[0076]

[0077] in, The momentum coefficient is used to control the balance between historical prototype information and the current stage category representation. Through this method, the system can continuously track and update the semantic center of the category using dynamic category prototypes without replaying historical original video data, thereby maintaining the model's stable ability to recognize historical categories.

[0078] Furthermore, to maintain the model's stable memory of historical category knowledge, a prototype consistency constraint loss is introduced during training. This is to encourage sample representations to move closer to their class prototypes and to maintain distinction from other class prototypes, and is defined as follows:

[0079]

[0080] in, Indicates sample The true category label, Indicates sample Belongs to the real category The corresponding category prototype vector, The temperature parameter is used. Through dynamic category prototypes and prototype consistency constraints, the model can continuously maintain historical category boundaries without storing a large number of historical original video samples, thus mitigating the catastrophic forgetting problem in incremental learning.

[0081] S4 Uncertainty-Based Multimodal Fusion Decision: Based on the classification outputs of text, visual, and audio modalities, the probability distribution of each category and its corresponding information entropy are calculated to characterize the uncertainty of the prediction results for each modality. Adaptive fusion weights are then generated based on the uncertainty to suppress the interference of low-confidence modalities on the overall judgment, outputting the final detection result of the target video.

[0082] Specifically, let's define the modal. The classification output logic value is ,in, Let m be the classification weight matrix corresponding to mode m. The two are bias vectors, and together they constitute the mode. The classification layer parameters are used to classify sample features. Mapping to classification logic Its corresponding category probability distribution is: The uncertainty of this mode is expressed using Shannon information entropy as follows:

[0083]

[0084] Generate modal fusion weights based on uncertainty:

[0085]

[0086] Then, the feature representations of each modality are weighted and fused to obtain the final fused representation:

[0087]

[0088] Finally, the detection results of the target video are output through the classification layer. :

[0089]

[0090] in, and These represent the weight parameters and bias parameters of the classification layer, respectively.

[0091] In this embodiment, when the video has weak text modality representation, audio modality noise interference, or visual modality camouflage, the system can reduce the impact of low-confidence modalities through an uncertainty weighting mechanism. This can effectively suppress the interference of low-confidence modalities on the overall judgment result and improve the detection robustness in complex scenarios.

[0092] Example 2

[0093] This embodiment provides a system architecture for implementing the aforementioned illegal video detection method. The system is built around two core mechanisms: cue calibration and continuous learning, and includes the following four functional modules:

[0094] The multimodal feature extraction module is used to receive target video data, extract features and perform unified representation mapping on text information, visual frame information and audio information in the video, and output an initial video representation. This module corresponds to step S1 in Embodiment 1.

[0095] The prompt calibration interaction module is used to maintain the prompt pool structure, generate query vectors based on the initial video representation, complete prompt key matching, prompt value aggregation, and prompt enhancement interaction, and output a sample representation calibrated based on historical experience. This module corresponds to step S2 in Embodiment 1 and is the core module for implementing prompt calibration.

[0096] The dynamic prototype update module maintains dynamic category prototypes corresponding to each violation category in a unified latent representation space and performs momentum updates on the category prototypes based on the cues from the current stage samples. Simultaneously, it calculates the prototype consistency constraint loss during the training phase to enhance the model's ability to preserve historical category knowledge. This module corresponds to step S3 in Embodiment 1 and is the core module for achieving continuous learning.

[0097] The uncertainty fusion determination module is used to calculate the uncertainty of the output results of different modes, generate mode fusion weights, perform weighted fusion of each mode representation, and output the final detection result of the target video. This module corresponds to step S4 in Embodiment 1.

[0098] In this embodiment, the four modules described above can be deployed on the same server or distributed computing platform to run collaboratively. Specifically, the multimodal feature extraction module provides a unified input representation; the prompting and calibration interaction module calibrates and enhances the current sample based on historical prompts; the dynamic prototype update module continuously retains category knowledge without relying on the playback of historical original samples; and the uncertainty fusion and determination module makes robust decisions on the final output. Through this system structure, stable and efficient continuous detection can be achieved in scenarios where illegal video content continues to evolve.

[0099] Application examples

[0100] The violation video detection method and system based on cue calibration and continuous learning provided in the embodiments are used to verify the continuous learning prediction effect on real video datasets. The datasets used include HateMM (containing 1083 videos from BitChute and Odysee, covering English multimodal violation text, visual and audio information), MultiHateClip-YouTube (MHClip-A, covering multilingual short videos on the YouTube platform), and MultiHateClip-Bilibili (MHClip-B, covering Chinese short videos on the Bilibili platform, dataset source is the same as above).

[0101] In this continuous learning evaluation process, the model is sequentially incrementally trained on three datasets, following the actual scenario where video content continuously evolves over time and across platforms. This simulates the continuous migration process of infringing videos across different platforms and representations: Task 1 (HateMM) → Task 2 (MHClip-A) → Task 3 (MHClip-B). During the learning of subsequent tasks, the system does not access the original text, image, and audio data from previous tasks to verify the effectiveness of this invention in mitigating catastrophic forgetting and maintaining historical recognition capabilities under conditions without historical sample playback.

[0102] Meanwhile, the method and system provided by this invention are compared and evaluated with several continuous learning baseline models. The experiments use average accuracy (Average Accuracy ↑) and average forgetting rate (Average Forgetting ↓) as evaluation metrics. A higher average accuracy indicates better overall detection performance after multiple rounds of incremental learning, while a lower average forgetting rate indicates stronger ability to retain historical task knowledge. The final performance comparison results of different methods in continuous multi-task video stream scenarios are shown in Table 1.

[0103] Table 1: Performance comparison in sequence continuous learning application scenarios.

[0104] ER 72.3 69.8 70.4 70.8 14.1 LwF 71.4 67.8 69.1 69.4 13.3 DER++ 74.7 70.5 72.7 72.6 12.2 L2P 75.2 71.3 73.8 73.4 10.4 DualPrompt 76.3 72.0 74.9 74.4 9.6 CODA-Prompt 77.6 73.7 74.2 75.2 9.3 CoLeCLIP 79.4 75.2 77.6 77.4 7.8 DST 79.0 74.5 75.2 76.2 5.6 DKR 78.3 73.9 74.5 75.6 4.9 This invention 82.8 78.5 79.8 80.3 2.7

[0105] The representative comparison feature method in the table is introduced as follows:

[0106] ER: The traditional experience replay mechanism maintains old knowledge by simply storing a small number of previous samples for retraining, but its generalization performance is generally poor due to the limitation of storage amount and inability to effectively cover the long tail of the distribution.

[0107] LwF: A non-replay method based on knowledge distillation, which relies on the mapping of old features output by the old model in the previous distillation stage to mitigate overwriting and forgetting.

[0108] DER++: It belongs to the deep replay network that optimizes the traditional replay strategy by superimposing a dark knowledge output matching mechanism.

[0109] L2P: A pioneering method that uses a cue pool technique to dynamically select cue items across different tasks to guide continuous feature learning without requiring major fine-tuning of the entire network.

[0110] DualPrompt is a decoupled explicit cueing learning method proposed for continuous learning. By separating the computation of task-invariant instructions and task-specific instructions, it achieves better forward transfer and resistance to catastrophic forgetting.

[0111] CODA-Prompt: An improved cue-based continuous learning model that uses key-value attention computation to guide feature generation and reconstruction of continuously evolving data.

[0112] CoLeCLIP: A method for adapting the capabilities of large-scale pre-trained visual-language models to the domain of continuous incremental training of specific downstream vocabularies.

[0113] DST: A cutting-edge approach that uses a decoupled self-updating tree combined with a multimodal model to address distribution drift in continuous learning.

[0114] DKR: A new baseline specifically designed to coordinate and resist interference in learning the latest class evolution by introducing a dual knowledge routing mechanism.

[0115] As shown in Table 1, the experimental results demonstrate that the violation video detection method and system based on cue calibration and continuous learning provided by this invention achieved an average accuracy of 80.3% after completing three incremental tasks in sequence, outperforming the baseline continuous learning method in overall performance. Particularly noteworthy is the average forgetting rate (AFR), a metric measuring the model's ability to retain historical knowledge, which is only 2.7% for this invention, significantly lower than other comparative methods such as the DKR model's 4.9% and CODA-Prompt's 9.3%. These results indicate that, when facing scenarios where violation video content evolves over time, this invention can effectively mitigate the problem of catastrophic forgetting by learning new tasks while maintaining a good ability to recognize historical tasks.

[0116] Further analysis reveals that the method of this invention achieves a low forgetting rate and high overall accuracy without relying on the playback of historical original video data. This is mainly attributed to the synergistic effect of the cue calibration mechanism and the dynamic category prototype update mechanism. Specifically, the cue calibration mechanism compensates and calibrates the current feature representation by retrieving historical cue information related to the current sample from the cue pool, thereby enhancing the model's adaptability to new violation variants. The dynamic category prototype update mechanism, on the other hand, continuously maintains the representation of each category center, enabling the model to maintain a relatively stable category discrimination boundary during incremental training. Thus, even without the constraint of historical sample playback, it is still possible to effectively retain historical knowledge and continuously learn new knowledge.

[0117] Furthermore, in complex multimodal scenarios, such as when textual information is missing, audio signals are interfered with by noise, or visual content is disguised or obfuscated, the uncertainty fusion judgment mechanism introduced in this invention can adaptively adjust the weight of each modality in the final decision based on the uncertainty of the output results of different modalities, thereby reducing the adverse impact of low-confidence modalities on the overall detection results. Therefore, the method of this invention still exhibits good detection robustness and stability when facing complex scenarios such as obscure expressions, modal inconsistencies, and cross-platform distribution variations.

[0118] In summary, the violation video detection method and system proposed in this invention, based on prompt calibration and continuous learning, can simultaneously take into account the ability to adapt to new knowledge and the ability to retain historical knowledge in incremental scenarios where violation video content continues to evolve. It is suitable for automatic video review and security governance tasks for social media platforms, and provides a technical solution that balances detection performance, storage overhead, and deployment feasibility for the continuous identification of abnormal violation videos and their variants.

[0119] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A video detection method based on cue calibration and continuous learning, characterized in that, Includes the following steps: Step S1: For the input target video, extract its text modality, visual modality and audio modality respectively, and obtain the corresponding modal feature representation by the pre-trained encoder of each modality. Then, project the different modal features into a unified latent representation space through the modality mapping layer. Finally, use a preset fusion function to fuse the modal representations to obtain the initial video representation. Step S2: Construct a learnable cue pool to store abstract cue information of violation videos at different time stages and under different evolution patterns; based on the cue pool, perform query mapping on the initial video representation to obtain query vectors for cue retrieval; Calculate the matching degree between the query vector and each prompt key in the prompt pool, and select the top few prompts most relevant to the current input based on the matching results; aggregate the prompt representations corresponding to the selected prompts to form historical prompt representations adapted to the current video features; use an attention mechanism to interact with the initial video representation and the historical prompt representations to generate sample prompts after prompt calibration. Step S3: For each category in the violation video detection task, maintain the corresponding dynamic category prototype in the unified latent representation space; during continuous learning, based on the sample representation at the current stage, incrementally adjust the prototype of each category using the momentum update method; introduce prototype consistency constraint loss during training to encourage the sample representation to move closer to its category prototype and maintain distinction from other category prototypes. Step S4: Based on the classification outputs of text modality, visual modality, and audio modality, calculate their category probability distributions and corresponding information entropy respectively; and generate adaptive fusion weights based on uncertainty to perform weighted fusion of the feature representations of each modality, and then output the final detection result of the target video through the classification layer.

2. The video detection method based on cue calibration and continuous learning according to claim 1, characterized in that, Step S2 is as follows: Build a learnable cue pool It stores abstract prompts for illegal videos at different time stages and under different evolutionary patterns, and its structure is defined as follows: in, Indicates the total number of prompts. Indicates the first A prompt button, This represents the sequence of prompt values ​​corresponding to the prompt key; Initial video representation Perform query mapping to obtain query vectors for suggestive retrieval. ,in Indicates the query mapping function; Then, the query vector is obtained using cosine similarity calculation. With all the prompt keys The correlation coefficient scores are used to select the top-ranked correlation coefficient scores. Each prompt item is identified, and its prompt values ​​are aggregated to obtain a historical prompt representation. ; Subsequently, an attention mechanism is used to interact with the initial video representation and the historical cue representation to generate a cue-enhanced representation that includes historical experience calibration information. And generate a calibrated sample representation using residuals: in, This represents the final generated sample.

3. The video detection method based on cue calibration and continuous learning according to claim 2, characterized in that, Step S3 is as follows: For each category in the illegal video detection task, a corresponding dynamic category prototype is maintained in the unified latent representation space. C represents the number of categories; for the current batch, those belonging to category... sample set The calibrated sample represents the result as indicated by its prompts. Calculate the within-class mean and incrementally adjust the class prototype using momentum updates: in, The momentum coefficient; Introducing prototype consistency constraint loss during training Its definition is: in, Indicates sample The true category label, Indicates sample Belongs to the real category The corresponding category prototype vector, This refers to the temperature parameter.

4. The video detection method based on cue calibration and continuous learning according to claim 3, characterized in that, Step S4 is as follows: Let the mode The classification output logic value is ,in, , For modal indexing, These represent visual, audio, and text modalities, respectively. Let m be the classification weight matrix corresponding to mode m. The two are bias vectors, and together they constitute the mode. The classification layer parameters are used to represent the samples. Mapping to classification logic The corresponding category probability distribution is The uncertainty of this mode is then expressed using Shannon information entropy. This is represented; then, modal fusion weights are generated based on the uncertainty: The modal feature representations are weighted and fused based on modal fusion weights to obtain the final fused representation. Finally, the detection results of the target video are output through the classification layer. : in, and These represent the weight parameters and bias parameters of the classification layer, respectively.

5. A video detection system based on cue calibration and continuous learning, characterized in that, The method according to any one of claims 1-4 specifically includes the following modules: The multimodal feature extraction module is used to receive target video data, extract features and perform unified representation mapping on text information, visual frame information and audio information in the video, and output an initial video representation; The prompt calibration interaction module is used to maintain the prompt pool structure, generate query vectors based on the initial video representation, complete prompt key matching, prompt value aggregation and prompt enhancement interaction, and output sample representations calibrated by historical experience; The dynamic prototype update module is used to maintain the dynamic category prototypes corresponding to each violation category in the unified latent representation space, and to perform momentum updates on the category prototypes based on the cues of the current stage samples. At the same time, the prototype consistency constraint loss is calculated during the training phase to enhance the model's ability to preserve historical category knowledge. The uncertainty fusion determination module is used to calculate the uncertainty of the output results of different modes, generate mode fusion weights, perform weighted fusion of each mode representation, and output the final detection result of the target video.

Images