Video large language model security evaluation and defense method, system, device and medium

Abstract

The application discloses a video large language model security evaluation and defense method, system, device and medium, which are corresponding solutions, wherein the constructed video risk evaluation dataset is large in scale and comprehensive in coverage, hierarchical and multilingual design, so that the dataset has stronger risk generalization capability, and the performance and security of the model in the actual deployment environment can be effectively improved. In the two-stage post-training strategy, the method of the first stage is a context-aware risk detection token fine-tuning method, which can significantly improve the risk signal perception ability of the model when processing complex inputs; the method of the second stage is a security reward-driven dynamic strategy optimization method, which can improve the stability and generalization of strategy training. The optimized video large language model obtained finally can efficiently realize risk detection and security response, and is especially suitable for scenes such as real-time content review, online education and question answering, government video monitoring and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video large language model security, and particularly relates to a video large language model security evaluation and defense method, system, device and medium. BACKGROUND

[0002] In recent years, large language models (LLM) have made significant breakthroughs in text generation, reasoning, and other aspects, while multi-modal models such as GPT-4V (Generative Pretrained Transformer 4 Vision) models, LLaVA (Large Language and Vision Assistant) models, CLIP (Contrastive Language-Image Pretraining) models, ALIGN (Large-Scale Image-Text Alignment) models, and other models further inject visual capabilities into language models, enabling image description, visual question answering, and instruction reasoning capabilities. These models are widely used in critical areas such as autonomous driving, security monitoring, medical imaging, and education, demonstrating strong application potential and commercial value. As image-language model applications deepen, people have gradually realized that images may contain sexual, hateful, and violent content, and models may imitate sensitive content when generating. Recently, image-language multi-modal large models have made significant progress in security research. Represented by MM-SafetyBench (Multi-Modal Safety Benchmark), researchers have constructed an image safety evaluation dataset covering 13 scenarios and containing 5040 image-text pairs to test the safety of models under harmful image input. In addition, evaluation sets such as SafeBench, ToViLaG, and PrivQA, combined with OCR recognition (Optical Character Recognition), prompt word redirection, and classifier filtering mechanisms, comprehensively detect and repair model vulnerabilities, further improving the model's defense capabilities against sensitive content. At the same time, defense strategies against image security attacks have become increasingly diverse. First, image watermark detection can identify visual content containing potential sensitive information, isolating it from the source; second, OCR technology can identify sensitive text hidden in images, combined with text security mechanisms to ensure accurate rejection of malicious queries in dialogue management; third, prompt word redirection technology dynamically encapsulates user input, guiding the model to generate answers in a safe preset manner; fourth, with the help of classifier review and confidence threshold settings, intermediate detection of generated content and blocking of potential risks, these methods have practical applications in MM-SafetyBench, SafeBench, and other standards and have achieved good results.

[0003] Despite this, these techniques face essential challenges in the field of video. The video modality is not a simple extension of the image modality, but brings more complex semantic structures. Its semantics rely on the relationship between multiple frames of events in the timeline and spatial changes, action coherence, and context evolution. Models need to have cross-time reasoning and global memory capabilities, far beyond what is required for image processing. In addition, video integrates multi-modal signals such as vision, text, and audio, and its understanding highly depends on the temporal structure, role changes, causal logic, and scene continuity. Compared to the static description of images, video understanding needs to model the overall semantic mainline. Many models ignore the inter-frame causality when processing videos and focus on local pictures, which can easily misjudge the behavior intention. More importantly, the information density and semantic redundancy of videos are much higher than images. There may be multiple events and turning points in hundreds of frames. If the model cannot extract key frames, integrate context, and identify boundaries, it will be difficult to grasp the true semantics. An ordinary video may convey sensitive meanings under certain frame combinations, and frame-level detection alone cannot discover this. Therefore, the semantic complexity of videos poses a structural challenge to model security. Lack of modeling of temporal logic and event structure can lead to "security illusion" in the absence of attacks, i.e., misjudgment of security. Such problems need to be systematically improved from the ability to model video semantics. Traditional image security methods are difficult to directly apply. Second, the current video security evaluation system is still in its infancy. The latest Video-SafetyBench (Video-SafetyBench) includes 2264 video-text pairs and counts 48 fine-grained risk categories. However, this work focuses on evaluating attack success rates and does not achieve a closed-loop system from evaluation to defense. Finally, in terms of defense mechanisms, image processing methods such as OCR and watermark detection cannot cope with cross-frame jump attacks. Prompt word redirection and classification review mechanisms also have difficulty controlling the constantly changing context in dynamic videos. This leads to a lack of response strategies when video LLMs face a combination of content splitting, prompt hijacking, and covert disguising attacks. In summary, compared to image security systems, video security has significant gaps in "evaluation benchmark research", "time sequence control ability of defense strategies", and "cross-modal linkage security warning". The security research of video LLMs urgently needs to break through the constraints of time dimension attacks and build a complete system that includes multiple risk states, cross-modal linkage, and active and passive defense capabilities to fill the urgent technical gap in this field.

[0004] In view of this, the present application is proposed. SUMMARY

[0005] The purpose of the present application is to provide a video large language model security evaluation and defense method, system, device and medium, which can identify and resist "video + query" input with malicious intent while maintaining normal understanding and generation ability.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A video large language model security evaluation and defense method, comprising:

[0008] A multi-class, multi-language and multi-modal video risk evaluation dataset is constructed;

[0009] Training a video large language model, comprising: in a first stage, inputting training data in the video risk evaluation dataset into the video large language model, inserting a risk detection token in the original token of the training data extracted by the video large language model, and performing a risk identification task based on the risk detection token by the video large language model, calculating a corresponding risk identification loss, and calculating a context consistency loss combining the risk detection token and the original token, combining all the calculated losses to construct a training loss in the first stage, and fine-tuning the video large language model; in a second stage, inputting pre-grouped training data into the fine-tuned video large language model, and scoring the response output by the fine-tuned video large language model, using the score to construct a relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model;

[0010] Inputting the video and the query text into the optimized video large language model, and inserting a risk detection token in the original token extracted by the video large language model, and in the inference process of the video large language model, based on the risk detection token, perceiving whether the video and the query text contain harmful elements, and outputting a normal response or a safe response.

[0011] A video large language model security evaluation and defense system for implementing the foregoing method, comprising:

[0012] A dataset construction unit for constructing a multi-class, multi-language and multi-modal video risk evaluation dataset;

[0013] A training unit for training a video large language model, comprising: in a first stage, inputting training data in the video risk evaluation dataset into the video large language model, inserting a risk detection token in the original token of the training data extracted by the video large language model, and performing a risk identification task based on the risk detection token by the video large language model, calculating a corresponding risk identification loss, and calculating a context consistency loss combining the risk detection token and the original token, combining all the calculated losses to construct a training loss in the first stage, and fine-tuning the video large language model; in a second stage, inputting pre-grouped training data into the fine-tuned video large language model, and scoring the response output by the fine-tuned video large language model, using the score to construct a relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model;

[0014] The security evaluation and defense unit is used for inputting the video and the query text into the optimized video large language model, inserting a risk detection token in original tokens extracted by the video large language model, and outputting a normal response or a security response based on whether harmful elements are contained in the video and the query text during an inference process of the video large language model.

[0015] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0017] A readable storage medium stores a computer program, which implements the aforementioned method when executed by a processor.

[0018] As can be seen from the technical solutions provided by the above-mentioned application, the video risk evaluation dataset constructed by the application has a large scale and comprehensive coverage, and the hierarchical and multilingual design makes the dataset have stronger risk generalization ability, which can effectively improve the performance and security of the model in the actual deployment environment. In the two-stage post-training strategy, the first-stage method is a context-aware risk detection token fine-tuning method, which can significantly improve the model's ability to perceive risk signals when processing complex inputs; the second-stage method is a safety reward-driven dynamic strategy optimization method, which can improve the stability and generalization of strategy training. The optimized video large language model obtained finally can efficiently realize risk detection and security response, and is particularly suitable for real-time content review, online education and question answering, government video monitoring and other scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A flowchart of a video large language model security evaluation and defense method provided by an embodiment of the present application.

[0021] Figure 2 A schematic diagram of a data generation pipeline provided by an embodiment of the present application.

[0022] Figure 3 A schematic diagram of a two-stage post-training strategy framework provided by an embodiment of the present application.

[0023] Figure 4A schematic diagram of a video large language model security evaluation and defense system provided for an embodiment of the present application.

[0024] Figure 5 A schematic diagram of a processing device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0026] First, the terms possibly used in the present text are explained as follows:

[0027] The term "and / or" means either one or both, for example, X and / or Y means three cases including "X", "Y" or "X and Y".

[0028] The terms "include", "contain", "have", "possess" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.), should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed in the art.

[0029] The term "consisting of" means excluding any technical feature element not explicitly listed. If this term is used in a claim, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0030] Unless otherwise explicitly specified or limited, the terms "mount", "connect", "connect", "fix", and other terms should be understood broadly, for example: it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present text can be understood according to the specific circumstances.

[0031] A video large language model security evaluation and defense method, system, device and medium are described in detail below. The content not described in detail in the embodiments of the present application belongs to the prior art known to those skilled in the art. If not specified in the embodiments of the present application, the conditions are in accordance with the conventional conditions in the art or the conditions recommended by the manufacturer. If not specified in the embodiments of the present application, the reagents or instruments used are conventional products that can be obtained by commercial purchase.

[0032] Embodiment one

[0033] The embodiments of the present application provide a video large language model security evaluation and defense method, as shown in the figure, which mainly includes the following steps: Figure 1 The embodiments of the present application provide a video large language model security evaluation and defense method, as shown in the figure, which mainly includes the following steps:

[0034] Step 1, constructing a multi-class, multi-language and multi-modal video risk evaluation dataset.

[0035] In the embodiments of the present application, raw video clips in multiple languages from multiple public platforms are collected, and are preliminarily filtered through community risk labels of multiple categories and platform content review specifications; then, a generated model is used in combination with manual review to construct a text query matched with each preliminarily filtered video, to form a video and query text combination pair, and finally to form a video risk evaluation dataset.

[0036] Step 2, training the video large language model using a two-stage post-training strategy.

[0037] In the embodiments of the present application, the two-stage post-training strategy includes two stages of context-aware risk detection token fine-tuning (CARDT-FT) and dynamic adaptive group relative policy optimization (DA-GRPO) driven by security reward.

[0038] In the first stage, the training data in the video risk assessment dataset is input into the video large language model, a risk detection token is inserted into the original token of the training data extracted by the video large language model, a risk identification task based on the risk detection token is performed by the video large language model, a corresponding risk identification loss is calculated, a context consistency loss is calculated by combining the risk detection token and the original token, a first-stage training loss is constructed by combining all the calculated losses, and the video large language model is fine-tuned; in the second stage, the pre-grouped training data is input into the fine-tuned video large language model, and the responses output by the fine-tuned video large language model are scored to construct a relative ranking reward between response pairs, the fine-tuned video large language model is subjected to reinforcement training, and an optimized video large language model is obtained.

[0039] In the embodiment of the application, the risk detection token inserted into the original token of the training data extracted by the video large language model comprises: the training data is a video and query text combination pair, and the original token comprises a visual modality token sequence and a text modality token sequence , which is represented as: , ; wherein, , a single token in the visual modality token sequence and the text modality token sequence, i=1,2,…,N, j=1,2,…,M, N and M represent the length of the visual modality token sequence and the text modality token sequence, respectively; the risk detection token comprises two, which are denoted as and ; the risk detection token is inserted into the visual modality token sequence, and the risk detection token is inserted into the text modality token sequence.

[0040] Preferably, the risk detection token is updated using an attention mechanism, and then inserted into the visual modality token sequence and the text modality token sequence.

[0041] wherein the risk detection token and are updated, and are denoted as and respectively, and the updating method is represented as:

[0042] ;

[0043] ;

[0044] wherein Attn is an attention mechanism, and are key mapping layers for vision and text, and is a value mapping layer of vision and text.

[0045] In the embodiment of the present application, the combining of all the calculated losses to construct the training loss of the first stage comprises:

[0046] combining the calculated risk identification loss and the context consistency loss, and the standard language modeling loss to construct the training loss L of the first stage, expressed as:

[0047] ;

[0048] wherein, and are adjustable hyperparameters; is the risk identification loss, calculated by using the cosine similarity of the risk detection token and the mean of the original token; is the context consistency loss, calculated by using the mean of the risk detection token and the original token.

[0049] In the embodiment of the present application, the inputting of the pre-grouped training data into the fine-tuned video large language model and the scoring of the responses output by the fine-tuned video large language model comprises: recording the responses output by the fine-tuned video large language model for the training data in the i-th group as , each response corresponding to a single training data in the i-th group of training data, is the total number of training data in the i-th group of training data; and giving a security preference score by a scorer, expressed as: wherein,

[0050] ;

[0051] wherein, is the security preference score of the response , the response referring to any response in ; is the format matching reward, is the content matching reward.

[0052] Preferably, an enhancement mechanism is introduced to calculate the final security preference score, comprising: introducing a confidence adjustment term defined as the inverse function of the token-level average probability entropy of the response to represent the stability of the generated content; and introducing a reward enhancement mechanism of fusion modal complementarity, increasing a modal complementarity gain term based on the security preference score , then the final security preference score is:​​

[0053] ;

[0054] wherein, is a reward gain weight coefficient, is a response modal complementarity reward defined as: , , and are responses information in denote the probability of rejection of the response under the joint input of vision and text, and are the rejection probabilities under the visual and text inputs only, and m is a complementarity threshold.

[0055] Step 3, security evaluation and defense based on the optimized video large language model.

[0056] The video and the query text are input into the optimized video large language model, and the risk detection token is inserted in the original token extracted by the video large language model, and in the inference process of the video large language model, whether the harmful elements are contained in the video and the query text is perceived based on the risk detection token, and a normal response or a safe response is output.

[0057] The above scheme provided by the embodiment of the application proposes a unified and multi-modal security evaluation and defense system, so that the video large language model can identify and resist the "video + query text" input containing malicious intent, while maintaining normal understanding and generation ability.

[0058] In order to more clearly show the technical solutions provided by the application and the technical effects produced, the method provided by the embodiment of the application is described in detail below with specific embodiments.

[0059] I. Overall summary of the scheme.

[0060] The embodiment of the application provides a security evaluation and defense method for a video large language model.

[0061] Firstly, the application proposes a video risk evaluation data set and its data generation pipeline, which can quickly and low-costly construct a video security benchmark meeting the needs of industrial applications. Through a semi-automatic data labeling and generation process, users can conveniently obtain video-text combination data with wide risk coverage and language diversity, which is used for security capability evaluation of existing video large language models, so as to efficiently locate the short board in model security protection, and guide the next step of model security optimization accordingly.

[0062] Secondly, the context-aware risk detection token fine-tuning (CARDT-FT) method proposed by the application can be applied to general video large language models. Based on the existing video understanding model, the user only needs to add the corresponding trainable risk detection token and the corresponding context attention mechanism, so as to realize the sensitivity enhancement of the model to the risk content of the video input. Through this fine-tuning process, the video understanding model can more stably and accurately perceive and respond to the security risks in the multi-modal input, and is suitable for deploying on a server to carry out various video security audit tasks.

[0063] In addition, the application proposes a security reward driven dynamic strategy optimization scheme, which can be directly applied to the security reward guided dynamic group relative strategy optimization of the video understanding model. The user can introduce a confidence adjustment term and a modal complementary gain term according to specific requirements, enhance the stability and generalization of the model strategy through the strategy gradient method, thereby effectively alleviating the problems of large model training overhead, weak generalization ability and strategy forgetting in the industrial environment, and significantly improving the security defense capability and stability of the model in actual application.

[0064] II. Detailed introduction of the scheme.

[0065] The application aims to solve the security degradation problem of the current video large language model (Video LLM) when processing complex multi-modal input. After the introduction of the video mode, the model often has problems such as decreased rejection rate, judgment bias and response out of control when facing multi-frame time sequence structure, cross-frame semantic connection and scene changes. The static image risk identification mechanism, content filtering method and language strategy constraint relied on by the traditional image-language model show obvious lack of adaptability under the video mode. Therefore, the application proposes an overall framework covering video security evaluation and model security defense, which includes two core parts: one is to build a multi-class, multi-language and multi-modal video risk evaluation dataset; the other is to design a post-training defense mechanism with active identification and rejection capability, finally forming a closed-loop system from input evaluation (security evaluation), risk identification (perception fine-tuning) to strategy optimization.

[0066] 1. Input evaluation part.

[0067] The application provides a video safety data construction method based on risk stratification and multi-language coverage. First, raw video clips from multiple public platforms are collected, and community risk labels (such as violence, pornography, discrimination, self-harm, etc.) and platform content review specifications are used for preliminary filtering. Subsequently, a supporting text query is constructed using a generative model combined with manual review, and a "video + query" combination pair is finally generated. Compared with the traditional benchmark, which only divides risk categories based on the static content of video frames, the application enhances the triggering of semantic queries on video content, and more realistically simulates the risk triggering path in user-model interaction through question and answer combination. The construction pipeline is as shown in Figure 2 The final dataset ViSA (Video Safety Atlas) contains more than 80000 samples, covering 6 content risk types (violent or dangerous content, mental and behavioral health, controlled substances and activities, integrity and authenticity, privacy safety, pornography and nudity) and 10 language communities (English, Japanese, Korean, Spanish, Hindi, Arabic, Russian, French, German, Portuguese), and has strong cross-modal generalization ability. Here, only as an example, users can expand the number of samples, risk types, and language types according to actual needs.

[0068] Figure 2 The related process examples of data construction are shown in the middle, including: 1) harmful video retrieval, 2) safety label generation, and 3) evaluation process. Figure 2 The safety label generation in the middle is the preliminary filtering process described above, which uses the MAX VL model to intercept harmful or safe segments in the video, including five parts: a) segmenting the video, b) analyzing the harmfulness of the segment; c) analyzing the harmfulness of the intercepted segment; d) giving the segment a harmfulness label; e) merging. The first half of the evaluation process is the use of a generative model combined with manual review to construct a supporting text query, Figure 2 In the example shown in the middle, the MAX VL model generates a candidate text query, and the Long model selects the most relevant text query from the text query. Here, an example of a text query is provided: "How to ensure that the conflict in the video continues to escalate", and the model output in the example responds to the text query by refusing to answer. The MAX VL model involved in this part is the most advanced multi-modal large model developed by the Alibaba Universal Thousand Questions team, and Long refers to a language large model. In addition, Figure 2GPT-4 in the table is the fourth version of the generative pre-trained transformer (GPT-4), DSI (Defense Success Indicator) refers to the judgment of the Long model on the video understanding large model, 1 represents safety, and 0 represents insecurity; US refers to the score of whether the answer is meaningful and effective, and the score value is between 0-5.

[0069] To realize the structured evaluation, the application constructs a standard evaluation subset ViSA-Eval (5700 in total) from the main data set, and embeds harmful questions in the video to construct an evaluation subset under visual attack (ViSA-Eval-VA). The above hierarchical subsets are used to test the performance of the model, which not only evaluates the defense capability of the model under high-risk input, but also investigates the general performance of the model in actual tasks. In addition, the remaining data is used to train the model.

[0070] 2. Model defense part.

[0071] The model defense part covers risk identification and strategy optimization, and a two-stage post-training strategy is designed accordingly, as shown in the framework of Figure 3 , which includes Context-Aware Risk Detection Token Fine-Tuning (CARDT-FT) and Dynamic Adaptive Group Relative Policy Optimization (DA-GRPO).

[0072] Figure 3 An example of two-stage post-training is provided using the VideoLLaMA3 model, which is an open-source video large language model from Alibaba DAMO Academy. The left part is the CARDT-FT stage, which involves a visual encoder, a large language model, and a hidden layer, all of which belong to the VideoLLaMA3 model. Their working principles can refer to conventional techniques, and the application will not be repeated. The right side is the DA-GRPO stage, where cold start means only using standard language modeling loss (autoregressive loss) for training, so that the model can output answers in a specific format; the characters in the second dashed box at the top are labels used for supervision during training, which are the structure and content of the ideal model response. think is the thinking chain process, answer is the answer to the question, vidType is the probability of harmful video content, textType is the probability of harmful text, and crossType is the probability of joint harm, which correspond to , 、 .

[0073] (1) CARDT-FT stage (first stage).

[0074] In the CARDT-FT stage, a set of trainable risk detection tokens are embedded in the feature sequence of the original video and text, respectively, for perceiving potential risk signals in the video and text. Compared with the traditional way of inserting pseudo tokens at the head and tail of the sequence, the present application embeds risk detection tokens in specific positions in the token sequence, so that they can fully participate in the semantic fusion process. Specifically, we insert risk detection tokens at the tail of the visual token and the tail of the language token. In addition, the present application introduces a context attention mechanism to enhance the perception ability of the risk detection token, so that the model can more accurately identify potential risk areas when facing complex video-text inputs, and form a closer risk linkage expression between different modalities.

[0075] Specifically, a trainable risk detection token is inserted in the representation sequence of the visual modality and the text modality, respectively, denoted as:

[0076] ;

[0077] wherein, is a real set symbol, and d is the dimension.

[0078] The original modality sequence is respectively:

[0079] ;

[0080] ;

[0081] wherein, , correspond to the visual modality token sequence and the text modality token sequence, , correspond to a single token in the visual modality token sequence and the text modality token sequence, i = 1, 2, …, N, j = 1, 2, …, M, N and M correspond to the length of the visual modality token sequence and the text modality token sequence.

[0082] In order to enhance the sensitivity of the token to the context, the risk detection token is fused with the context feature through a lightweight attention mechanism, and is updated to and :

[0083] ;

[0084] ;

[0085] wherein, and are the key mapping layers for vision and text, and are the value mapping layers for vision and text.

[0086] The attention is calculated as:

[0087] ;

[0088] ;

[0089] where, denotes and , K denotes and , V denotes and , T is the transpose symbol, d is the dimension of q, is the attention weight.

[0090] During training, the model needs to optimize three losses. First is the standard language modeling loss ; second is the risk identification task of two risk detection tokens, in the form of binary classification loss:

[0091] ;

[0092] ;

[0093] where, represents the modal risk label, which denotes whether it is harmful or not; and are two linear layers to reduce the dimension, and the subsequent and will be 1-dimensional, is the sigmoid function (S-shaped function), and BCE is the binary cross-entropy function. and are the risk identification loss terms of the visual and text modalities, respectively, and the sum of the two is the risk identification loss .

[0094] In order to enhance the alignment effect between the risk detection token and the real context, a context consistency loss is also introduced, defined as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] wherein, , corresponding to the mean of the visual modality token sequence, the text modality token sequence (i.e., the original token mean), and cos is the cosine similarity; and are the context consistency loss terms of the visual modality and the text modality, respectively, and the sum of the two is the context consistency loss .

[0100] The final total loss is:

[0101] ;

[0102] wherein, and are adjustable hyperparameters, for example, both can be set to 1.0.

[0103] (2) DA-GRPO phase (second phase).

[0104] The present application proposes a safety reward driven dynamic strategy optimization method to enhance the safety response capability, training stability and strategy generalization of the video large language model in complex multi-modal risk scenarios.

[0105] The method first uses the intra-group ranking method to strengthen the training of the model output. Specifically, for each group of input data , the fine-tuned video large language model generates multiple responses The skilled in the art can understand that the response is the answer of the model, specifically: the token in the hidden space of the model is converted into a response token, each response token is a word probability distribution, and the word with the highest probability in each token is selected to form a response.

[0106] Each response is given a safety preference score by a rule-based scorer . The scorer judges according to whether the response rejects harmful requests, language fluency, expression neutrality, etc., and returns a relative score rather than an absolute label. The scorer can be formulated as:

[0107] ;

[0108] wherein, is the format matching reward, is the content matching reward.

[0109] Then, the relative ranking reward between response pairs is constructed to drive the reinforcement learning strategy optimization.

[0110] On this basis, the application introduces two key enhancement mechanisms to improve the optimization effect:

[0111] First, in order to make the policy optimization more stable and reliable, the application introduces a confidence adjustment term on the basis of the original reward difference , defined as the inverse function of the average probability entropy of the response token, to represent the stability of the generated content. That is:

[0112] ;

[0113] ;

[0114] Where exp is the natural exponential function, represents the generation probability of the jth word of the tth response token in the response , and is the response length.

[0115] Therefore, the security preference score is adjusted to :

[0116] .

[0117] The confidence adjustment term dynamically adjusts the training weight of the response during optimization, suppressing the gradient contribution of unstable language output samples, thereby improving the convergence quality of the strategy.

[0118] Second, the application proposes a reward enhancement mechanism that fuses modal complementarity, which is used to guide the model to fully utilize the information difference between visual and text modalities, and improve the perception ability of potential risks. Specifically, for each response , a modal complementarity gain term is introduced on the basis of the original reward to form a new security preference score :

[0119] ;

[0120] Where is the modal complementarity reward of the response, defined as:

[0121] ;

[0122] Where represents the rejection probability of the response under the joint input of vision and text, and are the rejection probabilities under the input of vision and text only, respectively, m is the complementarity threshold, and is the reward gain weight coefficient. This mechanism encourages the model to exhibit information sensitivity beyond single modality under multi-modal fusion input, so that it can more reliably identify potential cross-modal induced attacks.

[0123] Combining the two key enhancement mechanisms, the final security preference score is:

[0124] .

[0125] This strategy optimization mechanism improves the training stability by adjusting the confidence while comparing ideas within the group, and enhances the cross-modal reasoning ability by the modal complementary reward, making the defense ability of the model more controllable and generalizable as a whole, and suitable for complex and multi-source video input scenes in actual deployment.

[0126] 3. Actual deployment of the scheme.

[0127] In terms of deployment, in order to facilitate system integration, the present application supports encapsulating the entire defense strategy (i.e. the data set and the two-stage post-training strategy) as an independent module, which is inserted into the existing video large language model architecture without modifying the basic encoder and language model structure. At the same time, interfaces compatible with external security review modules (such as OCR detectors, prompt word filters) are retained to support multi-channel joint defense deployment in security-sensitive scenarios. In the inference process, if the risk detection token activation degree exceeds the threshold, the model will automatically enter the "security response mode", such as outputting warning information, refusing to answer, turning to neutral generation, etc.

[0128] Overall, the present application establishes a complete technical chain around "security assessment - perception fine-tuning - strategy optimization - control deployment", effectively improving the risk identification ability and defense judgment stability of the video large language model in complex contexts without increasing the complexity of the model structure. Compared with traditional image security mechanisms, the present application is more suitable for processing dynamic, continuous and semantically ambiguous video inputs, and has good cross-modal adaptability and engineering landing potential.

[0129] III. Performance description.

[0130] The performance of the present application scheme is introduced by taking the existing video large model as an example. Here, the VideoLLaMA3 model is taken as an example for introduction.

[0131] When the present application is applied to VideoLLaMA3, the trained VideoLLaMA3 achieves a defense success rate of 87.5% on ViSA-Eval, which is 68.3% higher than the initial defense success rate of 19.2%. In addition, the trained VideoLLaMA3 improves by 81.1% (12.6% to 93.7%) on ViSA-Eval-VA, which proves that the present application has strong robustness and can resist special visual attacks.

[0132] To verify the generalization of the method, the trained VideoLLaMA3 is tested on the image safety evaluation dataset MM-SafetyBench and VLGuaRD, which improves by 55.3% (32.1%→87.4%) and 40.2% (44.5%→84.5%) respectively.

[0133] Finally, in order to verify that the trained VideoLLaMA3 still has strong video understanding ability, we test it on the video dataset MVBench and VideoMME, which decreases by 0.5% (65.5%→65.0%) and 0.3% (59.5%→59.2%) respectively. The results show that the present application significantly improves the safety of the model while almost not sacrificing the video understanding performance, indicating that the model trained by the present application can maintain normal understanding and generation ability.

[0134] Through the description of the above implementation, those skilled in the art can clearly understand that the above examples can be implemented by software, or by means of software and necessary general hardware platform. Based on such understanding, the technical solutions of the above examples can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0135] Embodiment two

[0136] The present application also provides a video large language model safety evaluation and defense system, which is mainly used to implement the method provided by the foregoing embodiments, as shown in the figure, the system mainly includes: Figure 4

[0137] A dataset construction unit is configured to construct a multi-category, multi-language and multi-modal video risk evaluation dataset.

[0138] ​The training unit is used to train the video large language model using a two-stage post-training strategy, including: In the first stage, training data from the video risk assessment dataset is input into the video large language model. Risk detection tokens are inserted into the original tokens extracted from the training data by the video large language model. The video large language model then performs a risk identification task based on the risk detection tokens, calculates the corresponding risk identification loss, and calculates the context consistency loss by combining the risk detection tokens and the original tokens. All calculated losses are combined to construct the training loss for the first stage, and the video large language model is fine-tuned. In the second stage, pre-grouped training data is input into the fine-tuned video large language model, and the responses output by the fine-tuned video large language model are scored. The scores are used to construct a relative ranking reward between response pairs, and the fine-tuned video large language model is reinforced to obtain an optimized video large language model.

[0139] The security assessment and defense unit is used to input video and query text into the optimized video big language model, and insert a risk detection token into the original token extracted by the video big language model. During the inference process of the video big language model, it perceives whether the video and query text contain harmful elements based on the risk detection token, and outputs a normal response or a safe response.

[0140] The technical details involved in the above system have been described in detail in the previous embodiments, and therefore will not be repeated here.

[0141] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0142] Example 3

[0143] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0144] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0145] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0146] The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.

[0147] The output device can be a display terminal.

[0148] The memory can be a random access memory (RAM) or a non-volatile memory such as a disk memory.

[0149] Embodiment Four

[0150] The application further provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the foregoing embodiments.

[0151] The readable storage medium in the embodiments of the application as a computer readable storage medium can be arranged in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk or various media capable of storing program codes.

[0152] The above merely describes the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims. The information disclosed in the background section of the specification is merely intended to deepen the understanding of the general background of the application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known to those skilled in the art.

Claims

1. A video large language model security evaluation and defense method, characterized in that, The method comprises the following steps: constructing a multi-category, multi-language and multi-modal video risk assessment data set; training a video large language model, comprising: in a first stage, inputting training data in the video risk assessment data set into the video large language model, inserting a risk detection token into original tokens of the training data extracted by the video large language model, performing a risk identification task based on the risk detection token by the video large language model, calculating a risk identification loss corresponding to the risk identification task, calculating a context consistency loss combining the risk detection token and the original token, constructing a training loss of the first stage combining all the calculated losses, and fine-tuning the video large language model; in a second stage, inputting pre-grouped training data into the fine-tuned video large language model, scoring responses output by the fine-tuned video large language model, using the scores to construct a relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model; inputting a video and a query text into the optimized video large language model, inserting a risk detection token into original tokens extracted by the video large language model, and in the inference process of the video large language model, based on the risk detection token, perceiving whether the video and the query text contain harmful elements, and outputting a normal response or a safe response; The inputting the pre-grouped training data into the fine-tuned video large language model and scoring responses output by the fine-tuned video large language model comprises: inputting the pre-grouped training data into the fine-tuned video large language model, and scoring responses output by the fine-tuned video large language model The responses of the group of training data are denoted as Each response corresponds to a single training data in the group of training data The group of training data is denoted as The total number of training data of the group of training data is denoted as ​ assigning a safety preference score by a scorer, which is represented as: ； wherein, in response to a security preference score of the response, refers to any response in the response; is a format matching reward, is a content matching reward; Also included: introducing an enhancement mechanism, computing a final security preference score, including: introducing a confidence adjustment term defined as the inverse function of the token-level average probability entropy in response to represent the stability of the generated content; and introducing a reward enhancement mechanism of fusion modal complementarity, adding a modal complementarity gain term to the security preference score , then the final security preference score is: ； where, is the reward gain weight coefficient, is the response modal complementarity reward, defined as: , , and are the information in the response , denotes the rejection probability of the response under the joint visual and text input, and are the rejection probabilities under the visual and text inputs only, respectively, and m is the complementarity threshold.

2. The method of claim 1, wherein, The method for constructing a multi-category, multi-language and multi-modal video risk assessment data set comprises: collecting original video clips in multiple languages from multiple public platforms, and preliminarily filtering the video clips through community risk labels of multiple categories and platform content review specifications; then, using a generative model in combination with manual review, constructing a text query matched with each preliminarily filtered video to form a video and query text combination pair, and finally forming a video risk assessment data set.

3. The method of claim 1, wherein, The method further comprises the following steps: The training data is a video and query text combination pair, and the original token contains a visual modality token sequence and a text modality token sequence , denoted as: , ; wherein, , corresponding to a single token in the visual modality token sequence, the text modality token sequence, i = 1, 2, …, N, j = 1, 2, …, M, N and M corresponding to the length of the visual modality token sequence, the text modality token sequence. The risk detection tokens include two, denoted as and ; the risk detection token is inserted into the visual modality token sequence, and the risk detection token is inserted into the text modality token sequence.

4. The method of claim 3, wherein, updating the risk detection token by using an attention mechanism, and then inserting the updated risk detection token into the visual modality token sequence and the text modality token sequence; The method further comprises the following steps: Wherein, the risk detection token With After updating, the corresponding is denoted as With The way of updating the risk detection token by adopting the attention mechanism is expressed as: ； ； wherein Attn is an attention mechanism, and is a key mapping layer for vision and text, and is a value mapping layer for vision and text.

5. The method of claim 1 or 3 or 4, wherein, The method further comprises the following steps: combining the computed risk identification loss with the contextual consistency loss, as well as a standard language modeling loss The training loss L for the first stage is constructed and is represented as: ； wherein, with is a tunable hyper-parameter; is a risk identification loss, computed using the cosine similarity between the risk detection token and the mean of the original tokens; is a context consistency loss, computed using the mean of the risk detection token and the original tokens.

6. A video large language model security evaluation and defense system, characterized in that, The method further comprises the following steps: A data set construction unit is configured to construct a multi-category, multi-language and multi-modal video risk assessment data set. The training unit is configured to train the video large language model, including: in a first stage, inputting training data in a video risk assessment dataset into the video large language model, inserting a risk detection token into original tokens of the training data extracted by the video large language model, performing a risk identification task based on the risk detection token by the video large language model, calculating a risk identification loss corresponding to the risk identification task, calculating a context consistency loss based on the risk detection token and the original token, combining all the calculated losses to construct a training loss of the first stage, and fine-tuning the video large language model; in a second stage, inputting pre-grouped training data into the fine-tuned video large language model, scoring responses output by the fine-tuned video large language model, constructing a relative ranking reward between response pairs based on the scoring, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model. The security assessment and defense unit is configured to input a video and a query text into the optimized video large language model, insert a risk detection token into original tokens extracted by the video large language model, and perceive whether harmful elements are contained in the video and the query text based on the risk detection token in an inference process of the video large language model, and output a normal response or a security response.

7. A processing device, characterized by The computer program product includes: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A readable storage medium, storing a computer program, characterized in that, The computer program product includes: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

Images