Video large language model security assessment and defense method, system, device and medium

Abstract

The invention discloses a video big language model security assessment and defense method, system, device and medium, which are corresponding schemes, and in the scheme, a constructed video risk assessment data set is large in scale, comprehensive in coverage, hierarchical and multi-language in design, so that the data set has stronger risk generalization ability, and the security assessment and defense of the video big language model is realized. And the performance and security of the model in an actual deployment environment can be effectively improved. In the two-stage post-training strategy, the method in the first stage is a context sensing type risk detection token fine adjustment method, and the risk signal sensing capacity of the model during complex input processing can be remarkably improved; the method in the second stage is a security reward-driven dynamic strategy optimization method, and the stability and generalization of strategy training can be improved. The finally obtained optimized video big language model can efficiently realize risk detection and safety response, and is especially suitable for scenes such as real-time content auditing, online education question answering, government affair video monitoring and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video large language model security, and particularly relates to a video large language model security evaluation and defense method, system, device and medium. BACKGROUND

[0002] In recent years, large language models (LLM) have made significant breakthroughs in text generation, reasoning, and other aspects, while multi-modal models such as GPT-4V (Generative Pretrained Transformer 4 Vision) models, LLaVA (Large Language and Vision Assistant) models, CLIP (Contrastive Language-Image Pretraining) models, ALIGN (Large-Scale Image-Text Alignment) models, and other models further inject visual capabilities into language models, enabling image description, visual question answering, and instruction reasoning capabilities. These models are widely used in critical areas such as autonomous driving, security monitoring, medical imaging, and education, demonstrating strong application potential and commercial value. As image-language model applications deepen, people have gradually realized that images may contain sexual, hateful, and violent content, and models may imitate sensitive content when generating. Recently, image-language multi-modal large models have made significant progress in security research. Represented by MM-SafetyBench (Multi-Modal Safety Benchmark), researchers have constructed an image safety evaluation dataset covering 13 scenarios and containing 5040 image-text pairs to test the safety of models under harmful image input. In addition, evaluation sets such as SafeBench, ToViLaG, and PrivQA, combined with OCR recognition (Optical Character Recognition), prompt word redirection, and classifier filtering mechanisms, comprehensively detect and repair model vulnerabilities, further improving the model's defense capabilities against sensitive content. At the same time, defense strategies against image security attacks have become increasingly diverse. First, image watermark detection can identify visual content containing potential sensitive information, isolating it from the source; second, OCR technology can identify sensitive text hidden in images, combined with text security mechanisms to ensure accurate rejection of malicious queries in dialogue management; third, prompt word redirection technology dynamically encapsulates user input, guiding the model to generate answers in a safe preset manner; fourth, with the help of classifier review and confidence threshold settings, intermediate detection of generated content and blocking of potential risks, these methods have practical applications in MM-SafetyBench, SafeBench, and other standards and have achieved good results.

[0003] Despite this, these techniques face essential challenges in the field of video. The video modality is not a simple extension of the image modality, but brings more complex semantic structures. Its semantics rely on the relationship between multiple frames of events in the timeline and spatial changes, action coherence, and context evolution. Models need to have cross-time reasoning and global memory capabilities, far beyond what is required for image processing. In addition, video integrates multi-modal signals such as vision, text, and audio, and its understanding highly depends on the temporal structure, role changes, causal logic, and scene continuity. Compared to the static description of images, video understanding needs to model the overall semantic mainline. Many models ignore the inter-frame causality when processing videos and focus on local pictures, which can easily misjudge the behavior intention. More importantly, the information density and semantic redundancy of videos are much higher than images. There may be multiple events and turning points in hundreds of frames. If the model cannot extract key frames, integrate context, and identify boundaries, it will be difficult to grasp the true semantics. An ordinary video may convey sensitive meanings under certain frame combinations, and frame-level detection alone cannot discover this. Therefore, the semantic complexity of videos poses a structural challenge to model security. Lack of modeling of temporal logic and event structure can lead to "security illusion" in the absence of attacks, i.e., misjudgment of security. Such problems need to be systematically improved from the video semantic modeling capability itself, and traditional image security methods cannot be directly applied. Second, the current video security evaluation system is still in its infancy. The latest Video-SafetyBench (Video-SafetyBench) includes 2264 video-text pairs and counts 48 fine-grained risk categories. However, this work focuses on evaluating attack success rates and does not achieve a closed-loop system from evaluation to defense. Finally, in terms of defense mechanisms, image processing methods such as OCR and watermark detection cannot handle cross-frame jump attacks. Prompt word redirection and classification review mechanisms also have difficulty controlling the constantly changing context in dynamic videos. This leads to a lack of response strategies when video LLMs face a combination of content splitting, prompt hijacking, and covert disguising attacks. In summary, compared to the image security system, video security has significant gaps in "evaluation benchmark research", "time sequence control ability of defense strategies", "cross-modal linkage security warning", etc. The security research of video LLMs urgently needs to break through the constraints of time dimension attacks and build a complete system that includes multiple risk states, cross-modal linkage, and active and passive defense capabilities to fill the urgent technical gap in this field.

[0004] In view of this, the present application is proposed. SUMMARY

[0005] The purpose of the present application is to provide a video large language model security evaluation and defense method, system, device and medium, which can identify and resist "video + query" input with malicious intent while maintaining normal understanding and generation ability.

[0006] The purpose of the present application is achieved by the following technical solutions: A video large language model security evaluation and defense method, comprising: A multi-class, multi-language and multi-modal video risk evaluation dataset is constructed; Training a video large language model, comprising: in the first stage, inputting training data in the video risk evaluation dataset into the video large language model, inserting a risk detection token in the original token of the training data extracted by the video large language model, and performing a risk identification task based on the risk detection token by the video large language model, and calculating the corresponding risk identification loss, and calculating the context consistency loss combined with the risk detection token and the original token, combining all the calculated losses to construct the training loss of the first stage, and fine-tuning the video large language model; in the second stage, inputting pre-grouped training data into the fine-tuned video large language model, and scoring the response output by the fine-tuned video large language model, using the score to construct the relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model; Inputting the video and the query text into the optimized video large language model, and inserting the risk detection token in the original token extracted by the video large language model, and in the inference process of the video large language model, based on the risk detection token, perceiving whether the video and the query text contain harmful elements, and outputting a normal response or a safe response.

[0007] A video large language model security evaluation and defense system for implementing the foregoing method, comprising: A dataset construction unit for constructing a multi-class, multi-language and multi-modal video risk evaluation dataset; A training unit for training a video large language model, comprising: in the first stage, inputting training data in the video risk evaluation dataset into the video large language model, inserting a risk detection token in the original token of the training data extracted by the video large language model, and performing a risk identification task based on the risk detection token by the video large language model, and calculating the corresponding risk identification loss, and calculating the context consistency loss combined with the risk detection token and the original token, combining all the calculated losses to construct the training loss of the first stage, and fine-tuning the video large language model; in the second stage, inputting pre-grouped training data into the fine-tuned video large language model, and scoring the response output by the fine-tuned video large language model, using the score to construct the relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model; The security evaluation and defense unit is used for inputting the video and the query text into the optimized video large language model, inserting a risk detection token in original tokens extracted by the video large language model, and outputting a normal response or a security response based on whether harmful elements are contained in the video and the query text during an inference process of the video large language model.

[0008] A processing device, comprising: one or more processors; a memory for storing one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0009] A readable storage medium, storing a computer program, when the computer program is executed by a processor, implementing the foregoing method.

[0010] As can be seen from the technical solutions provided by the above-mentioned application, the video risk evaluation data set constructed by the application has large scale and comprehensive coverage, and the hierarchical and multi-language design makes the data set have stronger risk generalization ability, which can effectively improve the performance and security of the model in the actual deployment environment. In the two-stage post-training strategy, the first-stage method is a context-aware risk detection token fine-tuning method, which can significantly improve the model's ability to perceive risk signals when processing complex inputs; the second-stage method is a safety reward-driven dynamic strategy optimization method, which can improve the stability and generalization of strategy training. The optimized video large language model obtained finally can efficiently realize risk detection and security response, and is particularly suitable for real-time content review, online education and question answering, government video monitoring and other scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A flowchart of a video large language model security evaluation and defense method provided by an embodiment of the present application.

[0013] Figure 2 A schematic diagram of a data generation pipeline provided by an embodiment of the present application.

[0014] Figure 3 A schematic diagram of a two-stage post-training strategy framework provided by an embodiment of the present application.

[0015] Figure 4A schematic diagram of a video large language model security evaluation and defense system provided for an embodiment of the present application.

[0016] Figure 5 A schematic diagram of a processing device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0018] First, the terms that can be used in this paper are explained as follows: The term "and / or" means either one or both, for example, X and / or Y means three cases including "X" or "Y" or "X and Y".

[0019] The terms "include", "contain", "have", "possess" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.), should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed in the art.

[0020] The term "consisting of" means excluding any technical feature element not explicitly listed. If this term is used in the claims, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0021] Unless otherwise specifically provided or limited, the terms "mount", "connect", "connect", "fix", and the like should be broadly understood, for example: it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in this paper can be understood according to the specific circumstances.

[0022] A video large language model security evaluation and defense method, system, device and medium are described in detail below. The content not described in detail in the embodiments of the present application belongs to the prior art known to those skilled in the art. Unless otherwise specified in the embodiments of the present application, the conventional conditions or manufacturer's recommended conditions are used. Unless otherwise specified in the embodiments of the present application, the reagents or instruments used are conventional products that can be obtained by commercial purchase.

[0023] Embodiment one The embodiments of the present application provide a video large language model security evaluation and defense method, as shown in the following figure, which mainly includes the following steps: Figure 1 As shown in the following figure, which mainly includes the following steps: Step 1, constructing a multi-class, multi-language and multi-modal video risk evaluation dataset.

[0024] In the embodiments of the present application, raw video clips in multiple languages from multiple public platforms are collected, and are preliminarily filtered through community risk labels of multiple categories and platform content review specifications; then, a text query is constructed for each preliminarily filtered video using a generative model combined with manual review, forming a video and query text combination pair, and finally forming a video risk evaluation dataset.

[0025] Step 2, training the video large language model using a two-stage post-training strategy.

[0026] In the embodiments of the present application, the two-stage post-training strategy includes two stages: context-aware risk detection token fine-tuning (CARDT-FT) and dynamic adaptive group relative policy optimization (DA-GRPO) driven by security reward.

[0027] In the first stage, the training data in the video risk evaluation dataset is input into the video large language model, the risk detection token is inserted into the original token of the training data extracted by the video large language model, the risk identification task based on the risk detection token is performed by the video large language model, the corresponding risk identification loss is calculated, the context consistency loss is calculated combining the risk detection token and the original token, the first stage training loss is constructed combining all the calculated losses, and the video large language model is fine-tuned; in the second stage, the pre-grouped training data is input into the fine-tuned video large language model, and the responses output by the fine-tuned video large language model are scored, the relative ranking reward between response pairs is constructed using the scores, and the fine-tuned video large language model is reinforced trained to obtain an optimized video large language model.

[0028] In the embodiment of the present application, the risk detection token inserted in the original token of the training data extracted by the video large language model comprises: the training data is a video and query text combination pair, and the original token contains a visual modal token sequence and a text modal token sequence , which is expressed as: , ; wherein, , correspond to a single token in the visual modal token sequence and the text modal token sequence, i=1, 2, …, N, j=1, 2, …, M, N and M correspond to the length of the visual modal token sequence and the text modal token sequence; the risk detection token contains two, which are respectively denoted as and ; the risk detection token is inserted into the visual modal token sequence, and the risk detection token is inserted into the text modal token sequence.

[0029] Preferably, the attention mechanism is used to update the risk detection token, and then the updated risk detection token is inserted into the visual modal token sequence and the text modal token sequence.

[0030] wherein the risk detection token and are updated, and the updated risk detection token is denoted as and , and the updating method is expressed as: ; ; wherein Attn is the attention mechanism, and are the key mapping layers of the visual and text, and are the value mapping layers of the visual and text.

[0031] In the embodiment of the present application, the first stage training loss is constructed by combining all the calculated losses, comprising: combining the calculated risk identification loss and the context consistency loss, and the standard language modeling loss to construct the first stage training loss L, which is expressed as: ; wherein, and are adjustable hyperparameters; is the risk identification loss, which is calculated by using the cosine similarity of the mean of the risk detection token and the original token; For the loss of context consistency, the mean of the risk detection token and the original token is used to calculate.

[0032] In the embodiment of the application, the scoring of the response output by the fine-tuned video large language model includes: scoring the response output by the fine-tuned video large language model as the first group of training data. Each response corresponds to a single training data in the first group of training data, and the total number of training data of the first group of training data is n. ; Wherein, is the security preference score of the response , and the response refers to any response in the response . is the format matching reward, and the content matching reward.

[0033] Preferably, an enhancement mechanism is introduced to calculate the final security preference score, including: introducing a confidence adjustment term defined as the inverse function of the token-level average probability entropy of the response to represent the stability of the generated content; and introducing a reward enhancement mechanism of fusion modal complementarity, increasing the modal complementarity gain term based on the security preference score , then the final security preference score is: ; Wherein, is the reward gain weight coefficient, is the modal complementarity reward of the response , defined as: , , and are information in the response , represents the rejection probability of the response under the joint input of vision and text, and are the rejection probabilities under only visual and text input, respectively, and m is the complementarity threshold.

[0034] Step 3, security evaluation and defense based on the optimized video large language model.

[0035] The video and the query text are input into the optimized video large language model, and a risk detection token is inserted in original tokens extracted by the video large language model, in an inference process of the video large language model, whether harmful elements are contained in the video and the query text is perceived based on the risk detection token, and a normal response or a safe response is output.

[0036] The above scheme provided by the embodiments of the present application proposes a unified and multi-modal security evaluation and defense system, so that the video large language model can identify and resist the input of "video + query text" containing malicious intent, while maintaining normal understanding and generation capabilities.

[0037] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiments of the present application is described in detail below with specific embodiments.

[0038] I. Overall summary of the scheme.

[0039] The embodiments of the present application provide a security evaluation and defense method for a video large language model.

[0040] Firstly, the present application proposes a video risk evaluation dataset and its data generation pipeline, which can quickly and low-costly construct a video security benchmark meeting the needs of industrial applications. Through a semi-automatic data labeling and generation process, users can conveniently obtain video-text combination data with wide risk coverage and language diversity, which is used to evaluate the security capabilities of existing video large language models, so as to efficiently locate the short board in model security protection, and guide the next step of model security optimization accordingly.

[0041] Secondly, the context-aware risk detection token fine-tuning (CARDT-FT) method proposed by the present application can be applied to general video large language models. Based on the existing video understanding model, the user only needs to add the corresponding trainable risk detection token and the corresponding context attention mechanism, so as to realize the sensitivity enhancement of the model to the risk content of the video input. Through this fine-tuning process, the video understanding model can more stably and accurately perceive and respond to security risks in multi-modal input, and is suitable for deployment on servers to carry out various video security audit tasks.

[0042] In addition, the present application proposes a security reward driven dynamic strategy optimization scheme, which can be directly applied to the security reward guided dynamic strategy optimization of the video understanding model. The user can introduce a confidence adjustment term and a modal complementary gain term according to specific needs, and enhance the stability and generalization of the model strategy through the strategy gradient method, so as to effectively alleviate the problems of large model training overhead, weak generalization ability and strategy forgetting in industrial environment, and significantly improve the security defense capability and stability of the model in actual application.

[0043] 2. Detailed introduction of the plan.

[0044] This invention aims to address the security degradation issues currently faced by the Video Large Language Model (Video LLM) when processing complex multimodal inputs. With the introduction of the video modality, the model often experiences reduced rejection rates, judgment biases, and uncontrolled responses when faced with multi-frame temporal structures, cross-frame semantic connections, and scene changes. Traditional image-language models rely on static image risk identification mechanisms, content filtering methods, and language policy constraints, which exhibit significant adaptability deficiencies in the video modality. Therefore, this invention proposes an integrated framework covering video security assessment and model security defense. The core framework consists of two parts: first, constructing a multi-category, multi-language, and multi-modal video risk assessment dataset; second, designing a post-training defense mechanism with active recognition and rejection capabilities, ultimately forming a closed-loop system from input assessment (security assessment), risk identification (perception fine-tuning), to policy optimization.

[0045] 1. Enter the evaluation section.

[0046] The present invention proposes a method for constructing video security data based on risk stratification and multi-language coverage. First, original video clips are collected from multiple public platforms and preliminarily filtered through community risk labels (such as violence, pornography, discrimination, self-harm, etc.) and platform content review specifications. Subsequently, a generative model is combined with manual review to construct a matching text query, and finally a "video + query" combination pair is generated. Compared with the traditional benchmark method of dividing risk categories only by the static content of video frames, the present invention strengthens the triggering of semantic queries on video content, and more realistically simulates the risk triggering path in user-model interaction through a combination of questions and answers. The construction pipeline is as follows: Figure 2 The final dataset, ViSA (Video Safety Atlas), contains over 80,000 samples, covering six content risk categories (violent or dangerous content, mental and behavioral health, regulated items and activities, integrity and authenticity, privacy and security, and pornography and nudity) and 10 language communities (English, Japanese, Korean, Spanish, Hindi, Arabic, Russian, French, German, and Portuguese). It boasts strong cross-modal generalization capabilities. This is just an example; users can expand the number of samples, risk categories, and language types based on their actual needs.

[0047] Figure 2 The article shows an example of the data construction process, which includes three parts: 1) harmful video retrieval, 2) safety label generation, and 3) evaluation process. Figure 2The security label generation in the video is the preliminary filtering process described above, which intercepts harmful or safe segments in the video through the MAX VL model, including five parts: a) segmenting the video, b) analyzing the harmfulness of the segment; c) analyzing the harmfulness of the intercepted segment by the interface; d) labeling the harmfulness of the given segment; e) merging. The first half of the evaluation process is the method of using the generated model combined with manual review described above, which constructs a matching text query, Figure 2 In the example shown, the candidate text query is generated by the MAX VL model, and the most relevant text query is selected from the text query by the Long model. Here, an example of a text query is provided: "How to ensure that the contradiction in the video continues to escalate", and the model output in the example gives a safe response, refusing to answer this text query. The MAX VL model involved in this part is the most advanced multi-modal large model developed by the AliBaba Universal Question Team, and Long refers to a language large model. In addition, Figure 2 GPT-4 in the video is a generative pre-trained transformer 4 (4 is the version number), and DSI (Defense Success Indicator) refers to the Long model's judgment of the safety of the answer given by the video understanding large model, with 1 indicating safety and 0 indicating insecurity; US refers to a score given to the answer indicating whether the answer is meaningful and effective, with the score value between 0 and 5.

[0048] To achieve structured evaluation, the present application constructs a standard evaluation subset ViSA-Eval (5700 entries in total) from the main data set, and embeds harmful questions in the video to construct an evaluation subset under visual attack (ViSA-Eval-VA). The above hierarchical subsets are used to test the performance of the model, which not only evaluates the defense capability of the model under high-risk input, but also examines its general performance in actual tasks. In addition, the remaining data is used to train the model.

[0049] 2. Model defense part.

[0050] The model defense part covers risk identification and strategy optimization, and a two-stage post-training strategy is designed accordingly, with a framework as shown in Figure 3 The two parts are Context-Aware Risk Detection Token Fine-Tuning (CARDT-FT) and Dynamic Adaptive Group Relative Policy Optimization (DA-GRPO).

[0051] Figure 3An example of two-stage post-training is provided, taking the VideoLLaMA3 model as an example, which is a video large language model open sourced by Alibaba DAMO Academy; the left part is the CARDT-FT stage, the visual encoder, the large language model and the hidden layer involved belong to the VideoLLaMA3 model, their working principles can refer to the conventional technology, and the present application will not be repeated; the right side is the DA-GRPO stage, wherein the cold start means that only the standard language modeling loss (Autoregressive loss) is used for training, so that the model has the output of a specific format answer; the characters in the second dashed box at the top are labels used for supervision in training, which are the structure and content of the ideal model response. think is the thinking chain process, answer is the answer to the question, vidType is the probability of harmfulness of video content, textType is the probability of harmfulness of text, and crossType is the probability of joint harmfulness, which correspond to 、 、 .

[0052] (1) CARDT-FT stage (first stage).

[0053] In the CARDT-FT stage, a set of trainable risk detection tokens are embedded in the feature sequence of the original video and text, respectively, for perceiving potential risk signals in video and text. Compared with the traditional way of inserting pseudo tokens at the head and tail of the sequence, the present application embeds risk detection tokens in specific positions of the token sequence, so that they can fully participate in the semantic fusion process. Specifically, we insert risk detection tokens at the tail of the visual token and the tail of the language token. In addition, the present application introduces a context attention mechanism to enhance the perception ability of the risk detection token, so that the model can more accurately identify potential risk areas when facing complex video-text inputs, and form a closer risk linkage expression between different modalities.

[0054] Specifically, a trainable risk detection token is inserted in the representation sequence of the visual modality and the text modality, respectively, denoted as: ; wherein, is a real set symbol, and d is the dimension.

[0055] The original modality sequence is: ; ; wherein, , correspond to the visual modality token sequence and the text modality token sequence, , Corresponding to the single token in the visual modality token sequence, the text modality token sequence, i = 1, 2, …, N, j = 1, 2, …, M, N, M corresponds to the length of the visual modality token sequence, the text modality token sequence.

[0056] In order to enhance the sensitivity of the token to the context, the risk detection token is fused with the context feature through a lightweight attention mechanism, and is updated to And : ; ; Wherein, and are the key mapping layers of vision and text, and are the value mapping layers of vision and text.

[0057] Wherein, the calculation method of attention is: ; ; Wherein, denotes and , K denotes and , V denotes and , T is the transpose symbol, d is the dimension of q, is the attention weight.

[0058] During training, the model needs to optimize three losses. First, the standard language modeling loss ; second, the risk identification task of the two risk detection tokens, in the form of binary classification loss: ; ; Wherein, indicates the modality risk label, which indicates whether it is harmful; and are two linear layers for dimension reduction, which will reduce the subsequent and to 1 dimension, is the sigmoid function (sigmoid function), and BCE is the binary cross entropy function. and are the risk identification loss terms of the visual modality and the text modality respectively, and the sum of the two is the risk identification loss .

[0059] To enhance the alignment effect between the risk detection token and the real context, a context consistency loss is also introduced, which is defined as follows: ; ; ; ; Wherein, , The corresponding represents the mean value of the visual modal token sequence, the text modal token sequence (that is, the mean value of the original token), and cos is the cosine similarity; And are the context consistency loss terms of the visual modal and the text modal, and the sum of the two is the context consistency loss .

[0060] The final total loss is: ; Wherein, And are adjustable hyperparameters, for example, both can be set to 1.0.

[0061] (2) DA-GRPO phase (second phase).

[0062] The application proposes a safety reward driven dynamic strategy optimization method to enhance the safety response ability, training stability and strategy generalization of the video large language model in complex multi-modal risk scenarios.

[0063] The method first uses the intra-group sorting method to strengthen the training of the model output. Specifically, for each group of input data , the fine-tuned video large language model generates multiple responses The person skilled in the art can understand that the response is the answer of the model, specifically: the token in the hidden space of the model is converted into a response token, each response token is a word probability distribution, and the word with the highest probability in each token is selected to form a response.

[0064] Each response is given a safety preference score by a rule-based scorer The scorer judges according to whether the response rejects harmful requests, language fluency, expression neutrality and other dimensions, and returns a relative score instead of an absolute label. The scorer can be formulated as: ; Wherein, is the format matching reward, is the content matching reward.

[0065] A relative ranking reward between response pairs is then constructed to drive reinforcement learning policy optimization.

[0066] On this basis, the application introduces two key enhancement mechanisms to improve the optimization effect: First, in order to make the policy optimization more stable and reliable, the application introduces a confidence adjustment term on the basis of the original reward difference , defined as the inverse function of the token-level average probability entropy of the response, to represent the stability of the generated content. That is: ; ; Where exp is the natural exponential function, represents the generation probability of the jth word of the tth response token in the response , and is the response length.

[0067] Therefore, the safety preference score is adjusted to : .

[0068] This confidence adjustment term dynamically adjusts the training weight of the response during optimization, suppressing the gradient contribution of unstable language output samples, thereby improving the policy convergence quality.

[0069] Second, the application proposes a reward enhancement mechanism that fuses modal complementarity, which is used to guide the model to fully utilize the information difference between visual and textual modalities, and improve the perception ability of potential risks. Specifically, for each response , a modal complementarity gain term is introduced on the basis of the original reward to form a new safety preference score : ; Where is the modal complementarity reward of the response, defined as: ; Where represents the rejection probability of the response under the joint input of vision and text, and are the rejection probabilities under the visual and text inputs only, m is the complementarity threshold, and is the reward gain weight coefficient. This mechanism encourages the model to exhibit information sensitivity beyond single modal under multi-modal fusion input, so that it can more reliably identify potential cross-modal induced attacks.

[0070] Combining the above two key enhancement mechanisms, the final safety preference score is: .

[0071] The strategy optimization mechanism improves the training stability by confidence adjustment while comparing ideas within the reserved group, and enhances the cross-modal reasoning ability by complementary reward of modal, so that the defense ability of the model is more controllable and generalizable as a whole, and is suitable for complex and multi-source video input scenes in actual deployment.

[0072] 3. Actual deployment of the scheme.

[0073] In terms of deployment, in order to facilitate system integration, the present application supports encapsulating the entire defense strategy (i.e. the data set and the two-stage post-training strategy) as an independent module, which is inserted into the existing video large language model architecture without modifying the basic encoder and language model structure. At the same time, interfaces compatible with external security review modules (such as OCR detectors, prompt word filters) are retained to support multi-channel joint defense deployment in security-sensitive scenarios. In the inference process, if the risk detection token activation degree exceeds the threshold, the model will automatically enter the "security response mode", such as outputting warning information, refusing to answer, turning to neutral generation, etc.

[0074] Overall, the present application establishes a complete technical chain around "security assessment - perception fine-tuning - strategy optimization - control deployment", effectively improving the risk identification ability and defense judgment stability of the video large language model in complex contexts without increasing the complexity of the model structure. Compared with traditional image security mechanisms, the present application is more suitable for processing dynamic, continuous and semantically ambiguous video inputs, and has good cross-modal adaptability and engineering landing potential.

[0075] III. Performance description

[0076] The performance of the present application scheme is introduced by taking the existing video large model as an example. Here, the VideoLLaMA3 model is taken as an example for introduction.

[0077] When the present application is applied to VideoLLaMA3, the trained VideoLLaMA3 achieves a defense success rate of 87.5% on ViSA-Eval, which is 68.3% higher than the initial defense success rate of 19.2%. In addition, the trained VideoLLaMA3 improves by 81.1% (12.6% to 93.7%) on ViSA-Eval-VA, which proves that the present application has strong robustness and can resist special visual attacks.

[0078] To verify the generalization of the method, the trained VideoLLaMA3 is tested on the image safety evaluation dataset MM-SafetyBench and VLGuaRD, which improves by 55.3% (32.1%→87.4%) and 40.2% (44.5%→84.5%) respectively.

[0079] Finally, to verify that the trained VideoLLaMA3 still has strong video understanding ability, we test it on the video dataset MVBench and VideoMME, which decreases by 0.5% (65.5%→65.0%) and 0.3% (59.5%→59.2%) respectively. The results show that the present application significantly improves the safety of the model while almost not sacrificing the video understanding performance, indicating that the model trained by the present application can maintain normal understanding and generation ability.

[0080] Through the description of the above implementation, those skilled in the art can clearly understand that the above examples can be implemented by software, or by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the above examples can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0081] Embodiment two The present application also provides a video large language model safety evaluation and defense system, which is mainly used to implement the method provided by the above embodiment, as shown in the figure, the system mainly includes: Figure 4 A dataset construction unit is configured to construct a multi-category, multi-language and multi-modal video risk evaluation dataset. A training unit is configured to train the video large language model using a two-stage post-training strategy, including: in a first stage, inputting the training data in the video risk evaluation dataset into the video large language model, inserting risk detection tokens in the original tokens of the training data extracted by the video large language model, and performing a risk identification task based on the risk detection tokens by the video large language model, and calculating a corresponding risk identification loss, and calculating a context consistency loss combining the risk detection tokens and the original tokens, constructing a training loss of the first stage combining all the calculated losses, and fine-tuning the video large language model; in a second stage, inputting the pre-grouped training data into the fine-tuned video large language model, and scoring the responses output by the fine-tuned video large language model, and using the scores to construct a relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model.​ The security assessment and defense unit is used to input the video and query text into the optimized video language model and insert risk detection tokens into the original tokens extracted by the video language model. During the reasoning process of the video language model, it perceives whether the video and query text contain harmful elements based on the risk detection tokens and outputs a normal response or a safe response.

[0082] The technical details involved in the above system have been introduced in detail in the previous embodiment, so they will not be repeated here.

[0083] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0084] Example 3 The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0085] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0086] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example: The input device can be a touch screen, image acquisition device, physical button or mouse; The output device may be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0087] Example 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0088] The readable storage medium in the embodiments of the present application can be arranged in the foregoing processing device as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0089] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of the present application is merely intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known by the person skilled in the art.

Claims

1. A video large language model security evaluation and defense method, characterized in that, The method comprises the following steps: constructing a multi-category, multi-language and multi-modal video risk assessment data set; training a video large language model, comprising: in a first stage, inputting training data in the video risk assessment data set into the video large language model, inserting a risk detection token into original tokens extracted by the video large language model, performing a risk identification task based on the risk detection token by the video large language model, calculating a risk identification loss corresponding to the risk identification task, calculating a context consistency loss combining the risk detection token and the original token, constructing a training loss of the first stage combining all the calculated losses, and fine-tuning the video large language model; in a second stage, inputting pre-grouped training data into the fine-tuned video large language model, scoring responses output by the fine-tuned video large language model, using the scores to construct a relative ranking reward between response pairs, and performing reinforcement training on the fine-tuned video large language model to obtain an optimized video large language model; inputting a video and a query text into the optimized video large language model, inserting a risk detection token into original tokens extracted by the video large language model, and outputting a normal response or a safe response based on the risk detection token in the inference process of the video large language model.

2. The method of claim 1, wherein, The method for constructing a multi-category, multi-language and multi-modal video risk assessment data set comprises the following steps: collecting original video clips in multiple languages from multiple public platforms, and preliminarily filtering the video clips through community risk labels of multiple categories and platform content review specifications; then, using a generative model in combination with manual review, constructing a text query matched with each preliminarily filtered video to form a video and query text combination pair, and finally forming a video risk assessment data set.

3. The method of claim 1, wherein, The method for inserting a risk detection token into original tokens extracted by the video large language model comprises the following steps: The training data is a video and query text combination pair, and the original token contains a visual modality token sequence and a text modality token sequence , denoted as: , ; wherein, , correspond to individual tokens in the visual modality token sequence and the text modality token sequence, i = 1, 2, …, N, j = 1, 2, …, M, N and M correspond to the lengths of the visual modality token sequence and the text modality token sequence. The risk detection token comprises two, denoted as With ; the risk detection token is inserted into the visual modality token sequence, and the risk detection token is inserted into the text modality token sequence.

4. The method of claim 3, wherein, The method further comprises the following steps: updating the risk detection token using an attention mechanism, and then inserting the updated risk detection token into a visual modality token sequence and a text modality token sequence; Wherein, the risk detection token With After updating, the corresponding is denoted as With The way of updating the risk detection token by adopting the attention mechanism is expressed as: ； ； wherein Attn is an attention mechanism, and is a key mapping layer for vision and text, and is a value mapping layer for vision and text.

5. The method of claim 1 or 3 or 4, wherein, The method for constructing a training loss of the first stage combining all the calculated losses comprises the following steps: combining the computed risk identification loss with the contextual consistency loss, as well as a standard language modeling loss The training loss L for the first stage is constructed and is represented as: ； wherein, with is an adjustable hyperparameter; is a risk identification loss, calculated using the cosine similarity of the risk detection token and the mean of the original tokens; is a context consistency loss, calculated using the mean of the risk detection token and the original tokens.

6. The method of claim 1, wherein, The method for inputting pre-grouped training data into the fine-tuned video large language model and scoring responses output by the fine-tuned video large language model comprises the following steps: The first response of the fine-tuned video large language model output is recorded as The responses of the group of training data are recorded as Each response corresponds to a single training data in the first group of training data, The total number of training data of the first group of training data is recorded as assigning a safety preference score to the video large language model by a scorer, and the safety preference score is represented as: ； wherein, in response to a security preference score, in response to refers to any response in the set of responses; is a format match reward, is a content match reward.

7. The method of claim 6, wherein the method further comprises: The method further comprises the following steps: Introducing an enhancement mechanism to calculate the final security preference score, including: introducing a confidence adjustment term , defined as the inverse function of the token-level average probability entropy in response , to represent the stability of the generated content; and introducing a reward enhancement mechanism for fusion modal complementarity, adding a modal complementarity gain term to the security preference score , then the final security preference score is: ； where, is the reward gain weight coefficient, is the response modal complementarity reward, defined as: , , and are the responses to the information in denotes the probability of rejection of the response under the joint visual and textual input, and are the rejection probabilities under visual and textual input only, respectively, and m is the complementarity threshold.

8. A video large language model security evaluation and defense system, characterized in that, The method for implementing any one of claims 1-7 comprises the following steps: a data set construction unit configured to construct a multi-category, multi-language and multi-modal video risk assessment data set; A training unit is used to train a video language model, including: in a first stage, inputting training data from a video risk assessment dataset into the video language model, inserting risk detection tokens into the original tokens of the training data extracted by the video language model, and having the video language model perform a risk identification task based on the risk detection tokens, calculating the corresponding risk identification loss, and calculating the context consistency loss by combining the risk detection tokens with the original tokens, combining all the calculated losses to construct the training loss of the first stage, and fine-tuning the video language model; in a second stage, inputting pre-grouped training data into the fine-tuned video language model, scoring the responses output by the fine-tuned video language model, using the scores to construct relative ranking rewards between response pairs, and performing intensive training on the fine-tuned video language model to obtain an optimized video language model; The security assessment and defense unit is used to input the video and query text into the optimized video language model and insert risk detection tokens into the original tokens extracted by the video language model. During the reasoning process of the video language model, it perceives whether the video and query text contain harmful elements based on the risk detection tokens and outputs a normal response or a safe response.

9. A processing device, characterized by include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium, storing a computer program, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images