A Multimodal Large-Model Security Command Fine-Tuning Method and System Based on Virtual Adversarial Testing

Abstract

This invention discloses a method and system for fine-tuning secure instructions for a multimodal large model based on virtual adversarial techniques, belonging to the field of large models. This invention performs two-stage fine-tuning of the multimodal large model: the first stage uses gradient-based virtual adversarial perturbation technology to securely align the large language model and cross-modal connection layers, ensuring the harmless nature of the model's output content and improving its resistance to subsequent malicious data; the second stage, with the parameters fixed after secure alignment, introduces a trainable implicit prompt word structure, achieving effective adaptation to user-personalized data only by adjusting its key-value pairs, preventing the reintroduction of harmful information during fine-tuning. This invention reduces dependence on preference data while achieving synergistic optimization of security and personalization needs, providing a reliable guarantee for the practical deployment of multimodal large models.

Description

Technical Field

[0001] This invention belongs to the field of large-scale models, and in particular relates to a method and system for fine-tuning security instructions for multimodal large-scale models based on virtual adversarial scenarios. Background Technology

[0002] The rapid development of multimodal large models has revolutionized the field of artificial intelligence, enabling machines to process and understand complex data across multiple modalities, such as text, images, and audio. These models, pre-trained on massive datasets, have demonstrated a strong ability to learn general cross-modal representations. However, removing toxic data present in the pre-training data remains a costly and difficult task, potentially leading to unethical outputs from pre-trained large models when given specific image or text prompts. This can have a significant negative social impact in the context of widespread multimodal large models. Therefore, a challenging research question is how to reduce harmful outputs while maintaining high accuracy in multimodal large models. Existing safe alignment methods include safe instruction fine-tuning and human feedback reinforcement learning. However, the former is prone to re-emergence of harmful content during subsequent downstream task fine-tuning, while the latter relies heavily on human preferences for different responses and performs poorly when data is scarce.

[0003] Meanwhile, considering the different distributions of pre-training data and target task data, the pre-trained multimodal large models, while capable of capturing general features, cannot be directly adapted to specific tasks. Therefore, if these pre-trained models are to be adapted to specific downstream tasks, it is usually necessary to perform safety instruction fine-tuning on task-related labeled data. Existing safety instruction fine-tuning methods have made significant progress, covering a variety of techniques, including full model fine-tuning, partial fine-tuning, adapter-based fine-tuning, and contrastive learning-assisted fine-tuning. Each method has unique advantages in terms of computational efficiency, parameter optimization, and task-specific adaptation. However, safety instruction fine-tuning still faces challenges such as data scarcity, modality imbalance, and overfitting risks, including the toxic data intrusion problem that this invention focuses on.

[0004] Modern multimodal model applications increasingly focus on generating user-personalized models, such as using hyperparameter networks to serve different users. Considering the following application scenario, internet businesses, without understanding the users' personalized fine-tuning data, first perform a safety alignment on a pre-trained multimodal model using specific data to significantly improve the model's safety. At this point, the model can be considered to still possess some toxicity, but requires deeper harmful guidance before alignment. Then, the safety-aligned model is provided to users. Users can upload personalized data conforming to the vendor's format to fine-tune the model to suit their specific downstream tasks. During the fine-tuning stage, users may intentionally or unintentionally inject small amounts of toxic data. This data can cause the originally aligned model to shift or re-establish an incorrect alignment, leading to a deep "leak" of toxicity—that is, outputting harmful responses with simple guidance. From another perspective, to prevent harmful output, the toxic model needs more accurate prompts to guide it. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing post-training multimodal large model secure alignment techniques, when faced with downstream fine-tuning tasks involving toxic data intrusion, can lead to significant deviations in the fine-tuned multimodal model compared to the securely aligned model, or even the re-establishment of incorrect alignments. This results in the toxicity "leaking" from deeper layers, meaning that even simple error prompts can lead to harmful responses. To address this, this invention proposes a multimodal large model secure instruction fine-tuning method and system based on virtual adversarial mechanisms to strengthen the secure alignment process and mitigate the toxicity "leakage" problem.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for fine-tuning security instructions in a multimodal large-scale model based on virtual adversarial tactics, comprising the following steps:

[0008] S1. Perform the first fine-tuning on the pre-trained multimodal large model that needs to be securely aligned to obtain a fully aligned multimodal large model. The pre-trained multimodal large model includes a large language model, a cross-modal connection layer, a visual encoder, and a text encoder. The first fine-tuning includes two complete forward and backward propagations: through the first complete forward and backward propagation, the gradients of the parameters in each network layer of the large language model are obtained, while the parameters of the multimodal large model are not updated. During the second forward propagation, the gradients of the parameters are scaled and added to the vector representation of each network layer as virtual adversarial perturbations. The parameters of the large language model and the cross-modal connection layer are updated through the second backward propagation. The parameters of the visual encoder and the text encoder remain fixed throughout the two fine-tuning processes.

[0009] S2. After the first fine-tuning, fix the parameters of the large language model and the cross-modal connection layer, introduce a trainable implicit prompt word, and perform a second fine-tuning on the safe-aligned multimodal large model on user personalized data. In the second fine-tuning, only the parameters in the implicit prompt word are updated. The implicit prompt word consists of several key-value pairs. Until the preset maximum number of iterations is reached, the fine-tuned multimodal large model is obtained.

[0010] Based on the above scheme, each step can be implemented in the following preferred manner.

[0011] As a preferred option for the first aspect mentioned above, both the secure alignment data and the user-personalized data are in standard triplet format, containing images and corresponding natural language questions and standard answers. The standard answers will be used as real labels to calculate the loss function of the multimodal large model.

[0012] As a preferred embodiment of the first aspect, in the first forward propagation of S1, the natural language problem in the securely aligned data is first converted into a text embedding by the text encoder. The visual encoder then generates a visual embedding based on the image in the securely aligned data. The visual embedding passes through a cross-modal connection layer, which maps the visual embedding to the text embedding space corresponding to the natural language problem. Thus, the text embedding and the visual embedding are concatenated to form a multimodal embedding, which is added to the positional encoding and used as the input of the large language model.

[0013] As a preferred embodiment of the first aspect, in the second forward propagation of S1, after introducing virtual adversarial perturbation into the vector representation of each layer of the large language model, the vector representation of the output of that layer is composed of two parts added together. The first part is the result of summing the vector representation of the input of that layer with the parameters of that layer element by element. The second part is the product of the virtual adversarial perturbation applied by that layer and the partial derivative of the vector representation. The partial derivative of the vector representation is the partial derivative of the vector representation of the output of that layer with respect to the parameters of that layer.

[0014] Furthermore, the first Vector representation of layer output Represented as:

[0015]

[0016] in, This indicates element-wise summation; For the first Layer parameters; For the first Vector representation of layer output; For the first Virtual adversarial perturbation of the layer; For the first Vector representation of layer output parameters of this layer The partial derivatives; Indicates the first Layer network layer parameters The partial derivatives; Indicates the first Layer network layer vector representation The partial derivatives of .

[0017] As a preferred embodiment of the first aspect above, the virtual adversarial perturbation of each layer is the product of a preset hyperparameter and a unit vector in the gradient direction. The unit vector is obtained by dividing the gradient of that layer obtained after the first complete forward and backward propagation by the magnitude of the gradient of that layer. The magnitude of the virtual adversarial perturbation is less than or equal to the hyperparameter.

[0018] Furthermore, the first Virtual adversarial perturbation of layers Represented as:

[0019]

[0020] in, express The modulus length; Indicates the first Gradient of layer parameters; Represents the loss function The partial derivatives; These are the preset hyperparameters.

[0021] As a preferred embodiment of the first aspect mentioned above, in the second fine-tuning of S2, the natural language problem in the user-personalized data is converted into a text embedding by the text encoder, which serves as the text feature vector. The image in the user-personalized data is processed sequentially by the visual encoder and the cross-modal connection layer, and the mapped visual embedding is used as the visual feature vector. By calculating the similarity between the visual feature vector and all key-value pairs, the text is extracted. The key-value pairs with the highest similarity are selected as target key-value pairs. These target key-value pairs are then fused with text feature vectors and visual feature vectors to form the input sequence for the large language model. The target number of key-value pairs.

[0022] As a preferred option of the first aspect mentioned above, in the second fine-tuning of S2, the optimization objective of the multimodal large model includes simultaneously maximizing the probability distribution of the true labels and the similarity constraint term. The similarity constraint term is obtained by multiplying the preset weight coefficient with the global similarity. The global similarity is the sum of the similarities between the key in the target key-value pair and the visual feature vector.

[0023] Furthermore, the optimization objective of the multimodal large model is:

[0024]

[0025] in, Indicates an implicit prompt; Indicates maximization; Represents the probability distribution of the true labels; This is a real label; These are visual feature vectors and text feature vectors, respectively. Represents a logarithmic function; For similarity constraints; This indicates the preset weighting coefficients; For global similarity; For similarity calculation; Indicates calculation and Similarity; Indicates the returned first The key in a key-value pair; Indicates the returned first Each key index.

[0026] Secondly, the present invention provides a multimodal large-scale question-answering system based on virtual adversarial interaction, comprising:

[0027] The model acquisition module is used to acquire the virtual adversarial-based multimodal large model security instruction fine-tuning method according to any of the above-mentioned first aspect schemes, fine-tune the pre-trained multimodal large model, and thus obtain the fine-tuned multimodal large model;

[0028] The answer generation module is used to obtain multimodal instructions sent by the user. The multimodal instructions consist of a target image and a question text corresponding to the target image. After the multimodal instructions are input into the fine-tuned multimodal large model, the answer text corresponding to the multimodal instructions is output and returned to the user.

[0029] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal large-scale security instruction fine-tuning method based on virtual adversarial as described in any of the solutions of the first aspect above.

[0030] Fourthly, the present invention provides a computer electronic device, including a memory and a processor;

[0031] The memory is used to store computer programs;

[0032] The processor is configured to, when executing the computer program, implement the virtual adversarial multimodal large-scale security instruction fine-tuning method as described in any of the first aspects above.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] This invention utilizes a virtual adversarial multimodal large model secure alignment method to ensure the harmlessness of the model's output content, while enhancing the large model's resistance to subsequent toxic fine-tuning data. Furthermore, compared to human feedback reinforcement learning, it reduces the need for preference data, enabling secure alignment tasks to be completed even under data-scarce conditions. This invention incorporates implicit cue word structures into the fine-tuning of the multimodal large model when user-personalized fine-tuning data is unknown, preventing the inclusion of toxic data in the fine-tuning data and avoiding the situation where the securely aligned large model re-outputs harmful content. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method of the present invention;

[0036] Figure 2 This is a flowchart illustrating the gradient process for obtaining secure alignment data in the method of this invention.

[0037] Figure 3 The flowchart for introducing virtual adversarial perturbation to achieve secure alignment of multimodal large models in the method of this invention is as follows:

[0038] Figure 4 This is a flowchart of the second fine-tuning of the multimodal large model in the method of the present invention;

[0039] Figure 5 This is a system block diagram of the present invention;

[0040] Figure 6 This is a schematic diagram of the components of a computer electronic device according to the present invention. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0042] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0043] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned multimodal large-scale security instruction fine-tuning method based on virtual adversarial tactics includes the following steps S1 to S2. The specific implementation process of each step will be described in detail below.

[0044] S1. Perform the first fine-tuning on the pre-trained multimodal large model that needs to be securely aligned to obtain a securely aligned multimodal large model. The pre-trained multimodal large model includes a large language model, a cross-modal connection layer, a visual encoder, and a text encoder. The first fine-tuning includes two complete forward and backward propagations: through the first complete forward and backward propagations, the gradients of the parameters in each network layer of the large language model are obtained, while the parameters of the multimodal large model are not updated. During the second forward propagation, the gradients of the parameters are scaled and added to the vector representation of each network layer as virtual adversarial perturbations. The parameters of the large language model and the cross-modal connection layer are updated through the second backward propagation. The parameters of the visual encoder and the text encoder remain fixed throughout the two fine-tuning processes.

[0045] It should be noted that in S1 of this invention, the secure alignment data is in standard triplet format, which may include an image and a natural language question and standard answer corresponding to the image. The standard answer will be used as the real label to calculate the loss function of the multimodal large model.

[0046] It should be noted that in S1 of this invention, the pre-trained multimodal large model can be selected from various sources, such as the Qwen series of Large Vision Language Models (LVLMs) – Qwen-VL-Chat, or the existing LLaVA-v1.5-7B model. Alternatively, a suitable network structure can be designed by those skilled in the art as needed, as long as it is based on the Transformer architecture and contains a visual encoder, a large language model, and cross-modal connection layers, capable of generating corresponding answers based on the input natural language question. Furthermore, the method of this invention is independent of the multimodal large model architecture; therefore, the structure of the multimodal large model is not limited in this invention.

[0047] It should be noted that in S1 of this invention, the method for obtaining the gradient of each network layer through the first complete forward and backward propagation is as follows:

[0048] like Figure 2As shown, in the first forward propagation, the natural language question in the securely aligned data is first converted into a text embedding by the text encoder. The visual encoder then generates a visual embedding based on the image in the securely aligned data. The visual embedding passes through a cross-modal connection layer, which maps the visual embedding to the text embedding space corresponding to the natural language question. This concatenates the text embedding and the visual embedding to form a multimodal embedding. and with position encoding Add them together to get the initial input. The initial input is processed through the first layer of the large language model. Other layers use the vector representation output from their preceding layers as input. Each layer utilizes operations common in large language models, such as self-attention, residual connections, and layer normalization, within the Transformer architecture to calculate its output vector representation. After processing by all layers of the large language model, the vector representation output from the last layer is passed through a linear layer and a softmax function to predict the probability distribution of the next word. Finally, the loss function is used. To measure the probability distribution of the prediction With real labels The differences between them.

[0049] During the first backpropagation process, the chain rule is mainly used to efficiently calculate the loss function for all parameters of the large language model. The gradient is calculated. Specifically, after obtaining the total loss, starting from the output layer of the network, the gradient is calculated layer by layer and module by module in the opposite direction to the forward propagation. Taking the first layer of a large language model as an example... Taking a layered network layer as an example, the parameters of this layer are... gradient This is equal to the vector representation output by that layer. Partial derivatives of the parameters of this layer Multiply by the partial derivative of the loss function representing the vector ,Right now:

[0050]

[0051] in, Represents the loss function The partial derivatives; Indicates the first Layer network layer parameters The partial derivatives; Indicates the first Layer network layer vector representation The partial derivatives of .

[0052] In large language models, this chain rule applies to self-attention (calculating the gradients of the Query, Key, and Value matrices), feedforward networks (calculating the gradients of the weight matrix and bias terms), and layer normalization (calculating the gradients of the weight matrix and bias terms). and The gradient of the gradient is calculated and derived, which will not be elaborated here.

[0053] It should be noted that in the second forward propagation of S1 of this invention, after introducing virtual adversarial perturbation into the vector representation of each layer of the large language model, the vector representation of the output of that layer is composed of two parts added together. The first part is the result of summing the vector representation of the input of that layer with the parameters of that layer element by element. The second part is the product of the virtual adversarial perturbation applied by that layer and the partial derivative of the vector representation. The partial derivative of the vector representation is the partial derivative of the vector representation of the output of that layer with respect to the parameters of that layer.

[0054] like Figure 3 As shown, in the second forward propagation process of this embodiment, a virtual adversarial perturbation needs to be added to the input of each network layer of the large language model. Taking the large language model's... Taking the layered network layer as an example, the virtual adversarial perturbation is denoted as Then for the first After adding perturbation to the layer, the first The vector representation of the layer output can be expressed as , For the first The vector representation of the layer output. Due to considerations It is a tiny perturbation that is approximately zero. According to the Taylor expansion, it can be approximated as:

[0055]

[0056] in, This represents element-wise summation. Based on the vector representation after each level of approximation, we can obtain the following using the recurrence relation:

[0057]

[0058] in, , , The first Layer, First Layer, First Vector representation of layer output; , The first Layer, First The corresponding virtual adversarial perturbation of the layer; , , The first Layer, First Layer, First Layer parameters; Indicates the first Layer parameters The partial derivatives; Indicates the first Layer parameters The partial derivatives; Indicates the first The vector output by the layer represents the partial derivatives with respect to the layer parameters; Indicates the first The vector representation of the layer output for the th Partial derivatives of layer parameters; Indicates the first The vector representation of the layer output for the th Partial derivatives of layer parameters; This represents the number of network layers in a large language model.

[0059] Therefore, the predicted probability distribution With real labels loss function between The final representation is as follows:

[0060]

[0061]

[0062] in, This represents the Softmax function; This indicates a linear layer.

[0063] Furthermore, the virtual adversarial perturbation of each layer is the product of a preset hyperparameter and a unit vector in the gradient direction. The unit vector is obtained by dividing the gradient of that layer obtained after the first complete forward and backward propagation by the magnitude of the gradient of that layer.

[0064] In this embodiment, an optimal virtual adversarial perturbation aims to disrupt the final loss function, thereby maximizing its value. However, considering the virtual adversarial perturbation... The influence of the magnitude on the loss function is determined by this invention using a hyperparameter. and demand , Let represent the magnitude of the virtual adversarial perturbation. Under the above conditions, the following conclusion can be derived:

[0065]

[0066] The above conclusion holds true when the following condition is met:

[0067]

[0068] in, express The modulus length. In the above calculation, The gradients obtained after the first complete forward and backward propagation can be substituted into this result, thus transforming it into an optimal virtual adversarial perturbation, which can then be added to the second forward propagation process. Furthermore, it should be noted that the aforementioned hyperparameters... The value should not be too large; it can generally be selected within the range of 0.01 to 10. In this embodiment, the hyperparameter... Values ​​of 0.01, 0.1, and 1.0 can all achieve satisfactory results.

[0069] In the second backpropagation process of this embodiment, the calculation method of the parameter gradient is basically the same as that of the first backpropagation. The difference is that the second backpropagation requires parameter updates, that is:

[0070]

[0071] in, For the first Layer parameters Updated results; The learning rate set during the safe alignment process.

[0072] S2. After the first fine-tuning, fix the parameters of the large language model and the cross-modal connection layer, introduce a trainable implicit prompt word, and perform a second fine-tuning on the safe-aligned multimodal large model on user personalized data. In the second fine-tuning, only the parameters in the implicit prompt word are updated. The implicit prompt word consists of several key-value pairs. Until the preset maximum number of iterations is reached, the fine-tuned multimodal large model is obtained.

[0073] It should be noted that in S2 of this invention, the user personalized data is also in the standard triplet format, which may include an image and three parts: a natural language question corresponding to the image and a standard answer.

[0074] In this embodiment S2, the second fine-tuning can be understood as a fine-tuning task that may be mixed with toxic data. Considering that the securely aligned multimodal large model can already significantly reduce the probability of outputting harmful content, in the second fine-tuning (i.e., the fine-tuning of the downstream task), this invention freezes the parameters of the securely aligned large language model and the parameters of the cross-modal connection layer to prevent erroneous shifts due to toxic data. Furthermore, this invention achieves fine-tuning by introducing a trainable implicit cue word; that is, the second fine-tuning only updates the relevant parameters in the implicit cue word, such as... Figure 4 As shown. The above implicit prompts. Include There are 1 key-value pairs, respectively , This represents the number of key-value pairs in the implicit prompt. For each key-value pair, The value in each key-value pair.

[0075] It should be noted that in the second fine-tuning of S2 in this invention, the natural language problem in the user personalized data is converted into a text embedding by the text encoder, which serves as the text feature vector. The image in the user personalized data is processed sequentially by the visual encoder and the cross-modal connection layer. The mapped visual embedding is used as the visual feature vector. By calculating the similarity between the visual feature vector and all key-value pairs, the text is extracted. The key-value pairs with the highest similarity are selected as target key-value pairs. These target key-value pairs are then fused with text feature vectors and visual feature vectors to form the input sequence for the large language model. The target number of key-value pairs.

[0076] In the second fine-tuning of S2 in this embodiment, for the image in the user's personalized data, after passing through a specific visual encoder and a cross-modal connection layer, a visual feature vector is obtained. ,in It is the sequence length of images in a multimodal large model. This refers to the hidden layer dimension. Similarly, the natural language questions in user-personalized data will be transformed into text embeddings, serving as text feature vectors. ,in This refers to the sequence length in a multimodal large-scale natural language processing problem. For each key-value pair in the implicit prompt, the similarity between the key and the visual feature vector is calculated sequentially, and the result is returned. The key index with the highest similarity is selected, and the key-value pairs corresponding to these key indices are used as the target key-value pairs, as shown in the following formula:

[0077]

[0078]

[0079] in, Indicates the returned A set consisting of key indexes; The returned numbers are respectively the first and second numbers. Each key index; Indicates the returned first Each key index; This indicates that the returned key index range is within a closed interval. Inside; This represents a function used to find the key index that results in the highest similarity. For similarity calculation; Indicates calculation and Similarity; Represents a set consisting of target key-value pairs; Then it means the returned number is... The nth key-value pair, i.e., the nth Target key-value pairs.

[0080] After obtaining the target key-value pairs, the values ​​in these key-value pairs are fused with the visual feature vector and the text feature vector. In this embodiment, feature fusion is achieved by concatenation, and the resulting sequence serves as the input sequence for the subsequent large language model. , This indicates a splicing operation.

[0081] Furthermore, the above similarity calculation can employ similarity measurement methods such as cosine similarity and Manhattan distance, which are not limited in this invention.

[0082] It should be noted that in the second fine-tuning of S2 in this invention, the optimization objective of the multimodal large model includes simultaneously maximizing the probability distribution of the true labels and the similarity constraint term. The similarity constraint term is obtained by multiplying the preset weight coefficient with the global similarity. The global similarity is the sum of the similarities between the key in the target key-value pair and the visual feature vector.

[0083] In this embodiment, the optimization objective of the multimodal large model is:

[0084]

[0085] in, Indicates maximization; Represents the probability distribution of the true labels; Represents a logarithmic function; For similarity constraints; This indicates the preset weighting coefficients; This represents the global similarity.

[0086] To better demonstrate the specific implementation and technical effects of the present invention, the method for fine-tuning multimodal large-scale security instructions based on virtual adversarial tactics shown in steps S1 to S2 of the above preferred implementation is applied to a specific example.

[0087] Example

[0088] The specific implementation process of the multimodal large-scale security instruction fine-tuning method based on virtual adversarial tactics used in this embodiment is as described above and will not be repeated here.

[0089] This embodiment verifies the implementation results on the publicly available multimodal security dataset SPA-VL, the multimodal question answering dataset VQA-Med, and GeoVQA.

[0090] The SPA-VL dataset contains 100,788 samples, each consisting of a quadruple: (question, image, selected response, rejected response), where the "selected response" is considered more safe and useful than the "rejected response." This embodiment randomly selects a certain number of samples from the SPA-VL dataset as the training set (1000 samples) and the test set (200 samples) for small-scale testing. Triples of (question, image, selected response) from the training set are extracted for the first fine-tuning of the multimodal large model's safe alignment, while triples of (question, image, rejected response) are considered harmful data.

[0091] The VQA-Med and GeoVQA datasets focus on image diagnostics, medical image understanding, and geometric understanding. This embodiment uses image diagnostics and geometric understanding as two specific downstream fine-tuning tasks, extracting 500 samples for fine-tuning the multimodal large model, and incorporating different proportions of harmful data. Additionally, 200 different samples are extracted as a test set.

[0092] For the specific test model, this embodiment selects LLaVA-v1.5-7B as the above-mentioned multimodal large model, whose visual encoder adopts CLIP's ViT-L / 14 model, and the large language model is based on Vicuna-7B (LLaMA architecture).

[0093] To evaluate the performance of the method of this invention, for the two specific downstream tasks mentioned above, this embodiment uses MD-Judge-v0_2 to evaluate the fine-tuned multimodal large model. The model outputs the scores of the answers on the SPA-VL test set. Specifically, MD-Judge-v0_2 scores the harmful question-answer pairs generated by the fine-tuned multimodal large model from 1 to 5, where 1 is completely harmless and 5 is extremely harmful. The average score is then used as the result to verify the harmlessness of the model. Simultaneously, the F1 score of the fine-tuned multimodal large model on the specific downstream task test set is calculated to ensure accuracy in the downstream tasks.

[0094] The results of the first fine-tuning for secure alignment are shown in Table 1. The harmfulness score is used to test the degree of toxicity "leakage" of the model under the incorporation of toxic data. The higher the harmfulness score, the more the multimodal large model leaks and the more "harmful" it is. As can be seen from Table 1, the multimodal large model with secure alignment obtained by the method of this invention outputs less harmful content. In addition, due to the small-scale experiment, the training set sample size is only 1000 samples, and the improvement effect on the multimodal large model may be limited.

[0095] Table 1. Harmfulness scores of different methods

[0096]

[0097] Table 2 shows the fine-tuning results of downstream tasks with toxic data mixed in, using GeoVQA as the downstream task dataset for fine-tuning. It can be seen that, compared to traditional safety instruction fine-tuning methods, the method of this invention does not significantly reduce the performance of the multimodal large model in specific downstream tasks; in fact, it improves performance in most cases, ensuring model accuracy. Furthermore, the method of this invention enables the model to better resist toxic fine-tuned data in specific downstream tasks, reducing toxicity "leakage" to some extent. However, an unexpected phenomenon in the experimental results is that the model fine-tuned with no toxic data (0%) is more harmful than the model with 1% toxic data, and performs worse in specific downstream tasks. This embodiment speculates that the toxic data and the previously safety-aligned data may have similar structures, which to some extent strengthens the model's ability to handle safety-aligned data. Additionally, data from different domains may enhance the model's generalization ability.

[0098] Table 2. Accuracy of different methods on the downstream task dataset GeoVQA

[0099]

[0100] It should also be noted that the virtual adversarial-based multimodal large-scale security instruction fine-tuning method in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a virtual adversarial-based multimodal large-scale question-answering system corresponding to the virtual adversarial-based multimodal large-scale security instruction fine-tuning method provided in the above embodiments, such as... Figure 5 As shown, it includes:

[0101] The model acquisition module is used to acquire the security instruction fine-tuning method for a multimodal large model based on virtual adversarial as described in the above embodiments, fine-tune the pre-trained multimodal large model, and thus obtain the fine-tuned multimodal large model.

[0102] The answer generation module is used to obtain multimodal instructions sent by the user. The multimodal instructions consist of a target image and a question text corresponding to the target image. After the multimodal instructions are input into the fine-tuned multimodal large model, the answer text corresponding to the multimodal instructions is output and returned to the user.

[0103] It is understood that the multimodal large-scale security instruction fine-tuning method based on virtual adversarial scenarios described in S1-S2 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the multimodal large-scale security instruction fine-tuning method based on virtual adversarial scenarios provided in the above embodiments. This product includes a computer program / instruction that, when executed by a processor, can implement the multimodal large-scale security instruction fine-tuning method based on virtual adversarial scenarios as described in the above embodiments.

[0104] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the multimodal large-scale security instruction fine-tuning method based on virtual adversarial scenarios provided in the above embodiments, such as... Figure 6 As shown, it includes a memory and a processor;

[0105] The memory is used to store computer programs;

[0106] The processor is configured to implement the virtual adversarial multimodal large-scale security instruction fine-tuning method in the above embodiments when executing the computer program.

[0107] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0108] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the virtual adversarial-based multimodal large-scale security instruction fine-tuning method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the virtual adversarial-based multimodal large-scale security instruction fine-tuning method in the above embodiments.

[0109] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0110] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0111] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for fine-tuning security instructions in a multimodal large-scale model based on virtual adversarial tactics, characterized in that, Includes the following steps: S1. Perform the first fine-tuning on the pre-trained multimodal large model that needs to be securely aligned to obtain a fully securely aligned multimodal large model. The pre-trained multimodal large model includes a large language model, a cross-modal connection layer, a visual encoder, and a text encoder. The first fine-tuning consists of two complete forward and backward propagations: through the first complete forward and backward propagation, the gradients of the parameters in each network layer of the large language model are obtained to securely align the data, without updating the parameters of the multimodal large model; during the second forward propagation, the gradients of the parameters are scaled and added to the vector representation of each network layer as virtual adversarial perturbations, and the parameters of the large language model and the cross-modal connection layer are updated through the second backward propagation. The parameters of the visual encoder and the text encoder remain fixed throughout the two fine-tuning processes. S2. After the first fine-tuning, fix the parameters of the large language model and the cross-modal connection layer, introduce a trainable implicit prompt word, and perform a second fine-tuning on the safe-aligned multimodal large model on user personalized data. In the second fine-tuning, only the parameters in the implicit prompt word are updated. The implicit prompt word consists of several key-value pairs. Until the preset maximum number of iterations is reached, the fine-tuned multimodal large model is obtained.

2. The method for fine-tuning security instructions for a multimodal large-scale model based on virtual adversarial tactics as described in claim 1, characterized in that, Both the securely aligned data and the user-personalized data are in standard triplet format, containing images and corresponding natural language questions and standard answers. The standard answers will be used as real labels for calculating the loss function of the multimodal large model.

3. The method for fine-tuning security instructions for a multimodal large-scale model based on virtual adversarial tactics as described in claim 2, characterized in that, During the first forward propagation of S1, the natural language question in the securely aligned data is first converted into a text embedding by the text encoder. The visual encoder generates a visual embedding based on the image in the securely aligned data. The visual embedding passes through a cross-modal connection layer, which maps the visual embedding to the text embedding space corresponding to the natural language question. Thus, the text embedding and the visual embedding are concatenated to form a multimodal embedding, which is added to the positional encoding and used as the input to the large language model.

4. The method for fine-tuning security instructions for a multimodal large-scale model based on virtual adversarial tactics as described in claim 1, characterized in that, In the second forward propagation of S1, after introducing virtual adversarial perturbation into the vector representation of the input of each layer of the large language model, the vector representation of the output of that layer is composed of two parts. The first part is the result of summing the vector representation of the input of that layer with the parameters of that layer element by element. The second part is the product of the virtual adversarial perturbation applied by that layer and the partial derivative of the vector representation. The partial derivative of the vector representation is the partial derivative of the vector representation of the output of that layer with respect to the parameters of that layer.

5. The method for fine-tuning security instructions for a multimodal large-scale model based on virtual adversarial tactics as described in claim 4, characterized in that, The virtual adversarial perturbation of each layer is the product of a preset hyperparameter and a unit vector in the gradient direction. The unit vector is obtained by dividing the gradient of that layer obtained after the first complete forward and backward propagation by the magnitude of the gradient of that layer. The magnitude of the virtual adversarial perturbation is less than or equal to the hyperparameter.

6. The method for fine-tuning security instructions for a multimodal large-scale model based on virtual adversarial tactics as described in claim 1, characterized in that, In the second fine-tuning of S2, the natural language problem in the user-personalized data is converted into a text embedding by the text encoder and used as a text feature vector. The images in the user-personalized data are processed by the visual encoder and the cross-modal connection layer in sequence. The mapped visual embedding is used as a visual feature vector. By calculating the similarity between the visual feature vector and all key-value pairs, the K key-value pairs with the highest similarity are selected as target key-value pairs. Then, the target key-value pairs are fused with the text feature vector and the visual feature vector to serve as the input sequence of the large language model. Here, K is the number of target key-value pairs.

7. The method for fine-tuning security instructions for a multimodal large-scale model based on virtual adversarial tactics as described in claim 6, characterized in that, In the second fine-tuning of S2, the optimization objective of the multimodal large model includes simultaneously maximizing the probability distribution of the true labels and the similarity constraint term. The similarity constraint term is obtained by multiplying the preset weight coefficient with the global similarity. The global similarity is the sum of the similarities between the key in the target key-value pair and the visual feature vector.

8. A multimodal large-scale question-answering system based on virtual adversarial interaction, characterized in that, include: The model acquisition module is used to fine-tune the pre-trained multimodal large model according to the virtual adversarial multimodal large model security instruction fine-tuning method as described in any one of claims 1 to 7, thereby obtaining the fine-tuned multimodal large model; The answer generation module is used to obtain multimodal instructions sent by the user. The multimodal instructions consist of a target image and a question text corresponding to the target image. After the multimodal instructions are input into the fine-tuned multimodal large model, the answer text corresponding to the multimodal instructions is output and returned to the user.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the multimodal large-scale security instruction fine-tuning method based on virtual adversarial as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement, when executing the computer program, the multimodal large-scale security instruction fine-tuning method based on virtual adversarial scenarios as described in any one of claims 1 to 7.

Images