Positioning method and device for safe attention head of multi-mode large model

Abstract

The invention discloses a positioning method and device for a safety attention head of a multi-mode large model, and relates to the technical field of computers, and the method comprises the steps: collecting a safety data set and a utility data set; respectively inputting the security data set and the utility data set into the original multi-mode large model, and obtaining original activation matrixes corresponding to the security data set and the utility data set; respectively performing ablation operation on attention heads in the original multi-mode large model to obtain a plurality of ablation multi-mode large models of which the number is the same as that of the attention heads, and respectively obtaining ablation activation matrixes corresponding to the security data set and the utility data set; obtaining a safe attention head set in the attention heads based on an original activation matrix and a plurality of ablation activation matrixes corresponding to the safe data set; acquiring a general attention head set in attention heads based on the original activation matrix and the plurality of ablation activation matrixes corresponding to the utility data set; and taking the set difference as a safety key attention head set. According to the application, the safety key attention head can be accurately positioned.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for locating a multimodal large-scale security attention head. Background Technology

[0002] In recent years, multimodal large models, especially large visual language models, have made significant progress in various cross-modal tasks by integrating visual and textual information processing capabilities.

[0003] In the field of multimodal large model technology, complex attention mechanisms are the core foundation for achieving efficient information processing and secure decision-making. Different attention heads play different functions when the model processes multi-source information such as visual and textual data, and some specific attention heads are key elements in ensuring the model's secure response (such as resisting data poisoning and avoiding the generation of harmful content). However, existing analysis methods based on weight entropy values ​​can only determine the degree of dispersion of information processing by attention heads, and cannot accurately locate the security-critical attention heads that are directly related to the security response. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for locating a safety attention head in a multimodal large model, which can accurately locate the safety-critical attention head.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for locating a secure attention head in a multimodal large model, the method comprising: Collect security datasets and utility datasets; Input the security dataset and utility dataset into the original multimodal large model respectively to obtain the original activation matrix corresponding to the security dataset and the original activation matrix corresponding to the utility dataset; Ablation operations are performed on the attention heads in the original multimodal large model to obtain multiple ablation multimodal large models with the same number of attention heads. Based on each ablation multimodal large model, the ablation activation matrix corresponding to the safety dataset and the ablation activation matrix corresponding to the utility dataset are obtained respectively. Based on the original activation matrix and multiple ablation activation matrices corresponding to the security dataset, obtain the set of security attention heads in the attention heads; Based on the original activation matrix and multiple ablation activation matrices corresponding to the utility dataset, obtain the general set of attention heads in the attention heads; Obtain the difference between the set of security attention heads and the set of general attention heads, and use it as the set of security-critical attention heads.

[0006] Secondly, this application provides a positioning device for a multimodal large-scale model's secure attention head, the device comprising: The data acquisition module is used to collect security datasets and utility datasets; The original activation matrix acquisition module is used to input the security dataset and the utility dataset into the original multimodal large model respectively, and obtain the original activation matrix corresponding to the security dataset and the original activation matrix corresponding to the utility dataset. The ablation activation matrix acquisition module is used to perform ablation operations on the attention heads in the original multimodal large model to obtain multiple ablation multimodal large models with the same number of attention heads. Based on each ablation multimodal large model, the ablation activation matrix corresponding to the security dataset and the ablation activation matrix corresponding to the utility dataset are obtained respectively. The security attention head set acquisition module is used to acquire the security attention head set in the attention head based on the original activation matrix and multiple ablation activation matrices corresponding to the security dataset. The general attention head set acquisition module is used to obtain the general attention head set in the attention heads based on the original activation matrix and multiple ablation activation matrices corresponding to the utility dataset. The safety critical attention head set acquisition module is used to obtain the set difference between the safety attention head set and the general attention head set, which is used as the safety critical attention head set.

[0007] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method and apparatus for locating safety attention heads in a multimodal large model. The method involves collecting a safety dataset and a utility dataset, inputting them into the original multimodal large model to obtain the original activation matrix, then performing an ablation operation on the attention heads in the model to obtain multiple ablated multimodal large models and corresponding ablation activation matrices. Based on the original activation matrices of the safety dataset and utility dataset, and the multiple ablation activation matrices, a set of safety attention heads and a set of general attention heads are obtained respectively. Finally, by calculating the difference between the two sets, the set of safety-critical attention heads directly related to the safety response is accurately located. This solves the problem that existing analysis methods based on weight entropy values ​​cannot accurately locate safety-critical attention heads. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating a method for locating a secure attention head in a multimodal large model, as provided in an embodiment of this application; Figure 2 A flowchart illustrating the processing of a security dataset provided in one embodiment of this application; Figure 3 A flowchart illustrating the processing of a utility dataset provided in one embodiment of this application; Figure 4 A schematic diagram of the functional modules of a positioning device for a multimodal large model safety attention head provided in an embodiment of this application; Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] In recent years, large multimodal models, especially large visual-language models, have made significant progress in various cross-modal tasks by integrating visual and textual information processing capabilities. However, these powerful models have also exposed new security risks. Research has found that large multimodal models are vulnerable to attacks by malicious or harmful inputs. However, the current understanding of how the internal security mechanisms of these models operate and their potential structural weaknesses remains insufficient.

[0013] The complex attention mechanisms within a model play a central role in information processing and decision-making, with certain specific attention heads potentially being crucial for the model's secure response. Accurately locating these "secure attention heads" and verifying whether their functional deficiencies constitute exploitable security vulnerabilities will contribute to a deeper understanding of the model's security weaknesses.

[0014] Figure 1 This is a flowchart illustrating a method for locating a secure attention head for a multimodal large model according to an exemplary embodiment, such as... Figure 1 As shown, the method includes the following steps S101-S106: In step S101, a security dataset and a utility dataset are collected.

[0015] The samples included in the security dataset can be successfully identified by the multimodal large model in its initial state, allowing it to either refuse execution or provide a safe response. The samples included in the utility dataset can evaluate the text-image question-answering functionality of the multimodal large model.

[0016] Secure Dataset Containing a series of harmful input image-text pairs, these samples were processed by a multimodal large model. The ability to successfully identify and refuse execution or provide a safe response in the initial state is key to exploring the model's security capabilities; the utility dataset... Containing a series of benign input samples representing normal application scenarios of the model, this dataset is used to identify attention heads important for the general functionality of the model, in order to distinguish them from safety-specific heads. General functionality refers to the multimodal model's ability to answer text-image questions, such as describing what is in an image or answering how to realize the scene in the image.

[0017] The security dataset can be obtained from the visual language model security benchmark; the utility dataset can be obtained from the visual language model benchmark.

[0018] The safe dataset can come from publicly available visual language model safety benchmarks, such as VLGuard, MM-SafetyBench, and Figstep. Samples rejected by the multimodal large model are selected from these data to form the safe dataset. The utility dataset can come from publicly available visual language model benchmarks, such as COCO-VQA, MM-VET, and ScienceQA.

[0019] In step S102, the security dataset and the utility dataset are input into the original multimodal large model respectively to obtain the original activation matrix corresponding to the security dataset and the original activation matrix corresponding to the utility dataset.

[0020] In one embodiment, step S102 involves inputting the security dataset and the utility dataset into the original multimodal large model to obtain the original activation matrix corresponding to the security dataset and the original activation matrix corresponding to the utility dataset, including the following sub-steps S1021-S1022: S1021. Input each sample in the secure dataset into the original multimodal large model for forward propagation, extract the original activation vector of the last layer, and stack them row by row to form the original activation matrix of the secure dataset.

[0021] S1022. Input each sample in the utility dataset into the original multimodal large model for forward propagation, extract the activation vector of the last layer, and stack them row by row to form the original activation matrix of the utility dataset.

[0022] For security datasets Each sample in Input to the original modal large model Perform forward propagation to extract the activation vector at the last position of the last layer. ,in, It refers to the hidden layer dimension. The activation vectors of all samples in the secure dataset are stacked together to form the original activation matrix on the secure dataset. Using utility datasets Repeat the above steps to obtain the original activation matrix on the utility dataset. .

[0023] When stacking, we can stack them row by row. For N activation vectors of dimension d, stacking them yields N... A matrix of size d.

[0024] This method constructs an activation matrix by extracting the activation vector at the end of the last layer, which can capture the high-level semantic representation of security and utility data in multimodal large models, unify the feature spaces of different modalities, and enhance the ability to identify security-sensitive information while simplifying computational complexity, making subsequent security-critical attention head localization based on activation matrix differences more accurate and efficient.

[0025] In step S103, ablation operations are performed on the attention heads in the original multimodal large model to obtain multiple ablation multimodal large models with the same number of attention heads. Based on each ablation multimodal large model, the ablation activation matrix corresponding to the security dataset and the ablation activation matrix corresponding to the utility dataset are obtained respectively.

[0026] In one embodiment, performing ablation operations on each attention head in the original multimodal large model to obtain multiple ablated multimodal large models with the same number of attention heads includes performing the following steps on each attention head in the original multimodal large model: The first in the original multimodal large model The attention head was dissolved, and the first... Each attention corresponds to an ablation multimodal large model.

[0027] Specifically, the first part of the original multimodal large model can be... The query matrix of the first attention head is multiplied by a preset coefficient, while the other attention heads remain unchanged, to obtain the second attention head. The ablation multimodal large model corresponding to the attention head, where the first attention head... Each attention point, when multiplied by a preset coefficient, can simulate a uniform distribution.

[0028] Assuming the original multimodal large model There are a total of H attention heads, for Each attention head in Ablation operations are performed separately for each attention head. Specifically, for each attention head... Create a new model - ablation multimodal large model The ablation multimodal large model and the original multimodal large model The structures are the same, but the first one is different. The attention head is ablated. The security dataset is... Enter again until the ablation is complete. Ablation of attention heads in a multimodal large model In this process, the ablation activation matrix is ​​obtained by stacking and ablating the last layer of the multimodal large model, which is the last activation vector of the last layer. Similarly, the utility dataset... Input to the ablation multimodal large model with the h-th attention head ablated In the process, the ablation activation matrix obtained from the last activation vector of the last layer of the stacked model is acquired. .

[0029] As an optional implementation, the attention head ablation operation can be implemented in various ways. In this embodiment, the attention head is split into a query matrix, a key matrix, and a value matrix, corresponding to... , and For the h-th attention head, its query matrix is... Multiply by a very small coefficient , ,For example This simulates a uniform distribution, achieving the effect of ablating the attention head. The h-th attention head is ablated using the following formula: in, This represents the h-th attention head after ablation. For the dimension of the hidden layer, The number of attention heads for each transformer layer. This represents the query matrix of the h-th attention head. This represents the key matrix of the h-th attention head. This represents the value matrix of the h-th attention head.

[0030] Since the original attention head has different attention to different parts of the input, by simulating uniform distribution, the attention head pays the same attention to all parts, thus achieving the effect of ablation of the attention head.

[0031] This ablation method selectively disables the functionality of specific heads by multiplying the query matrix of the attention heads by preset coefficients (simulating a uniform distribution), while preserving the overall model architecture. Its beneficial effects include: Controllable ablation: By adjusting the coefficients, progressive suppression of specific attention heads is achieved, avoiding the destruction of the model structure caused by direct deletion, and ensuring the smoothness and interpretability of ablation experiments.

[0032] Simulated uniform attention: Forcing the head's output to be uniformly distributed is equivalent to removing its selective attention to the input, thus isolating the head's unique contribution for evaluation.

[0033] Parametric efficiency: No need to retrain the model; head-level feature disabling can be achieved through matrix operations alone, significantly reducing computational costs.

[0034] In step S104, based on the original activation matrix and multiple ablation activation matrices corresponding to the security dataset, the set of security attention heads in the attention heads is obtained.

[0035] In one embodiment, obtaining the set of security attention heads in the attention heads based on the original activation matrix and multiple ablation activation matrices corresponding to the security dataset includes the following steps S1041-S1044: S1041. Perform singular value decomposition on the original activation matrix corresponding to the secure dataset, and obtain the first column vector of the first left singular matrix obtained after decomposition.

[0036] S1042. Perform singular value decomposition on each ablation activation matrix corresponding to the security dataset, and obtain the first column vector of the second left singular matrix obtained after decomposition. S1043. Calculate the principal angle between the first column vector of each first left singular matrix and the first column vector of each second left singular matrix. The principal angle is the first importance score of the ablated attention head in the ablated multimodal large model corresponding to the second left singular matrix on the safe dataset.

[0037] S1044. Obtain the first priority score with the highest first preset number of scores, wherein the attention heads corresponding to the first priority scores of the first preset number form a set of safe attention heads.

[0038] Specifically, in combination Figure 2 The original activation matrix corresponding to the secure dataset Singular value decomposition yields the first left singular matrix, which in turn gives the first column vector of the first left singular matrix. The ablation activation matrix obtained after ablation of the h-th attention head. Singular value decomposition yields the second left singular matrix, which in turn gives the first column vector of the second left singular matrix. This vector represents the ablation attention head. Subsequently, ablation of multimodal large model processing The main orientation of the data represents the space.

[0039] Calculate these two vectors and The main included angle between In this example, it is the inverse cosine of the cosine of the angle between the two vectors. This angle... The ablation attention head h was quantified for model processing. The data primarily represents the degree of influence of direction, and this principal angle... As attention head h in the security dataset The first importance score, that is: in This represents the vector dot product.

[0040] Collect all attention heads In security datasets First importance score According to the first preset quantity Find the score with the highest importance. Each attention point constitutes a set of secure attention points. : .

[0041] The first preset quantity can be preset or entered manually.

[0042] This method extracts the principal direction of the activation matrix through singular value decomposition (SVD) and calculates the principal angle between the original and ablation models as an importance score. Its beneficial effects are: Capturing structural changes: The first left singular vector of SVD represents the main direction of change of the activation matrix, and the change of the principal angle reflects the impact of the ablation operation on the core feature representation of the model.

[0043] Importance of the quantization head: The larger the principal angle, the more significant the change in the model's representation of security data after the head is ablated, which is directly related to the ability to execute security tasks.

[0044] Dimensionality reduction and stability: SVD effectively reduces the complexity of high-dimensional activation matrices, extracts the most discriminative feature directions, and improves the stability and interpretability of score calculation.

[0045] In step S105, based on the original activation matrix and multiple ablation activation matrices corresponding to the utility dataset, a general attention head set is obtained in the attention heads.

[0046] In one embodiment, step S105, based on the original activation matrix and multiple ablation activation matrices corresponding to the utility dataset, obtains a general set of attention heads in the attention heads, including the following sub-steps S1051-S1054: S1051. Perform singular value decomposition on the original activation matrix corresponding to the utility dataset to obtain the first principal left singular vector. S1052. Perform singular value decomposition on each ablation activation matrix corresponding to the utility dataset to obtain the second principal left singular vector. S1053. Calculate the principal angle between each first principal left singular vector and each second principal left singular vector. The principal angle is the second importance score of the ablated attention head in the ablated multimodal large model corresponding to the second principal left singular vector on the utility dataset. S1054. Obtain the second importance score with the highest second preset number of scores, wherein the attention heads corresponding to the second preset number of second importance scores form a general attention head set.

[0047] Similarly, combined Figure 3 For utility datasets The original activation matrix obtained above Perform singular value decomposition to obtain its first principal left singular vector. ; for the ablation activation matrix Perform singular value decomposition to obtain its second principal left singular vector. ; Calculate all attention heads In the utility dataset Importance score : ; Based on the second preset quantity, identify the score with the highest importance. Each attention head constitutes a general attention head set. : ; As an optional implementation, the selection of the security attention head set... The value is 20, referring to the selection of a general attention head set. It is 20.

[0048] In step S106, the difference between the safety attention head set and the general attention head set is obtained as the safety critical attention head set.

[0049] Computing the set of secure attention heads Compared to the general attention head set The difference set yields the set of security-critical attention heads. : ; This refers to a head that exhibits high importance when handling security-related inputs, but is relatively less important when handling general inputs.

[0050] In summary, this disclosure collects security datasets and utility datasets, inputs them into the original multimodal large model to obtain the original activation matrix, then performs ablation operations on the attention heads in the model to obtain multiple ablation multimodal large models and corresponding ablation activation matrices. Based on the original activation matrices of the security dataset and utility dataset and the multiple ablation activation matrices, a set of security attention heads and a set of general attention heads are obtained respectively. Finally, by calculating the difference between the two sets, the set of security-critical attention heads that are directly related to the security response is accurately located, solving the problem that existing analysis methods based on weight entropy values ​​cannot accurately locate security-critical attention heads.

[0051] In one embodiment, the method of this disclosure includes: The attention heads of the security-critical attention head set in the original multimodal large model are ablated to obtain the model to be verified; the security dataset is input into the model to be verified, and the change in attack success rate is calculated.

[0052] Select one or more attention heads from the set of security-critical attention heads for selective ablation, input the harmful dataset into the ablated model, and measure the change in attack success rate to verify the impact of the selectively ablated attention heads on model security and the existing security vulnerabilities.

[0053] Obtaining a set of safety-critical attention heads Then, the following steps are performed to verify its impact on model security and potential vulnerabilities: (1) Selective ablation: from the set Select one or more safety-critical attention heads. For the original model... Applying the ablation operation, we obtained a model that only ablated these selected safety critical heads (where... ).

[0054] (2) Testing: The original dataset belonged to the security dataset Harmful input samples are input into the ablated model.

[0055] (3) Evaluation: The model's response to these malicious inputs is automatically recorded and written to a JSON file. The Attack Success Rate (ASR) is calculated, which is the proportion of cases where the model fails to provide a security rejection but instead executes malicious instructions or generates inappropriate content. In other words, the model automatically determines whether an attack is successful. If the model's response contains a rejection word (e.g., "Sorry"), it is a rejected sample; otherwise, it is a successful sample, thus obtaining the attack success rate.

[0056] (4) Validation: Compare the models after ablation of attention heads. ASR and the original model ASR on the same harmful input. If ASR is significantly higher than The ASR indicates that the security-critical attention head has been ablated. Maintaining the model's security is crucial, as the lack of functionality in these attention heads constitutes an exploitable security vulnerability, and the model's security depends to some extent on these few key components.

[0057] This method analyzes the security vulnerabilities of large multimodal models from within, efficiently and accurately locating security-critical attention points within the model. It then verifies the security vulnerabilities exposed when these components are ablated through simulated attacks. This method requires no model fine-tuning, providing an important technical means for a deeper understanding and improvement of the security of large-scale artificial intelligence models.

[0058] Based on the same inventive concept, this application also provides a device for locating a secure attention head for a multimodal large model to implement the aforementioned method for locating a secure attention head for a multimodal large model. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the device for locating a secure attention head for a multimodal large model provided below can be found in the limitations of the method for locating a secure attention head for a multimodal large model described above, and will not be repeated here.

[0059] In one exemplary embodiment, such as Figure 4 As shown, a positioning device for a safety attention head of a multimodal large model is provided, comprising: Acquisition module 11 is used to acquire security datasets and utility datasets; The original activation matrix acquisition module 12 is used to input the security dataset and the utility dataset into the original multimodal large model respectively, and obtain the original activation matrix corresponding to the security dataset and the original activation matrix corresponding to the utility dataset. The ablation activation matrix acquisition module 13 is used to perform ablation operations on the attention heads in the original multimodal large model to obtain multiple ablation multimodal large models with the same number of attention heads. Based on each ablation multimodal large model, the ablation activation matrix corresponding to the security dataset and the ablation activation matrix corresponding to the utility dataset are obtained respectively. The security attention head set acquisition module 14 is used to acquire the security attention head set in the attention head based on the original activation matrix and multiple ablation activation matrices corresponding to the security dataset. The general attention head set acquisition module 15 is used to acquire the general attention head set in the attention heads based on the original activation matrix and multiple ablation activation matrices corresponding to the utility dataset. The safety critical attention head set acquisition module 16 is used to obtain the set difference between the safety attention head set and the general attention head set, and use it as the safety critical attention head set.

[0060] As an optional implementation, the original activation matrix acquisition module 12 is specifically used for: Each sample in the secure dataset is input into the original multimodal large model for forward propagation. The original activation vectors of the last layer are extracted and stacked row by row to form the original activation matrix of the secure dataset. Each sample in the utility dataset is input into the original multimodal large model for forward propagation. The activation vectors of the last layer are extracted and stacked row by row to form the original activation matrix of the utility dataset.

[0061] As an optional implementation, in order to perform ablation operations on the attention heads in the original multimodal large model to obtain multiple ablated multimodal large models with the same number of attention heads, the ablation activation matrix acquisition module 13 is specifically used for: Perform the following steps for each attention head in the original multimodal large model: The first in the original multimodal large model The attention head was dissolved, and the first... Each attention corresponds to an ablation multimodal large model.

[0062] As an optional implementation, the first step in the original multimodal large model... The attention head was dissolved, and the first... The ablation activation matrix acquisition module 13, which corresponds to each attention aspect of the ablation multimodal large model, is specifically used for: The first in the original multimodal large model The query matrix of the first attention head is multiplied by a preset coefficient, while the other attention heads remain unchanged, to obtain the second attention head. The ablation multimodal large model corresponding to the attention head, where the first attention head... Each attention point, when multiplied by a preset coefficient, can simulate a uniform distribution.

[0063] As an optional implementation, the security attention head set acquisition module 14 is specifically used for: Perform singular value decomposition on the original activation matrix corresponding to the security dataset, and obtain the first column vector of the first left singular matrix obtained after decomposition. Perform singular value decomposition on each ablation activation matrix corresponding to the security dataset, and obtain the first column vector of the second left singular matrix obtained after decomposition. Calculate the principal angle between the first column vector of each first left singular matrix and the first column vector of each second left singular matrix. The principal angle is the first importance score of the ablated attention head in the ablated multimodal large model corresponding to the second left singular matrix on the security dataset. Obtain the first priority score with the highest first preset number of scores, where the attention heads corresponding to the first priority scores of the first preset number form a set of safe attention heads.

[0064] As an optional implementation, the general attention head set acquisition module 15 is specifically used for: Singular value decomposition is performed on the original activation matrix corresponding to the utility dataset to obtain the first principal left singular vector. Singular value decomposition is performed on each ablation activation matrix corresponding to the utility dataset to obtain the second principal left singular vector; Calculate the principal angle between each first principal left singular vector and each second principal left singular vector. The principal angle is the second importance score of the ablated attention head in the ablated multimodal large model corresponding to the second principal left singular vector on the utility dataset. Obtain the second importance score with the highest second preset number of scores, where the attention heads corresponding to the second preset number of second importance scores form a general attention head set.

[0065] As an optional implementation, the device further includes: The ablation module is used to ablate the attention heads of the safety-critical attention head set in the original multimodal large model to obtain the model to be verified. The calculation module is used to input the security dataset into the model to be verified and calculate the change in the attack success rate.

[0066] As an optional implementation method, The security dataset is obtained from the visual language model security benchmark; Obtain utility datasets from visual language model benchmarks.

[0067] As an optional implementation, the samples included in the security dataset can be successfully identified by the multimodal large model in the initial state and either rejected or given a safe response; the samples included in the utility dataset can identify the attention heads of the text-to-image question-answering function in the multimodal large model.

[0068] In one exemplary embodiment, such as Figure 5 The diagram illustrates the physical structure of an electronic device, which may include a processor 10, a communication interface 20, a memory 30, and a communication bus 40. The processor 10, communication interface 20, and memory 30 communicate with each other via the communication bus 40. The processor 10 can invoke logical instructions stored in the memory 30 to execute the following methods: Collect security datasets and utility datasets; Input the security dataset and utility dataset into the original multimodal large model respectively to obtain the original activation matrix corresponding to the security dataset and the original activation matrix corresponding to the utility dataset; Ablation operations are performed on the attention heads in the original multimodal large model to obtain multiple ablation multimodal large models with the same number of attention heads. Based on each ablation multimodal large model, the ablation activation matrix corresponding to the safety dataset and the ablation activation matrix corresponding to the utility dataset are obtained respectively. Based on the original activation matrix and multiple ablation activation matrices corresponding to the security dataset, obtain the set of security attention heads in the attention heads; Based on the original activation matrix and multiple ablation activation matrices corresponding to the utility dataset, obtain the general set of attention heads in the attention heads; Obtain the difference between the set of security attention heads and the set of general attention heads, and use it as the set of security-critical attention heads.

[0069] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] In one exemplary embodiment, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0071] In one exemplary embodiment, a computer program product is provided, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to implement the steps in the above method embodiments.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for positioning a secure attention head of a multi-modal large model, characterized in that, The method comprises: collecting a safety data set and an utility data set; inputting the safety data set and the utility data set into an original multi-modal large model respectively to obtain an original activation matrix corresponding to the safety data set and an original activation matrix corresponding to the utility data set; performing an ablation operation on each attention head in the original multi-modal large model to obtain a plurality of ablation multi-modal large models with the same number of attention heads, and based on each ablation multi-modal large model, an ablation activation matrix corresponding to the safety data set and an ablation activation matrix corresponding to the utility data set are obtained respectively; based on the original activation matrix corresponding to the safety data set and the plurality of ablation activation matrices, a safety attention head set in the attention heads is obtained; based on the original activation matrix corresponding to the utility data set and the plurality of ablation activation matrices, a general attention head set in the attention heads is obtained; obtaining a set difference between the safety attention head set and the general attention head set as a safety-critical attention head set.

2. The method of claim 1, wherein, The safety data set and the utility data set are inputted into the original multi-modal large model respectively to obtain the original activation matrix corresponding to the safety data set and the original activation matrix corresponding to the utility data set, which comprises: inputting each sample in the safety data set into the original multi-modal large model for forward propagation, extracting the original activation vector at the last layer, and stacking the original activation vector by row to form the original activation matrix of the safety data set; inputting each sample in the utility data set into the original multi-modal large model for forward propagation, extracting the activation vector at the last layer, and stacking the activation vector by row to form the original activation matrix of the utility data set.

3. The method of claim 1, wherein, The ablation operation is performed on each attention head in the original multi-modal large model to obtain a plurality of ablation multi-modal large models with the same number of attention heads, which comprises: the following steps are performed on each attention head in the original multi-modal large model respectively: The first attention head in the original multi-modal large model is ablated to obtain an ablation multi-modal large model corresponding to the first attention. The first attention head in the original multi-modal large model is ablated to obtain an ablation multi-modal large model corresponding to the first attention. ​ 4. The method of claim 3, wherein, The first part of the original multimodal large model The attention head was dissolved, and the first one was obtained. The ablation multimodal large model corresponding to each attention includes: The query matrix of the first attention head in the original multi-modal large model is multiplied by a preset coefficient, and other attention heads remain unchanged to obtain an ablation multi-modal large model corresponding to the first attention head. The first attention head multiplied by the preset coefficient can simulate a uniform distribution. The query matrix of the first attention head in the original multi-modal large model is multiplied by a preset coefficient, and other attention heads remain unchanged to obtain an ablation multi-modal large model corresponding to the first attention head. The first attention head multiplied by the preset coefficient can simulate a uniform distribution.

5. The method of claim 1, wherein, The original activation matrix corresponding to the safety data set and the plurality of ablation activation matrices are used to obtain a safety attention head set in the attention heads, which comprises: singular value decomposition is performed on the original activation matrix corresponding to the safety data set to obtain a first column vector of a first left singular matrix obtained after decomposition; singular value decomposition is performed on each ablation activation matrix corresponding to the safety data set to obtain a first column vector of a second left singular matrix obtained after decomposition; the principal included angle between each first left singular matrix first column vector and each second left singular matrix first column vector is calculated, and the principal included angle is the first importance score of the ablated attention head in the ablation multi-modal large model corresponding to the second left singular matrix on the safety data set; the first importance scores with the highest score values are obtained, and the first importance scores with the highest score values correspond to the safety attention head set.

6. The method of claim 1, wherein, The original activation matrix corresponding to the utility data set and the plurality of ablation activation matrices are used to obtain a general attention head set in the attention heads, which comprises: singular value decomposition is performed on the original activation matrix corresponding to the utility data set to obtain a first principal left singular vector; singular value decomposition is performed on each ablation activation matrix corresponding to the utility data set to obtain a second principal left singular vector; calculate a principal angle between each first principal left singular vector and each second principal left singular vector, the principal angle being a second importance score of an ablated attention head in the ablated multi-modal large model corresponding to the second principal left singular vector on the utility dataset; obtain a second preset number of second importance scores with the highest score values, wherein the second preset number of second importance scores correspond to the attention heads in the universal attention head set.

7. The method of claim 1, wherein, The method further comprises: ablate the attention heads in the safety-critical attention head set in the original multi-modal large model to obtain a to-be-verified model; input the safety dataset into the to-be-verified model and calculate the change in the attack success rate.

8. The method of claim 1, wherein the safety dataset is obtained from a visual language model safety benchmark; the utility dataset is obtained from a visual language model benchmark.

9. The method of claim 1, wherein, The samples included in the safety dataset can be successfully identified by the multi-modal large model in the initial state and rejected from execution or given a safety response; the samples included in the utility dataset can identify the attention heads of the image-text question answering function in the multi-modal large model.

10. A positioning device for a safety attention head of a multimodal large model, characterized in that, The device comprises: a collection module configured to collect the safety dataset and the utility dataset; an original activation matrix acquisition module configured to input the safety dataset and the utility dataset into the original multi-modal large model to obtain an original activation matrix corresponding to the safety dataset and an original activation matrix corresponding to the utility dataset; an ablation activation matrix acquisition module configured to perform ablation operations on the attention heads in the original multi-modal large model to obtain a plurality of ablation multi-modal large models with the same number of attention heads, and based on each ablation multi-modal large model, to obtain an ablation activation matrix corresponding to the safety dataset and an ablation activation matrix corresponding to the utility dataset; a safety attention head set acquisition module configured to obtain a safety attention head set from the attention heads based on the original activation matrix corresponding to the safety dataset and the plurality of ablation activation matrices; a universal attention head set acquisition module configured to obtain a universal attention head set from the attention heads based on the original activation matrix corresponding to the utility dataset and the plurality of ablation activation matrices; a safety-critical attention head set acquisition module configured to obtain a set difference between the safety attention head set and the universal attention head set as the safety-critical attention head set.

Images