Intersection set constraint based defense method for large language model quantization attack

Abstract

The application discloses a large language model quantization attack defense method based on intersection and union constraint and belongs to the technical field of large model security. The application obtains a full-precision large language model to be defended and multiple target quantization configurations; determines a quantization stable interval of each candidate parameter in the full-precision large language model under each target quantization configuration according to the candidate parameter, and determines an intersection interval and a union interval according to the quantization stable intervals; determines the risk weight of each candidate parameter according to the intersection interval, and determines a high-risk parameter from the candidate parameters according to the risk weight; performs parameter correction on the high-risk parameter according to the intersection interval and the union interval, determines a corrected parameter, and outputs a corrected full-precision large language model containing the corrected parameter, thereby achieving the beneficial effect of improving the defense effect on the large model.

Description

Technical Field

[0001] This invention relates to the field of large model security technology, and in particular to a method for defending against large language model quantization attacks based on intersection and union constraints. Background Technology

[0002] Large language models typically contain billions to hundreds of billions of parameters. Direct deployment in full-precision format incurs high storage overhead, GPU memory usage, and inference computation costs. To reduce deployment costs, existing technologies often employ model quantization methods, converting model weights or activations from high-precision numerical formats such as FP32 and BF16 to low-bit numerical formats such as INT8, NF4, and FP4. This reduces model storage space and computational overhead, and improves inference deployment efficiency. However, the quantization process may introduce security risks. Quantization security methods primarily focus on passive security degradation caused by quantization itself, such as decreased model performance, weakened security alignment, or perturbation of some security parameters. These methods typically improve the performance of the quantized model through quantization-aware training, security patches, and high-precision protection of key parameters. However, these methods mainly address natural security degradation in non-adversarial scenarios and do not fully consider quantization attacks actively constructed by attackers. In quantization attacks, the model can maintain normal behavior in full-precision state, thus bypassing conventional full-precision detection; the hidden malicious behavior is only activated after the user performs low-bit quantization. Current defense methods are difficult to directly apply to large language model quantization attack scenarios such as ad injection, excessive refusal to answer, and jailbreak response.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a method for defending against large language model quantization attacks based on intersection and union constraints, aiming to improve the defense effectiveness against large models. To achieve the above objective, this invention provides a method for defending against large language model quantization attacks based on intersection and union constraints, which includes the following steps: Obtain the full-precision large language model to be defended and the quantization configuration of multiple targets; Based on each candidate parameter in the full-precision large language model, determine the quantization stability interval of the candidate parameter under each target quantization configuration, and determine the intersection interval and union interval based on each of the quantization stability intervals; The risk weight of each candidate parameter is determined based on the intersection interval, and the high-risk parameter is determined from the candidate parameters based on the risk weight; The high-risk parameters are corrected based on the intersection interval and the union interval to determine the corrected parameters, and a corrected full-precision large language model containing the corrected parameters is output.

[0005] Optionally, the step of determining the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model includes: When the target quantization is configured as uniform quantization, the quantization bucket boundary of the candidate parameter is determined according to the quantization scale and quantization step size corresponding to the uniform quantization, and the first quantization stable interval of the candidate parameter under the uniform quantization is determined according to the quantization bucket boundary. When the target quantization is configured as codebook quantization, the second quantization stability interval of the candidate parameter under the codebook quantization is determined according to the decision boundary of adjacent items in the codebook of the codebook quantization. The quantization stability interval is determined based on the first quantization stability interval and the second quantization stability interval.

[0006] Optionally, the step of determining the risk weight of each candidate parameter based on the intersection interval, and determining the high-risk parameter from the candidate parameters based on the risk weight, includes: Calculate the width of the intersection interval; The risk weight of the candidate parameter is calculated based on the intersection interval width and a preset numerical stability term, wherein the risk weight is negatively correlated with the intersection interval width; The candidate parameters are sorted according to the risk weights, and the candidate parameters with the highest sorting ratio are determined as the high-risk parameters.

[0007] Optionally, the step of correcting the high-risk parameter based on the intersection interval and the union interval includes: Determine the correction direction for the high-risk parameter, wherein the correction direction is the direction that causes the parameter value of the high-risk parameter to move away from the intersection interval; The upper limit of the correction magnitude of the high-risk parameter is determined based on the boundary of the union interval. The high-risk parameter is corrected according to the correction direction and the upper limit of the correction magnitude so that the corrected parameter value is outside the intersection interval and inside the union interval.

[0008] Optionally, the step of correcting the high-risk parameter according to the correction direction and the upper limit of the correction magnitude further includes: Obtain clean calibration data, and input the clean calibration data into the full-precision large language model and the corrected full-precision large language model respectively, and extract the corresponding hidden state data; Calculate the hidden state difference between the hidden state data corresponding to the full-precision large language model and the hidden state data corresponding to the corrected full-precision large language model; The upper limit of the correction magnitude is adjusted based on the hidden state difference value to constrain the difference in model output before and after correction for the high-risk parameter.

[0009] Optionally, the step of determining the correction direction of the high-risk parameter includes: Calculate the boundary distance between the parameter value of the high-risk parameter and each boundary of the intersection interval; The correction direction is determined based on the boundary distance, and the correction direction is the direction that causes the high-risk parameter to move away from the intersection interval with minimal amplitude shift.

[0010] Optionally, the parameter correction is applied to the full-precision large language model, wherein the corrected parameter values ​​are located outside the intersection interval, so as to destroy the vulnerable parameter mapping shared by multiple quantization attacks; The corrected parameter values ​​are located within the union interval to maintain the normal capability of the full-precision large language model in non-attack scenarios.

[0011] Furthermore, to achieve the above objectives, the present invention also provides a large language model quantization attack defense device based on intersection and union constraints, the large language model quantization attack defense device based on intersection and union constraints comprising: The acquisition module is used to acquire the full-precision large language model to be defended and the quantization configuration of multiple targets; The analysis module is used to determine the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model, and to determine the intersection interval and the union interval based on each quantization stability interval; The identification module is used to determine the risk weight of each candidate parameter based on the intersection interval, and to determine the high-risk parameter from the candidate parameters based on the risk weight; The correction module is used to correct the high-risk parameters based on the intersection interval and the union interval, determine the corrected parameters, and output a corrected full-precision large language model containing the corrected parameters.

[0012] Furthermore, to achieve the above objectives, the present invention also provides a large language model quantization attack defense device based on intersection and union constraints, the large language model quantization attack defense device based on intersection and union constraints comprising: The acquisition module is used to acquire the full-precision large language model to be defended and the quantization configuration of multiple targets; The analysis module is used to determine the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model, and to determine the intersection interval and the union interval based on each quantization stability interval; The identification module is used to determine the risk weight of each candidate parameter based on the intersection interval, and to determine the high-risk parameter from the candidate parameters based on the risk weight; The correction module is used to correct the high-risk parameters based on the intersection interval and the union interval, determine the corrected parameters, and output a corrected full-precision large language model containing the corrected parameters.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a large language model quantization attack defense device based on intersection and union constraints. The large language model quantization attack defense device based on intersection and union constraints includes: a memory, a processor, and a large language model quantization attack defense program based on intersection and union constraints stored in the memory and executable on the processor. The large language model quantization attack defense program based on intersection and union constraints is configured to implement the steps of the large language model quantization attack defense method based on intersection and union constraints described above.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a storage medium storing a large language model quantization attack defense program based on intersection and union constraints. When the large language model quantization attack defense program based on intersection and union constraints is executed by a processor, it implements the steps of the large language model quantization attack defense method based on intersection and union constraints described above.

[0015] This invention proposes a method for defending against large language model quantization attacks based on intersection and union constraints. This method acquires a full-precision large language model to be defended against and multiple target quantization configurations. It determines the quantization stability interval of each candidate parameter in the full-precision large language model under each target quantization configuration, and determines the intersection and union intervals based on these stability intervals. Based on the intersection interval, it determines the risk weight of each candidate parameter, and identifies high-risk parameters from the candidate parameters based on these risk weights. The high-risk parameters are then corrected according to the intersection and union intervals to determine the corrected parameters. A corrected full-precision large language model containing these corrected parameters is output, thereby disrupting the multi-quantization stable mapping relationship upon which the quantization attack relies. Simultaneously, the correction magnitude is constrained by the union interval, preventing excessive parameter shifts that could impair the model's original capabilities. Attached Figure Description

[0016] Figure 1This is a schematic diagram of the structure of a large language model quantization attack defense device based on intersection and union constraints in the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the large language model quantization attack defense method based on intersection and union constraints of the present invention. Figure 3 This is a flowchart illustrating the second embodiment of the large language model quantization attack defense method based on intersection and union constraints of the present invention. Figure 4 This is a schematic diagram of the intersection and union intervals of a single parameter in the large language model quantization attack defense method based on intersection and union constraints of the present invention. Figure 5 This is a flowchart illustrating the attack defense process for large language model quantization in this invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a large language model quantization attack defense device based on intersection and union constraints in the hardware operating environment of the embodiment of the present invention.

[0020] like Figure 1 As shown, the large language model quantization attack defense device based on intersection and union constraints may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, an interaction device 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The interaction device 1003 may include a display screen or an input unit such as a keyboard. Optionally, the interaction device 1003 may also be connected to the communication bus via standard wired or wireless interfaces. The network interface 1004 may optionally include standard wired or wireless interfaces (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0021] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on a large language model quantization attack defense device based on intersection and union constraints, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0022] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a large language model quantization attack defense program based on intersection and union constraints.

[0023] exist Figure 1 In the large language model quantization attack defense device based on intersection and union constraints shown, the network interface 1004 is mainly used for data communication with other devices; the interaction device 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the large language model quantization attack defense device based on intersection and union constraints of the present invention can be set in the large language model quantization attack defense device based on intersection and union constraints. The large language model quantization attack defense device based on intersection and union constraints calls the large language model quantization attack defense program based on intersection and union constraints stored in the memory 1005 through the processor 1001, and executes the large language model quantization attack defense method based on intersection and union constraints provided in the embodiment of the present invention.

[0024] This invention provides a method for defending against large language model quantization attacks based on intersection and union constraints, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a large language model quantization attack defense method based on intersection and union constraints according to the present invention.

[0025] In this embodiment, the large language model quantization attack defense method based on intersection and union constraints includes: Step S1: Obtain the full-precision large language model to be defended and the quantization configuration of multiple targets; In this embodiment, the full-precision large language model is a Transformer architecture autoregressive language model that uses FP32 or BF16 high-precision numerical format to represent parameters. The target quantization configuration includes at least two of INT8 uniform quantization, NF4 codebook quantization, and FP4 codebook quantization, forming a target quantization configuration set.

[0026] Step S2: Determine the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model, and determine the intersection interval and union interval based on each quantization stability interval; In this embodiment, linear layer weights participating in low-bit quantization are selected from the model parameters as a set of candidate parameters. For each candidate parameter, its quantization stability interval, which maintains the quantization result unchanged under each target quantization configuration, is calculated. Then, the intersection of all quantization stability intervals is taken as the intersection interval, and the union of all quantization stability intervals is taken as the union interval. The intersection interval reflects the shared stable region where the parameter maintains the original quantization result under multiple quantization configurations; the union interval reflects the overall coverage of the stable region of the parameter under multiple quantization configurations.

[0027] Specifically, this invention first defines a target quantization configuration set for the full-precision large language model to be deployed:

[0028] in, This indicates different quantization configurations. For example:

[0029] For each parameter to be corrected in the model The present invention calculates its value in each quantization configuration. The parameter stability interval that keeps the quantization results unchanged is denoted as:

[0030] in, Indicates parameters In quantitative allocation The lower bound of the stable interval is below. This indicates the boundary of the stability interval. When the parameter... When it changes within this range, it undergoes quantization. The quantized value remains unchanged.

[0031] After obtaining the parameters After establishing stable intervals under multiple quantization configurations, this invention further constructs an intersection interval, and the intersection interval is defined as follows:

[0032] The union interval is defined as:

[0033] in, Indicates parameters A shared stable region that maintains the quantization result unchanged under all target quantization configurations. This region can characterize the feasible space for an attacker to maintain malicious mappings under multiple quantization configurations simultaneously. If the intersection interval is narrow, it indicates that the space for this parameter to maintain the quantization result unchanged under multiple quantization formats is small, and only a small correction may be needed to destroy the stable mapping on which the attack depends. Indicates parameters The overall range of stable region coverage under multiple quantization configurations.

[0034] Step S3: Determine the risk weight of each candidate parameter based on the intersection interval, and determine the high-risk parameter from the candidate parameters based on the risk weight; In this embodiment, the narrower the intersection interval of a parameter, the smaller the space it has to maintain multiple quantization mappings simultaneously. This makes it easier to disrupt the stable mapping relationship upon which an attack relies with a small correction, thus posing a higher risk. The system calculates the risk weight of each candidate parameter based on the width of the intersection interval and selects a preset proportion of parameters with the highest risk weights, such as the top 20%, as high-risk parameters.

[0035] Step S4: Perform parameter correction on the high-risk parameters according to the intersection interval and the union interval, determine the corrected parameters, and output the corrected full-precision large language model containing the corrected parameters.

[0036] In this embodiment, the goal of parameter correction is to move high-risk parameters out of the intersection interval, thereby preventing the original quantization results under all target quantization configurations from being maintained simultaneously, and disrupting the stable multi-quantization mapping relationship upon which quantization attacks rely. Simultaneously, the correction magnitude is constrained by the union interval to avoid excessive parameter shifts that could impair the model's original capabilities. After correction, a corrected full-precision model is output, which can subsequently be quantized into INT8, NF4, FP4, or other low-bit formats for deployment.

[0037] Furthermore, based on the first embodiment, a second embodiment of the present invention for the defense method against large language model quantization attacks with intersection and union constraints is proposed. In this embodiment, referring to... Figure 3 The step of determining the quantization stability interval of each candidate parameter in the full-precision large language model under each target quantization configuration includes: Step S21: When the target quantization is configured as uniform quantization, the quantization bucket boundary of the candidate parameter is determined according to the quantization scale and quantization step size corresponding to the uniform quantization, and the first quantization stable interval of the candidate parameter under the uniform quantization is determined according to the quantization bucket boundary. In this embodiment, uniform quantization includes linear mapping quantization formats such as INT8. For uniform quantization, the upper and lower boundaries of the quantization bucket are calculated based on the quantization scale and quantization step size. When the parameters change within this boundary range, the quantization output value remains unchanged. This range is the first quantization stable interval.

[0038] Step S22: When the target quantization is configured as codebook quantization, the second quantization stability interval of the candidate parameter under the codebook quantization is determined according to the decision boundary of adjacent items in the codebook of the codebook quantization. Specifically, codebook quantization includes non-uniform mapping quantization formats such as NF4 and FP4. For codebook quantization, there are decision boundaries between adjacent terms in the codebook. When the parameter changes between adjacent decision boundaries, it is mapped to the same codebook term. This range is the second quantization stability interval.

[0039] Step S23: Determine the quantization stability interval based on the first quantization stability interval and the second quantization stability interval.

[0040] In this embodiment, the quantization stability interval is calculated using either step S21 or step S22, depending on the type of the target quantization configuration corresponding to the candidate parameter. For a parameter set involving multiple quantization types, the quantization stability intervals under each type are combined to form the complete quantization stability interval set for that parameter.

[0041] Furthermore, based on the first or second embodiment, a third embodiment of the present invention for the defense method against large language model quantization attacks with intersection and union constraints is proposed. In this embodiment, the step of determining the risk weight of each candidate parameter according to the intersection interval, and determining high-risk parameters from the candidate parameters according to the risk weight includes: Calculate the width of the intersection interval; The risk weight of the candidate parameter is calculated based on the intersection interval width and a preset numerical stability term, wherein the risk weight is negatively correlated with the intersection interval width; The candidate parameters are sorted according to the risk weights, and the candidate parameters with the highest sorting ratio are determined as the high-risk parameters.

[0042] In this embodiment, specifically, for the parameters The width of their intersection interval is defined as:

[0043] Based on this width, define the parameter risk weight:

[0044] in, This is a numerically stable term greater than 0. According to the above definition, the narrower the intersection interval, the better. The smaller the value, the higher the risk weight. The larger the value, the more likely the parameter is to become a key parameter in a quantization attack.

[0045] In one implementation, parameters ranking in the top 20% by risk weight can be selected as the parameters to be corrected. In this way, the present invention can focus the correction on parameters more likely to influence attack triggering, rather than indiscriminately perturbing all model parameters. (See also...) Figure 4 , Figure 4This diagram illustrates the intersection and union intervals of a single parameter. The width of the intersection interval is the difference between its upper and lower bounds. A narrower intersection interval indicates a smaller space for the parameter to maintain its quantization result under various quantization configurations. All candidate parameters are sorted in descending order of risk weight, and parameters with a pre-defined proportion of the highest rankings are selected as high-risk parameters. In this way, the correction operation focuses on the parameters most likely to trigger an attack, rather than indiscriminately perturbing all parameters, thereby improving the specificity of the defense and reducing the loss of model capabilities.

[0046] Furthermore, based on any of the above embodiments, a fourth embodiment of the present invention for the defense method against large language model quantization attacks with intersection and union constraints is proposed. In this embodiment, the step of parameter correction of the high-risk parameters based on the intersection interval and the union interval includes: Determine the correction direction for the high-risk parameter, wherein the correction direction is the direction that causes the parameter value of the high-risk parameter to move away from the intersection interval; The upper limit of the correction magnitude of the high-risk parameter is determined based on the boundary of the union interval. The high-risk parameter is corrected according to the correction direction and the upper limit of the correction magnitude so that the corrected parameter value is outside the intersection interval and inside the union interval.

[0047] In this embodiment, the correction direction is determined as follows: the boundary distance between the current parameter value of the high-risk parameter and each boundary of the intersection interval is calculated, and the direction that causes the parameter to leave the intersection interval with the minimum magnitude is selected as the correction direction. For the selected high-risk parameter, the correction objective of this invention is to make it leave the shared intersection interval under multiple quantization configurations, rather than simply adding random perturbations to the parameter or performing full model fine-tuning. The upper limit of the correction magnitude is determined by the boundary of the union interval, ensuring that the corrected parameter value is within the union interval and avoiding excessive parameter offset that would damage the original representation capability of the model. Specifically, parameter correction can be expressed as:

[0048] in:

[0049]

[0050]

[0051] in, Indicates the parameters before correction. Indicates the corrected parameters. Indicates the correction direction. This represents the displacement required to reach the boundary of the union interval along this direction. Indicates the normalized correction strength. This indicates the maximum correction ratio.

[0052] In this way, the present invention moves the high-risk parameter along a selected direction, causing the parameter to leave the intersection interval with a small displacement. This prevents the original quantization results from being maintained under all target quantization configurations simultaneously, thereby disrupting the stable multi-quantization mapping relationship upon which quantization attacks rely. Simultaneously, this invention utilizes the union interval... Constrain the correction range to avoid excessive parameter offset and reduce the impact on the model's original language understanding, instruction following, and reasoning abilities.

[0053] Furthermore, the step of correcting the high-risk parameter according to the correction direction and the upper limit of the correction magnitude also includes: Obtain clean calibration data, and input the clean calibration data into the full-precision large language model and the corrected full-precision large language model respectively, and extract the corresponding hidden state data; Calculate the hidden state difference between the hidden state data corresponding to the full-precision large language model and the hidden state data corresponding to the corrected full-precision large language model; The upper limit of the correction magnitude is adjusted based on the hidden state difference value to constrain the difference in model output before and after correction for the high-risk parameter.

[0054] In this embodiment, clean calibration data refers to publicly available harmless instruction data, general question-and-answer data, or task-independent text data, such as 128 clean calibration samples. Each calibration sample is input into both the pre-calibration model and the current post-calibration model, and the hidden state representations of the two at specified layers or locations are extracted. When the difference in hidden states exceeds a preset range, the parameter calibration intensity is reduced, the number of parameters to be calibrated is reduced, or the calibration direction of some parameters is adjusted, so that the internal representation of the calibrated model on normal input is as close as possible to the original model, thereby reducing the impact of calibration on the model's general capabilities.

[0055] Furthermore, the step of determining the correction direction of the high-risk parameter includes: Calculate the boundary distance between the parameter value of the high-risk parameter and each boundary of the intersection interval; The correction direction is determined based on the boundary distance, and the correction direction is the direction that causes the high-risk parameter to move away from the intersection interval with minimal amplitude shift.

[0056] Furthermore, based on any of the above embodiments, a fifth embodiment of the present invention for the defense method against quantization attacks on large language models with intersection and union constraints is proposed. In this embodiment, the parameter correction is applied to the full-precision large language model, wherein the corrected parameter value is located outside the intersection interval, so as to destroy the vulnerable parameter mapping shared by multiple quantization attacks. The corrected parameter values ​​are located within the union interval to maintain the normal capability of the full-precision large language model in non-attack scenarios.

[0057] In any of the above embodiments, a fifth embodiment of the present invention for defending against large language model quantization attacks based on intersection and union constraints is proposed. In this embodiment, the parameter correction is applied to the full-precision large language model, wherein the corrected parameter value is located outside the intersection interval to disrupt the vulnerable parameter mapping shared by multiple quantization attacks; the corrected parameter value is located within the union interval to maintain the normal capability of the full-precision large language model in non-attack scenarios.

[0058] In this embodiment, when the corrected full-precision model is subsequently quantized to INT8, NF4, FP4, or other low-bit formats, the stable mapping relationship upon which quantization attacks rely is disrupted because high-risk parameters have been pushed out of the shared intersection range of multiple quantization configurations. Therefore, the probability of triggering hidden malicious behaviors such as ad injection, excessive rejection, and jailbreak responses is reduced after quantization. At the same time, because the correction process is constrained by the union interval and clean calibration data, the model's original general question-answering ability, reasoning ability, and normal generation ability can be well preserved.

[0059] Further experimental results can be found in Table 1, which is as follows:

[0060] This method demonstrates good defense performance across various models, attack scenarios, and quantization formats. It significantly reduces the success rate of attacks after quantization in three scenarios: ad injection, excessive rejection, and jailbreak attacks. Meanwhile, general capability metrics such as Massive Multitask Language Understanding (MMLU) and TruthfulQA-MC2 (MC2) remain generally stable.

[0061] Furthermore, this invention also proposes a large language model quantization attack defense device based on intersection and union constraints, the large language model quantization attack defense device based on intersection and union constraints includes: The acquisition module is used to acquire the full-precision large language model to be defended and the quantization configuration of multiple targets; The analysis module is used to determine the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model, and to determine the intersection interval and the union interval based on each quantization stability interval; The identification module is used to determine the risk weight of each candidate parameter based on the intersection interval, and to determine the high-risk parameter from the candidate parameters based on the risk weight; The correction module is used to correct the high-risk parameters based on the intersection interval and the union interval, determine the corrected parameters, and output a corrected full-precision large language model containing the corrected parameters.

[0062] Furthermore, a specific embodiment of the large language model quantization attack defense device based on intersection and union constraints, which implements the aforementioned large language model quantization attack defense method based on intersection and union constraints, can be referred to... Figure 5 Flowchart for large language model quantification attack defense.

[0063] Furthermore, this invention also proposes a large language model quantization attack defense device based on intersection and union constraints. The large language model quantization attack defense device based on intersection and union constraints includes: a memory, a processor, and a large language model quantization attack defense program based on intersection and union constraints stored in the memory and executable on the processor. The large language model quantization attack defense program based on intersection and union constraints is configured to implement the steps of the large language model quantization attack defense method based on intersection and union constraints described above.

[0064] Furthermore, this embodiment of the invention also proposes a storage medium storing a large language model quantization attack defense program based on intersection and union constraints. When the large language model quantization attack defense program based on intersection and union constraints is executed by a processor, it implements the steps of the large language model quantization attack defense method based on intersection and union constraints described above.

[0065] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0066] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0068] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for defending against large language model quantization attacks based on intersection and union constraints, characterized in that, The large language model quantization attack defense method based on intersection and union constraints includes the following steps: Obtain the full-precision large language model to be defended and the quantization configuration of multiple targets; Based on each candidate parameter in the full-precision large language model, determine the quantization stability interval of the candidate parameter under each target quantization configuration, and determine the intersection interval and union interval based on each of the quantization stability intervals; The risk weight of each candidate parameter is determined based on the intersection interval, and the high-risk parameter is determined from the candidate parameters based on the risk weight; The high-risk parameters are corrected based on the intersection interval and the union interval to determine the corrected parameters, and a corrected full-precision large language model containing the corrected parameters is output. The step of correcting the high-risk parameter based on the intersection interval and the union interval includes: Determine the correction direction for the high-risk parameter, wherein the correction direction is the direction that causes the parameter value of the high-risk parameter to move away from the intersection interval; The upper limit of the correction magnitude of the high-risk parameter is determined based on the boundary of the union interval. The high-risk parameter is corrected according to the correction direction and the upper limit of the correction magnitude, so that the corrected parameter value is outside the intersection interval and inside the union interval; The step of correcting the high-risk parameter according to the correction direction and the upper limit of the correction magnitude further includes: Obtain clean calibration data, and input the clean calibration data into the full-precision large language model and the corrected full-precision large language model respectively, and extract the corresponding hidden state data; Calculate the hidden state difference between the hidden state data corresponding to the full-precision large language model and the hidden state data corresponding to the corrected full-precision large language model; The upper limit of the correction magnitude is adjusted based on the hidden state difference value to constrain the difference in model output before and after correction for the high-risk parameter; The step of determining the correction direction for the high-risk parameter includes: Calculate the boundary distance between the parameter value of the high-risk parameter and each boundary of the intersection interval; The correction direction is determined based on the boundary distance, and the correction direction is the direction that causes the high-risk parameter to move away from the intersection interval with minimal amplitude shift.

2. The method for defending against large language model quantization attacks based on intersection and union constraints as described in claim 1, characterized in that, The step of determining the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model includes: When the target quantization is configured as uniform quantization, the quantization bucket boundary of the candidate parameter is determined according to the quantization scale and quantization step size corresponding to the uniform quantization, and the first quantization stable interval of the candidate parameter under the uniform quantization is determined according to the quantization bucket boundary. When the target quantization is configured as codebook quantization, the second quantization stability interval of the candidate parameter under the codebook quantization is determined according to the decision boundary of adjacent items in the codebook of the codebook quantization. The quantization stability interval is determined based on the first quantization stability interval and the second quantization stability interval.

3. The method for defending against large language model quantization attacks based on intersection and union constraints as described in claim 1, characterized in that, The step of determining the risk weight of each candidate parameter based on the intersection interval, and determining the high-risk parameter from the candidate parameters based on the risk weight, includes: Calculate the width of the intersection interval; The risk weight of the candidate parameter is calculated based on the intersection interval width and a preset numerical stability term, wherein the risk weight is negatively correlated with the intersection interval width; The candidate parameters are sorted according to the risk weights, and the candidate parameters with the highest sorting ratio are determined as the high-risk parameters.

4. The method for defending against large language model quantization attacks based on intersection and union constraints as described in any one of claims 1 to 3, characterized in that, The parameter correction is applied to the full-precision large language model, wherein the corrected parameter values ​​are located outside the intersection interval, so as to destroy the vulnerable parameter mapping shared by multiple quantization attacks. The corrected parameter values ​​are located within the union interval to maintain the normal capability of the full-precision large language model in non-attack scenarios.

5. A defense device against large language model quantization attacks based on intersection and union constraints, characterized in that, The large language model quantization attack defense device based on intersection and union constraints includes: The acquisition module is used to acquire the full-precision large language model to be defended and the quantization configuration of multiple targets; The analysis module is used to determine the quantization stability interval of each candidate parameter under each target quantization configuration based on each candidate parameter in the full-precision large language model, and to determine the intersection interval and the union interval based on each quantization stability interval; The identification module is used to determine the risk weight of each candidate parameter based on the intersection interval, and to determine the high-risk parameter from the candidate parameters based on the risk weight; The correction module is used to correct the high-risk parameters based on the intersection interval and the union interval, determine the corrected parameters, and output a corrected full-precision large language model containing the corrected parameters. The step of correcting the high-risk parameter based on the intersection interval and the union interval includes: Determine the correction direction for the high-risk parameter, wherein the correction direction is the direction that causes the parameter value of the high-risk parameter to move away from the intersection interval; The upper limit of the correction magnitude of the high-risk parameter is determined based on the boundary of the union interval. The high-risk parameter is corrected according to the correction direction and the upper limit of the correction magnitude, so that the corrected parameter value is outside the intersection interval and inside the union interval; The step of correcting the high-risk parameter according to the correction direction and the upper limit of the correction magnitude further includes: Obtain clean calibration data, and input the clean calibration data into the full-precision large language model and the corrected full-precision large language model respectively, and extract the corresponding hidden state data; Calculate the hidden state difference between the hidden state data corresponding to the full-precision large language model and the hidden state data corresponding to the corrected full-precision large language model; The upper limit of the correction magnitude is adjusted based on the hidden state difference value to constrain the difference in model output before and after correction for the high-risk parameter; The step of determining the correction direction for the high-risk parameter includes: Calculate the boundary distance between the parameter value of the high-risk parameter and each boundary of the intersection interval; The correction direction is determined based on the boundary distance, and the correction direction is the direction that causes the high-risk parameter to move away from the intersection interval with minimal amplitude shift.

6. A defense device against large language model quantization attacks based on intersection and union constraints, characterized in that, The large language model quantization attack defense device based on intersection and union constraints includes: a memory, a processor, and a large language model quantization attack defense program based on intersection and union constraints stored in the memory and executable on the processor. The large language model quantization attack defense program based on intersection and union constraints is configured to implement the steps of the large language model quantization attack defense method based on intersection and union constraints as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a large language model quantization attack defense program based on intersection and union constraints. When the large language model quantization attack defense program based on intersection and union constraints is executed by the processor, it implements the steps of the large language model quantization attack defense method based on intersection and union constraints as described in any one of claims 1 to 4.

Images