Poisoning detection method for large language model

Abstract

The application relates to a poison detection method of a large language model and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a current activation value anomaly detection result of the large language model according to activation value time sequence evolution information and activation value distribution offset information of a first target neural network layer in the large language model at a current training step; determining a current semantic anomaly detection result of the large language model according to neighborhood distribution information of a current batch of training samples in a semantic space corresponding to the current training step; each training sample in the current batch of training samples comprises prompt information, positive response information corresponding to the prompt information and negative response information corresponding to the prompt information; and fusing the current activation value anomaly detection result, the current semantic anomaly detection result and a current meta-learning anomaly detection result of the large language model to obtain a current poison detection result of the large language model at the current training step. The method can reduce the hysteresis of poison detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for detecting poisoning in a large language model. Background Technology

[0002] Large language models are typically trained and optimized using reinforcement learning from human feedback (RLHF). However, during the RLHF training process of large language models, human feedback data may be maliciously poisoned, contaminating the training process and leading to erroneous inferences during the inference phase.

[0003] In related technologies, poisoning detection for large language models is mostly performed offline after the large language model has been trained. Therefore, if the large language model has been maliciously poisoned during the training process, the poisoned data has already contaminated the large language model by the time of poisoning detection. In other words, poisoning detection of large language models in related technologies is lagging. Summary of the Invention

[0004] Therefore, it is necessary to address the technical problem of lag in poison detection for large language models mentioned above, and to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for poison detection of large language models that can reduce the lag in poison detection.

[0005] Firstly, this application provides a poisoning detection method for large language models, including:

[0006] Determine the current training step of the large language model being trained, and obtain the current training step; the large language model includes multiple neural network layers;

[0007] Based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step, the current activation value anomaly detection result of the large language model under the current training step is determined.

[0008] Based on the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space, the current semantic anomaly detection result of the large language model under the current training step is determined; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information.

[0009] The current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model are fused to obtain the current poisoning detection result of the large language model under the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

[0010] Secondly, this application also provides a poisoning detection device for a large language model, comprising:

[0011] The training step determination module is used to determine the current training step of the large language model being trained, and to obtain the current training step; the large language model includes multiple neural network layers;

[0012] The activation value anomaly detection module is used to determine the current activation value anomaly detection result of the large language model in the current training step based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model in the current training step.

[0013] The semantic anomaly detection module is used to determine the current semantic anomaly detection result of the large language model under the current training step based on the neighborhood distribution information of the current batch of training samples in the semantic space corresponding to the current training step; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information;

[0014] The detection result determination module is used to fuse the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model to obtain the current poisoning detection result of the large language model under the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

[0015] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0016] Determine the current training step of the large language model being trained, and obtain the current training step; the large language model includes multiple neural network layers;

[0017] Based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step, the current activation value anomaly detection result of the large language model under the current training step is determined.

[0018] Based on the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space, the current semantic anomaly detection result of the large language model under the current training step is determined; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information.

[0019] The current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model are fused to obtain the current poisoning detection result of the large language model under the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

[0020] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0021] Determine the current training step of the large language model being trained, and obtain the current training step; the large language model includes multiple neural network layers;

[0022] Based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step, the current activation value anomaly detection result of the large language model under the current training step is determined.

[0023] Based on the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space, the current semantic anomaly detection result of the large language model under the current training step is determined; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information.

[0024] The current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model are fused to obtain the current poisoning detection result of the large language model under the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

[0025] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0026] Determine the current training step of the large language model being trained, and obtain the current training step; the large language model includes multiple neural network layers;

[0027] Based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step, the current activation value anomaly detection result of the large language model under the current training step is determined.

[0028] Based on the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space, the current semantic anomaly detection result of the large language model under the current training step is determined; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information.

[0029] The current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model are fused to obtain the current poisoning detection result of the large language model under the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

[0030] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for detecting poisoning in large language models can obtain the current activation value anomaly detection result of the large language model in the current training step by using the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step. By using the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space, the current semantic anomaly detection result of the large language model in the current training step can be obtained. By fusing the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model in the current training step, the current poisoning detection result of the large language model in the current training step can be obtained. Based on the above process, poisoning detection can be performed on the large language model that is being trained. Compared with offline poisoning detection after training is completed, poisoning behavior can be detected in a timely manner during training, reducing the lag in poisoning detection. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only one embodiment of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1This is a diagram illustrating the application environment of a poisoning detection method for a large language model in one embodiment.

[0033] Figure 2 This is a flowchart illustrating a poisoning detection method for a large language model in one embodiment;

[0034] Figure 3 This is a flowchart illustrating the poisoning detection method for a large language model in another embodiment;

[0035] Figure 4 This is a flowchart illustrating the steps for periodically determining meta-learning anomaly detection results in one embodiment;

[0036] Figure 5 This is a structural block diagram of a poison detection device for a large language model in one embodiment;

[0037] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0039] It is understood that terms such as "first" and "second" in this application are used only to distinguish similar objects and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. The term "connection" in the embodiments of this application refers to various connection methods, such as direct or indirect connections, to achieve communication between devices; this application does not impose any limitations on this.

[0040] It is understandable that "at least one" refers to one or more, while "multiple" refers to two or more.

[0041] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0043] The poisoning detection method for large language models provided in this application can be applied to, for example... Figure 1 In the application environment shown, server 102 communicates with terminal 104 via a network. A data storage system can store the data that server 102 needs to process. This data storage system can be integrated onto server 102 or located on a cloud or other network server. Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal 104 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Furthermore, server 102 and terminal 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0044] It should be noted that both server 102 and terminal 104 can be used individually to execute the poisoning detection method for the large language model provided in this embodiment, or they can be used together to execute the poisoning detection method for the large language model provided in this embodiment.

[0045] For example, firstly, server 102 determines the current training step of the large language model being trained, thus obtaining the current training step; the large language model includes multiple neural network layers; then, server 102 determines the current activation value anomaly detection result of the large language model in the current training step based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step; next, server 102 determines the current semantic anomaly detection result of the large language model in the current training step based on the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space; the current batch of training samples includes multiple training samples, each training sample including prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information; then, server 102 fuses the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model to obtain the current poisoning detection result of the large language model in the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process. After the server 102 completes the training of the large language model, it receives the input information sent by the terminal 104, inputs the input information into the large language model, the large language model returns the output information, and the server 102 returns the output information to the terminal 104.

[0046] In one embodiment, such as Figure 2 As shown, a poisoning detection method for large language models is provided, and this method is applied to... Figure 1 Taking a server as an example, it can be understood that this method can also be applied to a terminal, and can also be applied to a system that includes both a server and a terminal, and is implemented through the interaction between the server and the terminal. The method includes the following steps:

[0047] Step S202: Determine the current training step of the large language model being trained, and obtain the current training step.

[0048] Among them, the large language model is a large language model that is being trained, and the large language model includes multiple neural network layers; in practical applications, the large language model is the power grid large language model.

[0049] It should be noted that in the iterative deep learning training process of neural networks, training steps and batches are the core concepts describing the training process. A training step is the smallest unit for updating model parameters; that is, a training step refers to a complete process of updating the model parameters. A batch is a grouping unit of training data; that is, the training data is divided into multiple batches of training samples. A training step updates the model parameters once using one batch of training data.

[0050] In this step, the server obtains the large language model that is being trained and determines the current training step of the large language model, and defines the current training step as the current training step.

[0051] Step S204: Based on the temporal evolution information and distribution offset information of the activation values ​​of the first target neural network layer in the large language model under the current training step, determine the current activation value anomaly detection result of the large language model under the current training step.

[0052] The first target neural network layer is a subset of the multiple neural network layers in the large language model, and there are multiple first target neural network layers. Furthermore, the first target neural network layer includes at least one shallow network layer, at least one medium network layer, and at least one high network layer among the multiple neural network layers, thereby enabling it to capture changes in shallow features, evolution of medium representations, and changes in high-level semantics. For example, taking a 12-layer Transformer model as an example, the first target neural network layer includes the 3rd, 7th, and 11th neural network layers.

[0053] Among them, the activation value temporal evolution information of the first target neural network layer is used to characterize the temporal evolution of the activation value of the first target neural network layer during the training process; the activation value distribution offset information of the first target neural network layer is used to characterize the offset of the distribution of the activation value of the first target neural network layer relative to the corresponding reference distribution.

[0054] Among them, the activation value anomaly detection results of the large language model are used to characterize whether the activation values ​​of the large language model are abnormal during the training process.

[0055] In each training step, the server needs to obtain the activation values ​​of the large language model after the training step is completed, as the activation values ​​of the large language model in that training step. In this step, for each first target neural network layer, the server determines the temporal evolution information and activation value distribution offset information of the activation value of the first target neural network layer in the current training step based on the current activation value of the first target neural network layer in the current training step. Then, the server integrates the temporal evolution information and activation value distribution offset information of the activation values ​​of each first target neural network layer in the current training step to determine the anomaly detection result of the current activation value of the large language model in the current training step.

[0056] Step S206: Based on the neighborhood distribution information of the current batch of training samples in the semantic space corresponding to the current training step, determine the current semantic anomaly detection result of the large language model under the current training step.

[0057] The current batch of training samples includes multiple training samples. Each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information. The prompt information refers to the prompt words input into the large language model. The positive response information corresponding to the prompt information refers to the response information that is labeled as matching the prompt information. The negative response information corresponding to the prompt information refers to the response information that is labeled as not matching the prompt information. The response information refers to the information that the large language model needs to output.

[0058] Among them, the neighborhood distribution information is used to characterize the distribution of each training sample and its corresponding neighboring training samples in the current batch of training samples. The neighboring training samples of each training sample are the training samples that are closest to the training sample in the semantic space in the current batch of training samples.

[0059] Among them, the semantic anomaly detection results of the large language model are used to characterize whether the semantics of the training samples of the large language model are abnormal during the training process.

[0060] In this step, for each training sample in the current batch of training samples, the server determines the neighboring training samples in the semantic space, and determines the neighborhood distribution information of the training sample based on the training sample and its corresponding neighboring training samples; then, the server determines the neighborhood distribution information of each training sample as the current semantic anomaly detection result of the large language model in the current training step.

[0061] Step S208: The current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model are fused to obtain the current poisoning detection result of the large language model in the current training step.

[0062] Among them, the meta-learning anomaly detection results of the large language model are used to characterize whether the training process of the large language model is abnormal.

[0063] The current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during training. In some embodiments, the server determines the meta-learning anomaly detection result at each training step, while in other embodiments, the server determines the meta-learning anomaly detection result every certain number of training steps.

[0064] Among them, the meta-learning anomaly detection results of the large language model are used to characterize whether the gradient and loss values ​​of the large language model are abnormal during the training process.

[0065] The current poisoning detection results are used to characterize whether the large language model has been poisoned.

[0066] In this step, the server performs multi-dimensional fusion processing on the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result to obtain the current poisoning detection result of the large language model in the current training step. For example, taking the training samples in the current batch of training samples as units, the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result are fused at the sample level to obtain the sample-level anomaly detection result for each training sample. Then, the sample-level anomaly detection results of each training sample are fused at the batch level to obtain the batch-level anomaly detection result for the current training step. Finally, the batch-level anomaly detection results of the historical training steps and the batch-level anomaly detection results of the current training step are combined globally to obtain the current poisoning detection result.

[0067] In the aforementioned poisoning detection method for large language models, the server obtains the current activation value anomaly detection result of the large language model in the current training step by using the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step. The server obtains the current semantic anomaly detection result of the large language model in the current training step by using the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space. By fusing the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model in the current training step, the server obtains the current poisoning detection result of the large language model in the current training step. Based on the above process, poisoning detection can be performed on the large language model that is being trained. Compared with offline poisoning detection after training is completed, poisoning behavior can be detected in a timely manner during training, reducing the lag of poisoning detection.

[0068] In some embodiments, the server executes steps S202 to S208 above in each training step to perform a poisoning detection on the large language model in each training step.

[0069] In one embodiment, the number of first target neural network layers is at least one.

[0070] Step S204 above, based on the activation value temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step, determines the current activation value anomaly detection result of the large language model under the current training step, including the following steps: For each first target neural network layer, perform anomaly detection on the activation value temporal evolution information and activation value distribution offset information of the first target neural network layer under the current training step, and obtain the anomaly detection result of the activation value temporal evolution information and activation value distribution offset information of the first target neural network layer under the current training step; fuse the anomaly detection results of the activation value temporal evolution information and activation value distribution offset information of each first target neural network layer under the current training step to obtain the current activation value anomaly detection result.

[0071] In this embodiment, both the activation value temporal evolution information and the activation value distribution offset information are in numerical form. For each first target neural network layer, the server presets a corresponding activation value temporal evolution threshold and activation value distribution offset threshold.

[0072] For each first target neural network layer, the server performs anomaly detection on the temporal evolution of the activation values ​​of that first target neural network layer in the current training step based on the corresponding activation value temporal evolution threshold, obtaining the anomaly detection result of the activation value temporal evolution information of the first target neural network layer in the current training step. For example, if the value of the activation value temporal evolution information of the first target neural network layer in the current training step does not meet the activation value temporal evolution threshold, the anomaly detection result of the activation value temporal evolution information of the first target neural network layer in the current training step is determined to be anomaly. Furthermore, the server performs anomaly detection on the activation value temporal evolution information of the first target neural network layer in the current training step based on the corresponding preset activation value distribution offset threshold. Anomaly detection is performed on the activation value distribution offset information to obtain the anomaly detection result of the activation value distribution offset information of the first target neural network layer under the current training step. For example, if the value of the activation value distribution offset information of the first target neural network layer under the current training step does not meet the activation value distribution offset threshold, the anomaly detection result of the activation value distribution offset information of the first target neural network layer under the current training step is determined to be abnormal. Then, the server fuses the anomaly detection result of the activation value temporal evolution information of the first target neural network layer under the current training step and the anomaly detection result of the activation value distribution offset information according to the indicator function to obtain the activation value anomaly score of the first target neural network layer under the current training step.

[0073] Next, the server takes the average of the abnormal activation scores of each first target neural network layer in the current training step to obtain the abnormal activation score of the large language model in the current training step. The server determines the abnormal activation score of the large language model in the current training step as the current abnormal activation detection result of the large language model in the current training step.

[0074] In practical applications, the temporal evolution information of activation values ​​includes the acceleration of activation value changes and the divergence information of activation value trajectories. The temporal evolution threshold of activation values ​​includes the threshold for the acceleration of activation value changes and the threshold for the divergence of activation value trajectories. The server determines the abnormal activation value score of the large language model at a certain training step based on the following formula 1:

[0075] (Formula 1)

[0076] in, For the first One training step, For the first The first target neural network layer, The number of layers in the first target neural network; For the first The first target neural network layer in the first... Abnormal activation scores under each training step For large language models in the first Abnormal activation scores under each training step; This is an indicator function; its value is 1 when the input meets the condition, and 0 when the input does not meet the condition. For the first The first target neural network layer in the first... Acceleration of activation value changes under each training step For the first The first target neural network layer in the first... The threshold for the acceleration of activation value changes under each training step is typically set to 0.1. For the first The first target neural network layer in the first... Information on the divergence of activation value trajectories under each training step For the first The first target neural network layer in the first... The threshold for activation trajectory divergence under each training step is typically set to 0.5. For the first The first target neural network layer in the first... Activation value distribution offset information under each training step For the first The first target neural network layer in the first... The activation value distribution offset threshold under each training step is typically set to 0.05. These are the weights corresponding to the acceleration of activation value change, the divergence information of activation value trajectory, and the offset information of activation value distribution, respectively. In practical applications... .

[0077] In this embodiment, poisoned samples often cause sudden changes, oscillations, or violent fluctuations in activation values. These anomalies are particularly evident in temporal derivatives and trajectory divergence. Therefore, based on the temporal evolution information of activation values, the server can capture the abnormal changes in the activation space caused by poisoned samples. By introducing regularized distance ratios and outlier pruning, it can also ensure the numerical stability of calculations during high-dimensional random training.

[0078] In one embodiment, the activation value temporal evolution information and activation value distribution offset information of each first target neural network layer in the current training step are obtained as follows: For each first target neural network layer, based on the activation values ​​of the first target neural network layer in multiple first target training steps, the activation value change acceleration and activation value distance of the first target neural network layer in the current training step are determined; based on the ratio between the activation value distances of the first target neural network layer in each first target training step, the activation value distance ratio of the first target neural network layer in the current training step is obtained; the activation value distance ratio of the first target neural network layer in multiple second target training steps is fused to obtain the activation value trajectory divergence information of the first target neural network layer in the current training step; based on the activation value change acceleration and activation value trajectory divergence information of the first target neural network layer in the current training step, the activation value temporal evolution information of the first target neural network layer in the current training step is obtained; the distribution offset information between the activation value distribution information of the first target neural network layer in the current training step and the reference activation value distribution information is determined to obtain the activation value distribution offset information of the first target neural network layer in the current training step.

[0079] Among them, multiple first-target training steps include the current training step and the previous training step.

[0080] The multiple second-objective training steps include multiple consecutive training steps, with the current training step being the ending training step. That is, the multiple second-objective training steps include the current training step and the multiple training steps preceding the current training step. In practical applications, the number of multiple second-objective training steps is a first preset number.

[0081] In this embodiment, the temporal evolution information of the activation values ​​of the first target neural network layer in the current training step includes the acceleration of the activation value change and the divergence information of the activation value trajectory in the current training step. For each first target neural layer, the server determines the current training step and the previous training step as multiple first target training steps, and determines the current training step and the multiple training steps before the current training step as multiple second target training steps.

[0082] On one hand, regarding the acceleration of activation value changes, the server determines the rate of change of the activation values ​​of the first target neural network layer between the two first target training steps based on the activation values ​​of the first target neural network layer under the two first target training steps. This rate of change is then used as the acceleration of activation value changes of the first target neural network layer in the current training step. In practical applications, the server determines the acceleration of activation value changes of the first target neural network layer in a certain training step based on the following formula 2:

[0083] (Formula 2)

[0084] in, For the first The first target neural network layer in the first... Activation values ​​under each training step; For the first The first target neural network layer in the first... The training steps and the first The rate of change of activation values ​​between training steps; For the first The first target neural network layer in the first... The training steps and the first The acceleration of the change in activation values ​​between training steps.

[0085] On the other hand, regarding the activation value trajectory divergence information, the server determines the activation value distance between the activation values ​​of the first target neural network layer in the two first target training steps based on the activation values ​​of the first target neural network layer in the two first target training steps, and determines the ratio of the activation value distances of the first target neural network layer in the two first target training steps as the activation value distance ratio of the first target neural network layer in the current training step; then the server takes the average of the activation value distance ratios of the first target neural network layer in multiple second target training steps to obtain the initial activation value trajectory divergence index information of the first target neural network layer in the current training step. Then, the server performs outlier pruning on this initial activation value trajectory divergence index information to obtain the activation value trajectory divergence index information of the first target neural network layer in the current training step. In practical applications, the server determines the activation value trajectory divergence information of the first target neural network layer in a certain training step based on the following formula 3:

[0086] (Formula 3)

[0087] in, For the first The first target neural network layer in the first... The training steps and the first The distance between activation values ​​under each training step; For the first The first target neural network layer in the first... The ratio of activation values ​​to training steps; The number of training steps for the second objective; For the first The first target neural network layer in the first... Information on the divergence of the initial activation value trajectory under each training step. For the first The first target neural network layer in the first... Information on the divergence of activation value trajectories under each training step For the cropping operation, For the first The first target neural network layer in the first... Preset clipping thresholds for each training step.

[0088] On the other hand, regarding the activation value distribution offset information, both the activation value distribution information and the reference activation value distribution information are in the form of distribution functions. Each first target neural network layer has corresponding reference activation value distribution information in each training step. The server determines the activation value distribution information of the first target neural network layer in the current training step based on the activation values ​​of the first target neural network layer in the current training step, and measures the offset between the activation value distribution information of the first target neural network layer in the current training step and the reference activation value distribution information by using the maximum mean difference, thus obtaining the activation value distribution offset information of the first target neural network layer in the current training step. In practical applications, the server determines the activation value distribution offset information of the first target neural network layer in a certain training step based on the following formula 4:

[0089] (Formula 4)

[0090] in, For the first The first target neural network layer in the first... Activation value distribution offset information under each training step; This represents the maximum mean difference. For the first The first target neural network layer in the first... Activation value distribution information under each training step For the first The first target neural network layer in the first... Information on the distribution of reference activation values ​​under each training step.

[0091] In this embodiment, the server can combine the activation values ​​under the current training step and the activation values ​​under the historical training steps to determine the temporal evolution information and the distribution offset information of the activation values ​​under the previous training steps.

[0092] In one embodiment, before determining the distribution offset information between the activation value distribution information and the reference activation value distribution information of the first target neural network layer in the current training step, the method further includes the following step: determining the reference activation value distribution information of the first target neural network layer in the current training step: updating the historical reference activation value distribution information of the first target neural network layer according to the activation value distribution information of the first target neural network layer in the current training step, thereby obtaining the reference activation value distribution information of the first target neural network layer in the current training step.

[0093] Among them, the historical reference activation value distribution information is the reference activation value distribution information of the first target neural network layer in the previous training step of the current training step.

[0094] In this embodiment, the server updates the reference activation value distribution information of the first target neural network layer in the previous training step based on the activation value distribution information of the first target neural network layer in the current training step, thereby obtaining the reference activation value distribution information of the first target neural network layer in the current training step.

[0095] In practical applications, as shown in Formula 5, the server uses an exponentially weighted moving average method to dynamically update the distribution information of the reference activation values:

[0096] (Formula 5)

[0097] in, For the first The first target neural network layer in the first... Information on the distribution of reference activation values ​​under each training step; For the first The first target neural network layer in the first... Activation value distribution information under each training step; This is the smoothing coefficient, typically taken as 0.95. Furthermore, It was built using clean training samples.

[0098] In this embodiment, the server can adapt to the natural evolution of the large language model during the training process and reduce false tests by dynamically updating the reference activation value distribution information.

[0099] In one embodiment, the neighborhood distribution information of the current batch of training samples includes the neighborhood distribution information of each training sample.

[0100] Step S206 above, which determines the current semantic anomaly detection result of the large language model under the current training step based on the neighborhood distribution information of the current batch of training samples in the semantic space, includes the following steps: for each current training sample, determine the neighboring training samples of the current training sample from the other training samples based on the semantic vector of each training sample; determine the neighborhood distribution information of the current training sample based on the current training sample and the neighboring training samples; perform anomaly detection on the neighborhood distribution information of the current training sample to obtain the semantic anomaly detection result of the current training sample; and obtain the current semantic anomaly detection result based on the semantic anomaly detection results of each training sample.

[0101] The semantic vector of each training sample is the differential encoding result between the positive and negative response information of the training sample.

[0102] The current training sample is any one of the training samples in the current batch, and the remaining training samples are all training samples in the current batch except for the current training sample.

[0103] Among them, the neighbor training samples are training samples whose distance from the current training sample meets the distance condition, such as training samples whose distance from the current training sample is less than or equal to a preset distance threshold, or the K training samples closest to the current training sample.

[0104] In this embodiment, firstly, for each training sample in the current batch of training samples, the server performs differential encoding on the positive and negative response information in the training sample based on an encoder derived from the large language model, to obtain the semantic vector of the training sample. In practical applications, the training sample is denoted as... ,in, For the purpose of displaying information, This is a positive response message. Negative response information; the semantic vector of the training sample is... .

[0105] Then, the server iterates through each training sample in the current batch. For each training sample encountered, the server designates it as the current training sample and, based on the semantic vectors of each training sample, searches for the K nearest neighboring training samples among the remaining training samples. This results in a list of the current training sample's multiple neighboring training samples. In practical applications, K is adaptively determined based on the number N of training samples in the current batch. For example… Next, the server determines the neighborhood distribution information of the current training sample based on the current training sample and its multiple neighboring training samples. The neighborhood distribution information is in numerical form. The server performs anomaly detection on the neighborhood distribution information of the current training sample based on a preset neighborhood distribution threshold to obtain the semantic anomaly detection result of the current training sample.

[0106] Finally, the server determines the semantic anomaly detection results of each training sample in the current batch of training samples as the current semantic anomaly detection results of the large language model in the current training step.

[0107] In this embodiment, the server can determine whether there are poisoned samples that deviate from the normal distribution in the current batch of training samples by analyzing the distribution characteristics of the training samples in the semantic embedding space.

[0108] In one embodiment, the above step of determining the neighborhood distribution information of the current training sample based on the current training sample and neighboring training samples includes the following steps: determining the neighborhood density, neighborhood isolation information, and deviation information between the current training sample and neighboring training samples based on the semantic vector of the current training sample and the semantic vector of neighboring training samples; determining the first neighborhood consistency information and the second neighborhood consistency information of the current training sample based on the deviation information between the current training sample and neighboring training samples; and determining the neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information of the current training sample as the neighborhood distribution information of the current training sample.

[0109] The neighborhood density is used to characterize the distribution density of the current training sample and its corresponding neighboring training samples.

[0110] Among them, neighborhood isolation information is used to characterize the degree of isolation of the current training sample compared to its corresponding neighboring training samples.

[0111] Among them, the first neighborhood consistency information is used to characterize the average value of the deviation information between the current training sample and each of its neighboring training samples.

[0112] The second neighborhood consistency information is used to characterize the proportion of the target neighbor training sample in each neighbor training sample of the current training sample; the target neighbor training sample is the neighbor training sample whose corresponding deviation information satisfies the deviation information condition. For example, the deviation information is in numerical form, and the target neighbor training sample is the neighbor training sample whose corresponding deviation information value is less than or equal to a preset deviation threshold.

[0113] The neighborhood distribution information for each training sample includes neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information.

[0114] In this embodiment, the neighborhood distribution information includes neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information.

[0115] Regarding neighborhood density, the server determines the neighborhood density of the current training sample based on the semantic vector of the current training sample and the semantic vectors of its neighboring training samples. In practical applications, the server calculates the neighborhood density of each training sample according to Formula 6:

[0116] (Formula 6)

[0117] in, For the first Neighborhood density of each training sample; For the first The semantic vectors of each training sample; For the first The training sample of the th training sample The semantic vector of a training sample from its neighbors; the lower the neighborhood density of a training sample, the more likely the training sample is to be an abnormal sample, that is, more likely it is a poisoned sample.

[0118] Regarding neighborhood isolation information, the server determines the neighborhood isolation information of the current training sample based on the semantic vector of the current training sample and the semantic vectors of each neighboring training sample. In practical applications, the server calculates the neighborhood isolation information of each training sample according to Formula 7:

[0119] (Formula 7)

[0120] in, For the first The neighborhood isolation information of a training sample; the higher the neighborhood isolation information of a training sample, the more likely the training sample is to be an abnormal sample, that is, more likely it is a poisoned sample.

[0121] Based on the first neighborhood consistency information and the second neighborhood consistency information, the server calculates the cosine similarity between the semantic vector of the current training sample and the semantic vector of each neighboring training sample, and determines the calculated cosine similarity as the deviation information between the current training sample and each neighboring training sample.

[0122] Then, the server calculates the average of the deviation information between the current training sample and each neighboring training sample, and determines this average as the first neighborhood consistency information of the current training sample. In practical applications, the server calculates the first neighborhood consistency information of each training sample according to Formula 8:

[0123] (Formula 8)

[0124] in, For the first The first neighborhood consistency information of a training sample; the lower the first neighborhood consistency information of a training sample, the more likely the training sample is to have semantic differences from normal samples, that is, the more likely it is to be a poisoned sample.

[0125] Simultaneously, the server searches for neighbor training samples whose deviation information is less than or equal to a preset deviation threshold, such as neighbor training samples with deviation information less than or equal to 0.5, and identifies the found neighbor training samples as target neighbor training samples. Next, the server calculates the ratio of the number of target neighbor training samples to the number of neighbor training samples, which is the proportion of target neighbor training samples in all neighbor training samples. Finally, the server determines this ratio as the second neighborhood consistency information of the current training sample. The higher the second neighborhood consistency information of the training sample, the more likely the training sample is to have semantic differences from normal samples, that is, the more likely it is to be a poisoned sample.

[0126] In this embodiment, normal RLHF training samples often form dense clusters in the semantic space, while poisoned samples often appear as isolated outliers due to their abnormality. Therefore, by calculating multiple indicators such as neighborhood density, neighborhood isolation degree, first neighborhood consistency information and second neighborhood consistency information, the degree of abnormality of training samples can be measured from different perspectives, thereby determining whether there are poisoned samples in the current batch of training samples.

[0127] In one embodiment, the number of neighborhood distribution information for each training sample is multiple.

[0128] The above steps involve performing anomaly detection on the neighborhood distribution information of the current training sample to obtain the semantic anomaly detection result of the current training sample. This includes the following steps: for each current training sample, performing anomaly detection on each neighborhood distribution information of the current training sample to obtain the anomaly detection result of each neighborhood distribution information; and fusing the anomaly detection results of each neighborhood distribution information to obtain the current semantic anomaly detection result of the current training sample.

[0129] In this embodiment, each neighborhood distribution information is in numerical form, and the server presets a corresponding neighborhood distribution threshold for each neighborhood distribution information. For each current training sample, the server performs anomaly detection on each neighborhood distribution information of the current training sample based on the preset neighborhood distribution threshold, and obtains the anomaly detection result for each neighborhood distribution information of the current training sample. For example, if the value of the neighborhood distribution information does not meet the corresponding neighborhood distribution threshold, the anomaly detection result of the neighborhood distribution information is determined to be abnormal. Then, the server fuses the anomaly detection results of each neighborhood distribution information of the current training sample according to the indicator function to obtain the semantic anomaly score of the current training sample. The server determines the semantic anomaly score of the current training sample as the semantic anomaly detection result of the current training sample.

[0130] In practical applications, the neighborhood distribution information of each training sample includes neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information. Preset neighborhood distribution thresholds include neighborhood density threshold, neighborhood isolation threshold, first neighborhood consistency threshold, and second neighborhood consistency threshold. The server determines the semantic anomaly score of each training sample based on the following formula 9:

[0131] (Formula 9)

[0132] in, For the first The semantic anomaly score of each training sample; For the first The neighborhood density of each training sample. The neighborhood density threshold; For the first Neighborhood isolation information of each training sample The threshold for neighborhood isolation; For the first The first The first neighborhood consistency information of each training sample The first neighborhood consistency threshold; For the first The first The second neighborhood consistency threshold for each training sample The second neighborhood consistency threshold; These are the weights corresponding to neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information, respectively. In practical applications, .

[0133] In this embodiment, the server can combine the neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information of the training samples to obtain the semantic anomaly detection results of the training samples.

[0134] In one embodiment, the current meta-learning anomaly detection result is obtained as follows: upon reaching the third objective training step, multiple fourth objective training steps are determined; based on the gradient of the second objective neural network layer in the large language model under each fourth objective training step, the current gradient information of the large language model is determined; based on the loss value of the large language model under each fourth objective training step, the current loss value information of the large language model is determined; the current gradient information and the current loss value information are combined to obtain the current meta-learning signature information of the large language model; anomaly detection is performed on the current meta-learning signature information to obtain the current meta-learning anomaly detection result.

[0135] The third objective training step is the training step that reaches the preset number of training steps between the fifth objective training step and the fifth objective training step. For example, if the fifth objective training step is the 100th step and the preset number of training steps is 100, then the third objective training step is reached at the 200th training step.

[0136] The fifth objective training step refers to the training step in which the large language model was last determined when the meta-learning anomaly detection result of the large language model was last determined. For example, if the training step in which the large language model was last determined when the meta-learning anomaly detection result of the large language model was last determined was step 100, then the fifth objective training step is step 100.

[0137] The multiple fourth-objective training steps include a series of consecutive training steps, with the third-objective training step as the ending training step. That is, the multiple fourth-objective training steps include the third-objective training step and the multiple training steps preceding the third-objective training step. In practical applications, the number of multiple second-objective training steps is a second preset number. Furthermore, the second preset number is the same as the preset number of training steps. That is, assuming the fifth-objective training step is the 100th step and the preset number of training steps is 100 steps, then the third-objective training step is the 200th training step, and the multiple fourth-objective training steps are from the 101st training step to the 200th training step.

[0138] The second target neural network layer is a portion of the neural network layers near the output end among the multiple neural network layers of the large language model, and there are multiple second target neural network layers; further, the second target neural network layer includes the last 2 to 3 neural network layers among the multiple neural network layers.

[0139] In this embodiment, in each training step, the server first determines whether the third target training step has been reached, that is, whether the number of training steps between the current training step and the previous training step that determined the meta-learning anomaly detection result of the large language model has reached the preset number of training steps.

[0140] If the third target training step is reached, then, for each second target neural network layer, the server determines the gradient characteristics of the second target neural network layer based on the gradient of the second target neural network layer in each fourth target training step, including but not limited to the mean, standard deviation, coefficient of variation, gradient direction consistency, and gradient change smoothness of the gradient norm; then, the server determines the gradient characteristics of each second target neural network layer as the current gradient information of the large language model.

[0141] On the other hand, the server determines the current loss value information of the large language model based on the loss value of the large language model at each fourth objective training step, including but not limited to loss curve features and loss curvature features. The loss curve features include but are not limited to loss descent rate, loss oscillation amplitude, and loss landscape roughness. The loss curvature features include but are not limited to Hessian matrix features calculated using the diagonal Hessian approximation and Hutchinson trace estimation, such as the trace, norm, and maximum value of the diagonal elements.

[0142] Then, the server concatenates the current gradient information and the current loss value information of the large language model to obtain the current meta-learning signature information of the large language model. Next, the server performs anomaly detection on the current meta-learning signature information through a pre-trained meta-learning detector. The meta-learning detector outputs the anomaly probability of the current meta-learning signature information, and the server determines the anomaly probability of the current meta-learning signature information as the current meta-learning anomaly detection result of the large language model.

[0143] In practical applications, during the training phase of the meta-learning detector, the server simulates various poisoning attack types, such as label flipping, backdoor injection, distribution shift, gradient pollution, etc., and collects meta-learning signature information during normal training and poisoning training meta-learning signature information on a small-scale large language model to train the meta-learning detector.

[0144] In this embodiment, the current meta-learning anomaly detection result obtained each time is used in the next preset number of training steps, so as to avoid running the computationally intensive meta-learning signature information anomaly detection every time poisoning is detected, thus balancing the detection effect and computational efficiency of poisoning detection.

[0145] In one embodiment, the current semantic anomaly detection result includes the semantic anomaly detection result for each of the training samples.

[0146] Step S208 above, which fuses the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model to obtain the current poisoning detection result of the large language model in the current training step, includes the following steps: For each training sample, fuse the current activation value anomaly detection result, the current meta-learning anomaly detection result, and the semantic anomaly detection result of the training sample to obtain the sample-level anomaly detection result of the training sample; fuse the sample-level anomaly detection results of each training sample to obtain the batch-level anomaly detection result of the large language model in the current training step; fuse the batch-level anomaly detection results of the large language model in multiple sixth-objective training steps to determine the comprehensive batch-level anomaly detection result of the large language model in the current training step; fuse the batch-level anomaly detection result of the large language model in the current training step and the comprehensive batch-level anomaly detection result to obtain the current poisoning detection result of the large language model in the current training step.

[0147] Among them, the multiple sixth-objective training steps are a series of consecutive training steps, with the current training step being the ending training step; that is, the multiple sixth-objective training steps include the current training step and the multiple training steps preceding the current training step; in practical applications, the number of multiple second-objective training steps is the third preset number.

[0148] In this embodiment, the server performs sample-level fusion, batch-level fusion, and global fusion on the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result to obtain the current poisoning detection result of the large language model in the current training step.

[0149] For sample-level fusion, the server fuses the current activation value anomaly detection result, the current meta-learning anomaly detection result, and the semantic anomaly detection result for each training sample in the current batch of training samples to obtain the sample-level anomaly detection result for that training sample. In practical applications, the server determines the sample-level anomaly detection result for each training sample using the following formula 10:

[0150] (Formula 10)

[0151] in, For the first The training step of the first... Sample-level anomaly detection results for each training sample; This represents the current meta-learning anomaly detection result; These are the weights corresponding to the anomaly detection results for the current activation value, the current meta-learning anomaly detection results, and the semantic anomaly detection results, respectively. In practical applications, .

[0152] For batch-level fusion, the server merges the sample-level anomaly detection results of each training sample in the current batch to obtain the batch-level anomaly detection result of the large language model at the current training step. In practical applications, the server determines the batch-level anomaly detection result of the large language model at a certain training step using the following formula 11:

[0153] (Formula 11)

[0154] in, For large language models in the first Batch-level anomaly detection results under each training step; For the first The number of training samples in the current batch of training samples corresponding to each training step; The batch-level fusion weight is usually set to 0.2; the first term in Formula 11 represents the average value, which is used to reflect the overall degree of anomaly; the second term represents the maximum value, which is used to capture extreme anomaly samples.

[0155] For global fusion, firstly, the server identifies the current training step and the multiple training steps preceding it as multiple sixth-objective training steps. Then, based on the batch-level anomaly detection results of the large language model under these multiple sixth-objective training steps, the server determines the average trend and upward trend of the batch-level anomaly detection results of the large language model under the current training step. These average trend and upward trend are then used as the comprehensive batch-level anomaly detection result of the large language model under the current training step. Next, the server fuses the batch-level anomaly detection result and the comprehensive batch-level anomaly detection result of the large language model under the current training step to obtain the current poisoning detection result of the large language model under the current training step. In practical applications, the server determines the current poisoning detection result of the large language model under a certain training step using the following formula 12:

[0156] (Formula 12)

[0157] in, The number of training steps for the sixth objective; For large language models in the first Average trend of batch-level anomaly detection results under each training step; For large language models in the first The upward trend of batch-level anomaly detection results under each training step; For large language models in the first Current poisoning detection results under each training step; These represent the average trend of batch-level anomaly detection results, the upward trend of batch-level anomaly detection results, and the weights corresponding to batch-level anomaly detection results, respectively. In practical applications... .

[0158] In this embodiment, the server can perform multi-dimensional fusion of the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result, thereby determining the current poisoning detection result of the large language model in the current training step. Among them, sample-level fusion reflects the degree of anomaly of each training sample; batch-level fusion considers both the average degree of anomaly of training samples within a batch and captures extreme anomaly samples within a batch; global-level fusion combines historical trends to determine whether there are abnormal trends in the overall training process. Through the above multi-granularity fusion, short-term anomalies and long-term trends can be combined to improve the accuracy of detection.

[0159] In one embodiment, after obtaining the current poisoning detection result of the large language model in the current training step in step S208 above, the following steps are also included: if the current poisoning detection result indicates that the large language model has not been poisoned, train the large language model for the next training step according to the next batch of training samples; if the current poisoning detection result indicates that the large language model has been poisoned, suspend the training of the large language model.

[0160] In this embodiment, if the current poisoning detection result indicates that the large language model has not been poisoned, the server obtains the next batch of training samples and trains the large language model for the next training step based on the next batch of training samples; if the current poisoning detection result indicates that the large language model has been poisoned, the server pauses the training of the large language model.

[0161] In practical applications, if the current poisoning detection result is less than the first risk threshold, the server determines the poisoning risk level of the large language model to be normal. If the current poisoning detection result is greater than or equal to the second risk threshold and less than the third risk threshold, the server determines the poisoning risk level of the large language model to be medium risk. If the current poisoning detection result is greater than or equal to the third risk threshold, the server determines the poisoning risk level of the large language model to be high risk. If the poisoning risk level of the large language model is normal or medium risk, the server continues training the large language model for the next step and caches the model parameters of the large language model under the current training step. If the poisoning risk level of the large language model is high risk, the server immediately freezes the model parameters of the large language model and rolls back to the most recently cached model parameters, while generating a poisoning risk report for manual verification. Furthermore, if the poisoning risk level of the large language model is medium risk, the server reduces the learning rate for training.

[0162] In this embodiment, the server adopts a graded response mechanism to take intervention measures of different intensities for different risk levels. While ensuring security, it minimizes the impact on training efficiency and can ensure that it can quickly recover to a safe state in the event of a poisoning attack, avoiding continuous contamination of the model. It can both detect and prevent poisoning attacks in a timely manner and avoid oversensitivity affecting normal training.

[0163] In a specific embodiment, such as Figure 3 As shown, the poisoning detection method for large language models provided in this application includes the following steps:

[0164] Step S302: For each training step of the large language model, the training step is determined as the current training step.

[0165] Step S304: For each first target neural network layer of the large language model, determine the activation value change acceleration and activation value distance of the first target neural network layer in the current training step based on the activation values ​​of the first target neural network layer in multiple first target training steps; the multiple first target training steps include the current training step and the previous training step.

[0166] Step S306: Based on the ratio of activation value distances between the first target neural network layer in each first target training step, obtain the activation value distance ratio of the first target neural network layer in the current training step; fuse the activation value distance ratios of the first target neural network layer in multiple second target training steps to obtain the activation value trajectory divergence information of the first target neural network layer in the current training step; the multiple second target training steps include multiple consecutive training steps, and the current training step is the ending training step.

[0167] Step S308: Determine the activation value distribution information of the first target neural network layer under the current training step; update the historical reference activation value distribution information of the first target neural network layer according to the activation value distribution information of the first target neural network layer under the current training step to obtain the reference activation value distribution information of the first target neural network layer under the current training step; and obtain the activation value distribution offset information of the first target neural network layer under the current training step according to the distribution offset information between the activation value distribution information of the first target neural network layer under the current training step and the reference activation value distribution information.

[0168] Step S310: Perform anomaly detection on the activation value change acceleration, activation value trajectory divergence information, and activation value distribution offset information of the first target neural network layer under the current training step, and obtain the anomaly detection results for the activation value change acceleration, activation value trajectory divergence information, and activation value distribution offset information of the first target neural network layer under the current training step.

[0169] Step S312: In the current training step, the abnormal detection results of the activation value change acceleration of each first target neural network layer, the abnormal detection results of the activation value trajectory divergence information, and the abnormal detection results of the activation value distribution offset information are fused to obtain the current activation value abnormal detection result.

[0170] Step S314: For each training sample in the current batch of training samples, determine the training sample as the current training sample, determine the neighboring training samples of the current training sample from the remaining training samples based on the semantic vector of each training sample, and determine the neighborhood density, neighborhood isolation information, first neighborhood consistency information and second neighborhood consistency information of the current training sample based on the semantic vector of the current training sample and the semantic vector of the neighboring training samples.

[0171] Step S316: Perform anomaly detection on the neighborhood density, neighborhood isolation information, first neighborhood consistency information and second neighborhood consistency information of the current training sample respectively, and obtain the anomaly detection results of the neighborhood density, neighborhood isolation information, first neighborhood consistency information and second neighborhood consistency information of the current training sample.

[0172] Step S318: The anomaly detection results of the neighborhood density, the anomaly detection results of the neighborhood isolation information, the anomaly detection results of the first neighborhood consistency information and the anomaly detection results of the second neighborhood consistency information of the current training sample are fused to obtain the semantic anomaly detection results of the current training sample.

[0173] Step S320: For each training sample in the current batch of training samples, fuse the current activation value anomaly detection result, the current meta-learning anomaly detection result, and the semantic anomaly detection result of the training sample to obtain the sample-level anomaly detection result of the training sample.

[0174] Step S322: Merge the sample-level anomaly detection results of each training sample to obtain the batch-level anomaly detection results of the large language model in the current training step.

[0175] Step S324: Integrate the batch-level anomaly detection results of the large language model under multiple sixth-objective training steps to determine the comprehensive batch-level anomaly detection result of the large language model under the current training step; the multiple sixth-objective training steps are consecutive training steps, and the current training step is the ending training step.

[0176] Step S326: Combine the batch-level anomaly detection results and the comprehensive batch-level anomaly detection results of the large language model in the current training step to obtain the current poisoning detection results of the large language model in the current training step.

[0177] Step S328: If the current poisoning detection result indicates that the large language model has not been poisoned, the large language model is trained for the next training step based on the next batch of training samples. If the current poisoning detection result indicates that the large language model has been poisoned, the training of the large language model is paused.

[0178] Among them, such as Figure 4 As shown, the poisoning detection method for large language models provided in this application further includes the following steps for periodically determining the meta-learning anomaly detection results:

[0179] Step S402: If the third target training step is reached, determine multiple fourth target training steps; the third target training step is the training step in which the number of training steps between the third target training step and the fifth target training step reaches a preset number of training steps; the fifth target training step is the training step in which the large language model was at when the meta-learning anomaly detection result of the large language model was determined last time; the multiple fourth target training steps include multiple consecutive training steps, and the third target training step is the ending training step.

[0180] Step S404: Determine the current gradient information of the large language model based on the gradient of the second target neural network layer in each fourth target training step, and determine the current loss value information of the large language model based on the loss value of the large language model in each fourth target training step.

[0181] Step S406: Combine the current gradient information and the current loss value information to obtain the current meta-learning signature information of the large language model, perform anomaly detection on the current meta-learning signature information, and obtain the current meta-learning anomaly detection result.

[0182] This embodiment has the following beneficial effects:

[0183] 1. Multi-dimensional collaborative detection: The training process is monitored from three complementary dimensions: activation evolution, semantic distribution, and training signature, which can more comprehensively capture the abnormal behavior of different types of poisoning attacks.

[0184] 2. No prior knowledge of attack types is required: It learns the general features of abnormal training patterns through meta-learning methods, and has the ability to detect unseen attack types as well.

[0185] 3. Real-time online detection: Real-time monitoring during training can promptly detect and prevent poisoning attacks, avoiding continuous contamination of the model.

[0186] 4. Low false alarm rate: By combining multi-granularity fusion and trend analysis, short-term anomalies and long-term trends are integrated to effectively reduce the false alarm rate.

[0187] 5. Adaptive adjustment mechanism: The detection threshold is dynamically adjusted based on historical feedback to adapt to different training scenarios and data distributions.

[0188] 6. Tiered response strategy: Take different levels of intervention measures according to the risk level to minimize the impact on training efficiency while ensuring safety.

[0189] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0190] Based on the same inventive concept, this application also provides a poisoning detection device for a large language model to implement the poisoning detection method for the large language model described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the poisoning detection device for a large language model provided below can be found in the limitations of the poisoning detection method for a large language model described above, and will not be repeated here.

[0191] In one embodiment, such as Figure 5 As shown, a poisoning detection device for a large language model is provided, comprising: a training step determination module 502, an activation value anomaly detection module 504, a semantic anomaly detection module 506, and a detection result determination module 508, wherein:

[0192] The training step determination module 502 is used to determine the current training step of the large language model being trained, and obtain the current training step; the large language model includes multiple neural network layers.

[0193] The activation value anomaly detection module 504 is used to determine the current activation value anomaly detection result of the large language model in the current training step based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model in the current training step.

[0194] The semantic anomaly detection module 506 is used to determine the current semantic anomaly detection result of the large language model under the current training step based on the neighborhood distribution information of the current batch of training samples in the semantic space corresponding to the current training step; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information.

[0195] The detection result determination module 508 is used to fuse the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model to obtain the current poisoning detection result of the large language model in the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

[0196] In one embodiment, the number of first target neural network layers is at least one.

[0197] The activation value anomaly detection module 504 is further configured to perform anomaly detection on the activation value temporal evolution information and activation value distribution offset information of each first target neural network layer under the current training step, respectively, to obtain the anomaly detection results of the activation value temporal evolution information and activation value distribution offset information of the first target neural network layer under the current training step; and to fuse the anomaly detection results of the activation value temporal evolution information and activation value distribution offset information of each first target neural network layer under the current training step to obtain the current activation value anomaly detection result.

[0198] In one embodiment, the activation value anomaly detection module 504 is further configured to, for each first target neural network layer, determine the activation value change acceleration and activation value distance of the first target neural network layer in the current training step based on the activation values ​​of the first target neural network layer under multiple first target training steps; the multiple first target training steps include the current training step and the previous training step; obtain the activation value distance ratio of the first target neural network layer in the current training step based on the ratio between the activation value distances of the first target neural network layer under each first target training step; and fuse the activation values ​​of the first target neural network layer under multiple second target training steps. The activation value distance ratio is used to obtain the activation value trajectory divergence information of the first target neural network layer under the current training step; multiple second target training steps include multiple consecutive training steps, with the current training step being the ending training step; based on the activation value change acceleration and activation value trajectory divergence information of the first target neural network layer under the current training step, the temporal evolution information of the activation value of the first target neural network layer under the current training step is obtained; the distribution offset information between the activation value distribution information of the first target neural network layer under the current training step and the reference activation value distribution information is determined to obtain the activation value distribution offset information of the first target neural network layer under the current training step.

[0199] In one embodiment, the activation value anomaly detection module 504 is further configured to update the historical reference activation value distribution information of the first target neural network layer according to the activation value distribution information of the first target neural network layer in the current training step, so as to obtain the reference activation value distribution information of the first target neural network layer in the current training step; the historical reference activation value distribution information is the reference activation value distribution information of the first target neural network layer in the previous training step of the current training step.

[0200] In one embodiment, the neighborhood distribution information of the current batch of training samples includes the neighborhood distribution information of each training sample.

[0201] The semantic anomaly detection module 506 is further configured to, for each current training sample, determine the neighboring training samples from the remaining training samples based on the semantic vectors of each training sample; the semantic vector of each training sample is the differential encoding result between the positive and negative response information of the training sample; the current training sample is any training sample in the current batch of training samples, and the remaining training samples are training samples in the current batch other than the current training sample; the neighboring training samples are training samples whose distance from the current training sample satisfies the distance condition; the neighborhood distribution information of the current training sample is determined based on the current training sample and the neighboring training samples; anomaly detection is performed on the neighborhood distribution information of the current training sample to obtain the semantic anomaly detection result of the current training sample; and the current semantic anomaly detection result is obtained based on the semantic anomaly detection results of each training sample.

[0202] In one embodiment, the semantic anomaly detection module 506 is further configured to determine the neighborhood density, neighborhood isolation information, and deviation information between the current training sample and its neighbors based on the semantic vector of the current training sample and the semantic vector of its neighbors; determine the first neighborhood consistency information and the second neighborhood consistency information of the current training sample based on the deviation information between the current training sample and its neighbors; the first neighborhood consistency information is used to characterize the average value of the deviation information; the second neighborhood consistency information is used to characterize the proportion of the target neighbor training sample in the neighbor training samples, where the target neighbor training sample is the neighbor training sample whose corresponding deviation information satisfies the deviation information condition; and determine the neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information of the current training sample as the neighborhood distribution information of the current training sample.

[0203] In one embodiment, the number of neighborhood distribution information for each training sample is multiple.

[0204] The semantic anomaly detection module 506 is also used to perform anomaly detection on each neighborhood distribution information of each current training sample for each current training sample, and obtain the anomaly detection result of each neighborhood distribution information; and to fuse the anomaly detection results of each neighborhood distribution information to obtain the current semantic anomaly detection result of the current training sample.

[0205] In one embodiment, the poisoning detection device for the large language model further includes a meta-learning anomaly detection module, used to determine multiple fourth-target training steps when a third-target training step is reached; the third-target training step is a training step in which the number of training steps between it and the fifth-target training step reaches a preset number of training steps; the fifth-target training step is the training step in which the large language model was at the time the meta-learning anomaly detection result of the large language model was last determined; the multiple fourth-target training steps include multiple consecutive training steps, with the third-target training step as the ending training step; the current gradient information of the large language model is determined based on the gradient of the second-target neural network layer in the large language model under each fourth-target training step, and the current loss value information of the large language model is determined based on the loss value of the large language model under each fourth-target training step; the current gradient information and the current loss value information are combined to obtain the current meta-learning signature information of the large language model; anomaly detection is performed on the current meta-learning signature information to obtain the current meta-learning anomaly detection result.

[0206] In one embodiment, the current semantic anomaly detection result includes the semantic anomaly detection result for each training sample.

[0207] The module for determining the detection results is also used to, for each training sample, fuse the current activation value anomaly detection result, the current meta-learning anomaly detection result, and the semantic anomaly detection result of the training sample to obtain the sample-level anomaly detection result of the training sample; fuse the sample-level anomaly detection results of each training sample to obtain the batch-level anomaly detection result of the large language model in the current training step; fuse the batch-level anomaly detection results of the large language model in multiple sixth-objective training steps to determine the comprehensive batch-level anomaly detection result of the large language model in the current training step; the multiple sixth-objective training steps are consecutive training steps, with the current training step as the ending training step; fuse the batch-level anomaly detection results of the large language model in the current training step and the comprehensive batch-level anomaly detection result to obtain the current poisoning detection result of the large language model in the current training step.

[0208] In one embodiment, the poisoning detection device for the large language model further includes a model training decision module, which is used to train the large language model for the next training step based on the next batch of training samples if the current poisoning detection result indicates that the large language model has not been poisoned; and to suspend the training of the large language model if the current poisoning detection result indicates that the large language model has been poisoned.

[0209] Each module in the aforementioned large language model poisoning detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0210] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores model parameters for the large language model at each training step and relevant data calculated during each poisoning detection process. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a poisoning detection method for a large language model.

[0211] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0212] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0213] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0214] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0215] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0216] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0217] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting poisoning in a large language model, characterized in that, The method includes: Determine the current training step of the large language model being trained, and obtain the current training step; the large language model includes multiple neural network layers; Based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step, the current activation value anomaly detection result of the large language model under the current training step is determined. Based on the neighborhood distribution information of the current batch of training samples corresponding to the current training step in the semantic space, the current semantic anomaly detection result of the large language model under the current training step is determined; the current batch of training samples includes multiple training samples, and each training sample includes prompt information, positive response information corresponding to the prompt information, and negative response information corresponding to the prompt information. The current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model are fused to obtain the current poisoning detection result of the large language model under the current training step; the current meta-learning anomaly detection result is determined based on the gradient and loss value of the large language model during the training process.

2. The method according to claim 1, characterized in that, The number of layers in the first target neural network is at least one; The step of determining the current activation value anomaly detection result of the large language model in the current training step based on the temporal evolution information and activation value distribution offset information of the first target neural network layer in the large language model under the current training step includes: For each first target neural network layer, anomaly detection is performed on the temporal evolution information of the activation values ​​and the distribution offset information of the activation values ​​of the first target neural network layer under the current training step, respectively, to obtain the anomaly detection results of the temporal evolution information of the activation values ​​and the anomaly detection results of the distribution offset information of the activation values ​​of the first target neural network layer under the current training step; The anomaly detection results of the activation value temporal evolution information of each of the first target neural network layers under the current training step are fused with the anomaly detection results of the activation value temporal evolution information to obtain the current activation value anomaly detection result.

3. The method according to claim 2, characterized in that, The temporal evolution information of activation values ​​and the distribution offset information of activation values ​​for each first target neural network layer under the current training step are obtained in the following way: For each first target neural network layer, the activation value change acceleration and activation value distance of the first target neural network layer in the current training step are determined based on the activation values ​​of the first target neural network layer under multiple first target training steps; the multiple first target training steps include the current training step and the previous training step of the current training step; The ratio of the activation values ​​of the first target neural network layer in the current training step is obtained by comparing the ratios between the activation values ​​of the first target neural network layer in each of the first target training steps. By fusing the activation value distance ratios of the first target neural network layer under multiple second target training steps, the activation value trajectory divergence information of the first target neural network layer under the current training step is obtained; the multiple second target training steps include multiple consecutive training steps, and the current training step is the ending training step; Based on the acceleration of activation value change and the divergence information of activation value trajectory of the first target neural network layer under the current training step, the temporal evolution information of activation value of the first target neural network layer under the current training step is obtained. Determine the distribution offset information between the activation value distribution information and the reference activation value distribution information of the first target neural network layer under the current training step, and obtain the activation value distribution offset information of the first target neural network layer under the current training step.

4. The method according to claim 3, characterized in that, Before determining the distribution offset information between the activation value distribution information and the reference activation value distribution information of the first target neural network layer under the current training step, the method further includes: Based on the activation value distribution information of the first target neural network layer under the current training step, update the historical reference activation value distribution information of the first target neural network layer to obtain the reference activation value distribution information of the first target neural network layer under the current training step; the historical reference activation value distribution information is the reference activation value distribution information of the first target neural network layer under the previous training step under the current training step.

5. The method according to claim 1, characterized in that, The neighborhood distribution information of the current batch of training samples includes the neighborhood distribution information of each training sample; The step of determining the current semantic anomaly detection result of the large language model under the current training step based on the neighborhood distribution information of the current batch of training samples in the semantic space corresponding to the current training step includes: For each current training sample, neighboring training samples are determined from the remaining training samples based on the semantic vectors of each training sample; the semantic vector of each training sample is the differential encoding result between the positive and negative response information of the training sample; the current training sample is any one of the training samples in the current batch, and the remaining training samples are the training samples in the current batch other than the current training sample; the neighboring training samples are training samples whose distance from the current training sample satisfies a distance condition. Based on the current training sample and the neighboring training samples, determine the neighborhood distribution information of the current training sample; Anomaly detection is performed on the neighborhood distribution information of the current training sample to obtain the semantic anomaly detection result of the current training sample; The current semantic anomaly detection result is obtained based on the semantic anomaly detection results of each training sample.

6. The method according to claim 5, characterized in that, The step of determining the neighborhood distribution information of the current training sample based on the current training sample and the neighboring training samples includes: Based on the semantic vector of the current training sample and the semantic vector of the neighboring training samples, determine the neighborhood density, neighborhood isolation information, and deviation information between the current training sample and the neighboring training samples. Based on the deviation information between the current training sample and the neighboring training samples, a first neighborhood consistency information and a second neighborhood consistency information of the current training sample are determined; the first neighborhood consistency information is used to characterize the average value of the deviation information; the second neighborhood consistency information is used to characterize the proportion of the target neighbor training sample in the neighboring training samples, where the target neighbor training sample is the neighbor training sample whose corresponding deviation information satisfies the deviation information condition. The neighborhood density, neighborhood isolation information, first neighborhood consistency information, and second neighborhood consistency information of the current training sample are determined as the neighborhood distribution information of the current training sample.

7. The method according to claim 5, characterized in that, The number of neighborhood distribution information for each training sample is multiple; The step of performing anomaly detection on the neighborhood distribution information of the current training sample to obtain the semantic anomaly detection result of the current training sample includes: For each current training sample, anomaly detection is performed on each neighborhood distribution information of the current training sample to obtain the anomaly detection result of each neighborhood distribution information; The anomaly detection results of the current training sample are obtained by fusing the anomaly detection results of each neighborhood distribution information.

8. The method according to claim 1, characterized in that, The current meta-learning anomaly detection result is obtained in the following way: Upon reaching the third target training step, multiple fourth target training steps are determined; the third target training step is the training step in which the number of training steps between the fifth target training step and the training step reaches a preset number of training steps. The fifth objective training step is the training step in which the large language model was last determined when the meta-learning anomaly detection result of the large language model was determined; the multiple fourth objective training steps include multiple consecutive training steps, and the third objective training step is the ending training step. Based on the gradient of the second target neural network layer in the large language model under each fourth target training step, the current gradient information of the large language model is determined; based on the loss value of the large language model under each fourth target training step, the current loss value information of the large language model is determined. By combining the current gradient information and the current loss value information, the current meta-learning signature information of the large language model is obtained; Anomaly detection is performed on the current meta-learning signature information to obtain the current meta-learning anomaly detection result.

9. The method according to claim 1, characterized in that, The current semantic anomaly detection result includes the semantic anomaly detection result for each of the training samples; The fusion of the current activation value anomaly detection result, the current semantic anomaly detection result, and the current meta-learning anomaly detection result of the large language model to obtain the current poisoning detection result of the large language model in the current training step includes: For each training sample, the current activation value anomaly detection result, the current meta-learning anomaly detection result, and the semantic anomaly detection result of the training sample are fused to obtain the sample-level anomaly detection result of the training sample; By fusing the sample-level anomaly detection results of each training sample, the batch-level anomaly detection results of the large language model under the current training step are obtained; By integrating the batch-level anomaly detection results of the large language model under multiple sixth-objective training steps, the comprehensive batch-level anomaly detection result of the large language model under the current training step is determined; the multiple sixth-objective training steps are consecutive training steps, and the current training step is the ending training step; By integrating the batch-level anomaly detection results and the comprehensive batch-level anomaly detection results of the large language model in the current training step, the current poisoning detection result of the large language model in the current training step is obtained.

10. The method according to any one of claims 1 to 9, characterized in that, After obtaining the current poisoning detection result of the large language model in the current training step, the method further includes: If the current poisoning detection result indicates that the large language model has not been poisoned, the large language model will be trained for the next training step based on the next batch of training samples. If the current poisoning detection results indicate that the large language model has been poisoned, training of the large language model shall be suspended.

Images