Model training method and apparatus, electronic device and storage medium

Abstract

The present application relates to the technical field of neural networks, and discloses a model training method and apparatus, an electronic device and a storage medium. The model training method comprises: according to a determined attack strategy, controlling an attack network model to output an attack vector to an LLM model, wherein the attack vector is used for simulating an attack behavior for the LLM model; controlling a defense network model to be trained on the basis of the attack vector and response data to obtain an updated defense network model; and updating the LLM model on the basis of the updated defense network model.

Description

Model training method and device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on February 26, 2024, with application number 202410208218.X and entitled “Model training method and device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application belongs to the field of neural network technology, and specifically relates to a model training method and device, electronic equipment and storage medium. Background Art

[0004] In related technologies, neural network models, such as Large Language Models (LLMs), are being used in various fields and face numerous security threats. Attackers can manipulate LLMs through carefully crafted attack inputs (i.e., prompts, which refer to a text or set of instructions that guide the language model to perform a specific task). This can cause the LLMs to output inaccurate and biased information, compromising their security. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a model training method and device, electronic device and storage medium, which can improve the security of the LLM model.

[0006] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0007] According to the determined attack strategy, the attack network model is controlled to output the attack vector to the LLM model. The attack vector is used to simulate the attack behavior against the LLM model.

[0008] The control defense network model is trained based on the attack vector and response data to obtain an updated defense network model;

[0009] Update the LLM model according to the updated defense network model.

[0010] In a second aspect, the present application provides a model training device, the training device comprising:

[0011] The attack module is used to control the attack network model to output attack vectors to the LLM model according to the determined attack strategy. The attack vectors are used to simulate attack behaviors against the LLM model.

[0012] The acquisition module is used to obtain the response data of the LLM model to the attack vector;

[0013] The training module is used to control the defense network model to train based on the attack vector and response data to obtain an updated defense network model;

[0014] The updating module is used to update the LLM model according to the updated defense network model.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method of the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0019] In an embodiment of the present application, the attack network model is controlled to output an attack vector to the LLM model according to a determined attack strategy, and the attack vector is used to simulate the attack behavior against the LLM model; the response data of the LLM model to the attack vector is obtained; the defense network model is controlled to be trained based on the attack vector and the response data to obtain an updated defense network model; and the LLM model is updated according to the updated defense network model, so that the updated LLM model can effectively defend against attacks and improve the robustness and security of the LLM model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 shows a flow chart of a method for training a model according to some embodiments of the present application;

[0021] FIG2 is a schematic diagram showing the structure of a training system for an LLM model according to some embodiments of the present application;

[0022] FIG3 shows a structural block diagram of a model training device according to some embodiments of the present application;

[0023] FIG4 shows a structural block diagram of an electronic device according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. Specific embodiments

[0025] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0026] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The following, in conjunction with the accompanying drawings, describes in detail the model training method and device, electronic device and storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0028] In some embodiments of the present application, a model training method is provided. FIG1 shows a flow chart of the model training method in some embodiments of the present application. As shown in FIG1 , the method includes:

[0029] Step 102: Control the attack network model to output an attack vector to the LLM model according to the determined attack strategy. The attack vector is used to simulate attack behaviors against the LLM model.

[0030] In an embodiment of the present application, Figure 2 shows a structural diagram of the training system of the LLM model of some embodiments of the present application. As shown in Figure 2, the training system 200 includes an attack network model 202, a defense network model 204, an LLM model 206 and an evaluation network model 208. The attack network model 202 is used to simulate possible attacks on the LLM model, especially attacks on the LLM model 206, to provide actual attack samples and attack scenarios for optimizing and iterating defense strategies.

[0031] Specifically, the attack network model is controlled to output an attack vector (Prompt) to the LLM model according to a determined attack strategy. For example, the attack vector can be a piece of text or a set of instructions that can guide the language model to perform a specific task. If the attack vector successfully attacks the LLM model, the LLM model will output abnormal output data.

[0032] Step 104: Obtain response data of the LLM model to the attack vector.

[0033] In an embodiment of the present application, the response data of the LLM model to the attack vector specifically includes the output data of the LLM model to the attack vector, and the way in which the LLM model processes the attack vector. Specifically, the attack vector can induce the LLM model to output abnormal data, and in order to avoid such attacks, the LLM model can rely on the defense network model and defense strategy to perform certain processing on the attack vector, thereby avoiding the output of abnormal data. As shown in Figure 2, the defense network model 204 can identify and defend against possible attack vectors through defense strategies. For example, when an input vector is identified as an aggressive attack vector, the LLM model or the defense network model can make a certain degree of adjustment to the input vector, such as padding the input vector that may be an attack vector, that is, adding or overwriting random information in the input vector, or scaling the input vector that may be an attack vector.

[0034] Exemplarily, the defense strategy includes adversarial training samples, role hedging strategies, and the like.

[0035] If the defense is successful, the attack vector that has been processed will not induce the LLM model to output abnormal output data. If the defense fails, the attack vector will induce the LLM model to output abnormal data.

[0036] Therefore, by collecting the response data of the LLM model, it can be determined whether the defense network model and LLM have successfully identified and processed the attack behavior of the attack network model.

[0037] Step 106 : Control the defense network model to be trained based on the attack vector and the response data to obtain an updated defense network model.

[0038] In an embodiment of the present application, the defense network model obtains the attack vector used in each attack behavior and the response data of the LLM model to the attack vector to determine whether the attack behavior simulated by the attack network model is effectively defended or intercepted by the LLM model.

[0039] When an attack is not effectively defended or intercepted, the defense network model is trained and iterated based on the attack vector of the successful attack and the defense strategy of the failed defense, so that the defense network model can "remember" this attack vector and "learn" how to deal with this attack vector, thereby updating the defense model.

[0040] It is understandable that the above process can be continuously automated, so that the defense network model can be continuously iterated and evolved. Since the attack behavior of the attack network model is uninterrupted, the defense network model can continuously evolve and update the defense strategy under the stimulation of simulated attacks. Therefore, the defense strategy can be continuously expanded, and the updated defense strategy can effectively defend against all or most of the attack behaviors.

[0041] By using attack network models and defense network models to simulate possible real attack behaviors, there is no need for manual operation to simulate attack and defense behaviors, thus achieving more intelligent adversarial training.

[0042] Step 108: Update the LLM model according to the updated defense network model.

[0043] In an embodiment of the present application, the updated defense network model can effectively defend against the attack behavior of the attack network model. Therefore, the updated defense network model can provide a newer and more effective defense strategy. Through the defense strategy provided by the updated defense network model, the LLM model body is updated, so that the LLM model has the ability to defend against different known or unknown attack behaviors, thereby reducing the possibility of the LLM model being successfully attacked.

[0044] It is understandable that the above training process can be performed once every certain period of time, so that the defense network model and the LLM model can continuously adapt to new attack methods, thereby realizing an automated training process.

[0045] The embodiment of the present application controls the attack network model to output an attack vector to the LLM model according to a determined attack strategy, and the attack vector is used to simulate the attack behavior against the LLM model; obtains the response data of the LLM model to the attack vector; controls the defense network model to be trained based on the attack vector and the response data to obtain an updated defense network model; and updates the LLM model according to the updated defense network model, so that the updated LLM model can effectively defend against attacks and improve the robustness and security of the LLM model.

[0046] In some embodiments of the present application, the training method further includes:

[0047] At least one of the attack network model, the updated defense network model, and the updated LLM model is updated by evaluating the network model.

[0048] In an embodiment of the present application, as shown in FIG2 , the training system 200 further includes a judgment network model 208 , which can collect the output data of the LLM model under each simulated attack of the attack network model, determine whether the attack is a successful attack based on the output data, and form statistical data.

[0049] For example, the statistical data records the vector features of the attack vector output in each attack behavior of the attack network model, the response mode of the LLM model when receiving the attack vector, and whether the attack is successful.

[0050] After collecting statistical data including each attack behavior of the attack network model, the attack network model, the updated defense network model and / or the LLM model are further updated using these statistical data.

[0051] Specifically, the evaluation network model can determine whether an attack is a successful attack. If an attack is a successful attack, the defense network model and the LLM model are mainly updated so that the defense network model and the LLM model can be immune to known successful attacks.

[0052] If an attack is unsuccessful, for example, the defense network model or LLM model successfully identifies and defends against the attack, the attack vector in this attack will be marked as an invalid attack vector, and the attack network model, especially the attack strategy, will be evolved and expanded, thereby improving the attack diversity of the attack network model and enabling the attack network model to simulate more complex attack behaviors.

[0053] By conducting adversarial training on the attack network model, defense network model, and LLM model, the attack network model, defense network model, and LLM model can be evolved synchronously. The better the attack performance of the attack network model, the better the defense capabilities of the defense network model and LLM model after simulated attack training will be. Ultimately, the updated defense network model and updated LLM model can effectively defend against various known or unknown attack behaviors.

[0054] The embodiment of the present application sets up a judgment network model to objectively evaluate the attacker and defender, thereby guiding the training of the attacker, defender, and LLM model, that is, the attack network model, the defense network model, and the LLM model are subjected to adversarial training, which can improve the training effect, make the updated LLM model more immune to attacks, and improve the reliability, security, and robustness of the LLM model.

[0055] In some embodiments of the present application, updating at least one of the attack network model, the updated defense network model, and the updated LLM model by evaluating the network model includes:

[0056] At least one of the attack network model, the updated defense network model, and the updated LLM model is updated based on the attack vector and the response data by evaluating the network model.

[0057] In the embodiment of the present application, the evaluation network model records the attack vector used in each simulated attack behavior and records the response data corresponding to the attack vector, thereby determining whether a simulated attack behavior is a successful attack behavior.

[0058] The evaluation network model finds attack vectors of successful attacks based on the response data. By analyzing the characteristics of these attack vectors and the vulnerabilities of the LLM model or the defense strategy targeted by these attack vectors, the network parameters of the LLM model or the defense strategy of the defense network model are adjusted. Continuous training and iteration are carried out to ensure that the updated LLM model can effectively identify, intercept and defend against the same attack vector the next time it encounters it.

[0059] At the same time, the evaluation network model identifies unsuccessful attack vectors based on the response data and marks them as invalid. It also adjusts the attack strategy of the attack network model, such as removing invalid attack vectors from the strategy and randomly generating new attack vectors or obtaining them from a pre-set attack vector library. The attack strategy and attack network model are then updated simultaneously, allowing the attack network model to continuously expand its attack methods and approaches, and to identify attack vectors that are likely to succeed. By training the attack network model, the defense network model, and the LLM model in an adversarial manner, both training intensity and effectiveness can be improved.

[0060] Before training, the LLM model is initialized, including setting operating parameters for the large language model network LLM, including memory allocation, setting processor time and other model-specific configuration options.

[0061] For evaluating network models, define the criteria for determining the success of an attack and the success of a defense strategy, including but not limited to attack organization rate, defense success rate, and false alarm rate. Evaluation metrics also include indicators used to measure the effectiveness of attack and defense strategies, such as recognition rate, response time, and resource consumption.

[0062] The evaluation network model determines the success of various attack and defense mechanisms, with the sole goal of accurate judgment and providing reliable evaluation results for the system. Implementation methods for the evaluation network model include, but are not limited to, a general LLM semantic analysis and evaluation mechanism, a multi-model decision-making mechanism, and a dedicated LLM semantic analysis and evaluation mechanism. This multi-dimensional evaluation system provides accurate evaluation results.

[0063] When initializing the attack network model, the attack network model loads a pre-set attack sample library and sets the rules and parameters for each attack instance. For example, attack strategies are selected in order of complexity from simple to complex to cover different possible attack vectors and attack scenarios.

[0064] The goal of initializing the network model is to simulate actual attack scenarios, launching various prompt-related attacks with the sole goal of successful attacks. This provides realistic attack samples and scenarios for defending the network model. Attack methods include, but are not limited to, simple role-playing attacks, DAN role-playing attacks, and automated attacks based on greedy prompts. The security of the LLM is tested through various attack strategies.

[0065] When the defense network model is initialized, it is loaded with an initial defense strategy. For example, this strategy includes adversarial training samples and role hedging strategies. A dynamic adjustment mechanism for the defense strategy is established to enable real-time adjustments and optimizations based on real-time feedback from the evaluation network model during adversarial training.

[0066] The goal of the defense network model is to protect LLMs from attacks by designing and implementing various mechanisms, with the sole objective of successful defense. The defense strategy is optimized through real-time adversarial testing. The defense network model's defense methods include, but are not limited to, actor hedging schemes, multi-actor decision-making schemes, synthetic samples, adversarial training, and Greedy Prompt automated attack countermeasures. The defense strategy is adjusted in real time to respond to different attack modes.

[0067] After initializing each network model, configure a reasonable adversarial testing environment. Set up an automated process so that the attack network model and the defense network model can run automatically without human intervention. Also set the adversarial training cycle, test frequency, and duration of each test round.

[0068] Build a monitoring and logging system, set up real-time monitoring tools to track the performance and security status of the LLM model system, and define logging rules to ensure that all important events are recorded and available for subsequent analysis.

[0069] In some implementations, a visual user interface can also be created to help testers monitor the training and testing status in real time, adjust parameters, view reports, and perform other operations.

[0070] The embodiments of the present application simulate possible attacks against the LLM model to automatically test and conduct adversarial training on the defense strategy of the LLM model, so that the defense strategy of the LLM model can effectively target known or unknown possible attacks, thereby improving the robustness and security of the LLM model.

[0071] In some embodiments of the present application, before controlling the attack network model to output the attack vector to the LLM model according to the determined attack strategy, the method further includes:

[0072] Obtaining an initial attack vector set, where the initial attack vector set includes multiple initial attack vectors;

[0073] Get the fitness value corresponding to the initial attack vector;

[0074] Add the target initial attack vector whose fitness value is greater than the threshold to the target attack vector set;

[0075] The attack vector is a vector in the target attack vector set.

[0076] In an embodiment of the present application, before conducting adversarial training, an initial attack vector set is established. The initial attack vector set needs to include a variety of different attack vectors to simulate various possible attack behaviors. When the attack network model performs a simulated attack, it selects one or more attack vectors from the initial attack vector set in a certain order to attack the LLM model.

[0077] Specifically, an initial attack vector set is obtained. Multiple initial vectors included in the initial attack vector set may serve as input vectors for the LLM model. A fitness function is defined to determine the fitness value of each initial vector. The fitness value can reflect the effectiveness of an attack on the LLM model using an initial vector.

[0078] The target initial attack vectors with fitness greater than the threshold are attack vectors that may be able to effectively attack the LLM model. These target initial attack vectors are added to the target attack vector set, and the attack strategy is determined by the target attack vector set, so as to control the attack network model to simulate the attack behavior against the LLM model according to the attack strategy.

[0079] A vector with a fitness value greater than a threshold is selected as the first attack vector in the attack vector set. This first attack vector is a known attack vector. To increase the number and diversity of attack vectors in the attack vector set and enhance the simulation of unknown attack vectors, after obtaining the first attack vector, vector multiplication is performed on the first attack vector based on its vector characteristics to obtain a second attack vector that shares certain characteristics with the first attack vector.

[0080] For example, the first attack vector is used as the parent attack vector and reproduced to obtain child attack vectors. Then, through roulette wheel selection, tournament selection, or other selection methods, a child attack vector with a higher attack success rate, specifically reflected in a fitness value greater than a threshold, is selected from the child attack vectors, i.e., the second attack vector.

[0081] Exemplarily, the feature points between different vectors can be cross-recombined between the first attack vector and the second attack vector, or between multiple second attack vectors, such as by combining the features between different vectors in a single-point crossover, multi-point crossover, or uniform crossover manner to generate a new offspring vector, that is, a new second vector.

[0082] Exemplarily, the offspring vector, that is, the vector feature of the second attack vector, may be randomly mutated to introduce new mutation points, wherein the random mutation may be in the form of adding new words, replacing original words, or deleting part of the original words.

[0083] By randomly mutating a first attack vector that can successfully launch an attack, multiple second attack vectors are obtained. The second attack vector is an unknown attack vector. The attack vector set formed by the first attack vector and the second attack vector can cover the attack methods and locations of known attacks, thereby comprehensively testing and iteratively training the defense network model and defense strategy. The defense network model and defense strategy trained by this method can not only cope with known attack methods, but also predict and target unknown attack methods.

[0084] The evaluation network monitors the LLM's response to each attack sample in real time to determine which attacks are effective. For any detected effective attacks, the network analyzes the LLM's weaknesses and the characteristics of the attack sample. By collecting data on the LLM's response to all attack samples and performing statistical analysis, the network identifies successful attack cases and conducts in-depth analysis to enable the defense network model to understand and design effective defense strategies.

[0085] As the training iterative process progresses, the attack network model continuously receives statistical data sent by the judgment network model and finds successful attack cases in the statistical data, namely, the third attack vector that successfully passes through the defense network model and attacks the LLM model.

[0086] The third attack vectors from these successful cases are multiplied in the same manner as the second attack vector was generated from the first attack vector, using methods such as crossover recombination and random mutation. This yields a child vector of the third attack vector, the fourth attack vector mentioned above. The attack vector set is then expanded using the third and fourth attack vectors. Valid attack samples and strategies are stored in a sample library for future attack simulations and defense strategy testing. The attack sample library is regularly updated to include emerging attack patterns and strategies.

[0087] In other words, during adversarial training, the attack capability of the attack network model gradually increases, as does the variety of attack vectors. During this process, the attack network model continuously explores more possible unknown attack methods, enabling the LLM model to adapt to these potential unknown attack methods in advance. By leveraging the LLM's response and feedback from the evaluation network model, attack strategies are optimized, increasing the sophistication and complexity of attacks.

[0088] The attack network model continuously expands the set of attack vectors based on the data collected in the adversarial test, and tests the defense network model and defense strategy through the continuously expanding set of attack vectors. At the same time, the defense network model establishes a dynamic adjustment mechanism for the defense strategy. As the attack vectors continue to expand, the defense strategy will also continue to evolve to adapt to various possible attack methods.

[0089] In the embodiment of the present application, known attack vectors are mutated and expanded to obtain an attack vector set that includes both known and unknown attack methods. The attack module tests and conducts adversarial training on the defense module and defense strategy through the attack vector set, so that the trained LLM model has the ability to resist known and unknown attack methods, thereby improving the robustness of the defense strategy.

[0090] In some embodiments of the present application, obtaining a fitness value corresponding to an initial attack vector includes:

[0091] The fitness value corresponding to the initial attack vector is obtained according to the manipulation ability score value, the concealment score value, and the diversity score value of the initial attack vector.

[0092] In an embodiment of the present application, the fitness value can be determined based on the manipulation ability score (Manipulation Score), the stealthiness score (Stealthiness Score) and the diversity score (Diversity Score). Among them, the manipulation ability score is to calculate the consistency between the output of the LLM after a given attack prompt and the expected target of the attack module. This can be achieved through natural language processing technology, such as semantic similarity calculation. The stealthiness score is used to evaluate the stealthiness of the attack prompt. A good attack prompt should not be obvious, it should be able to manipulate the LLM without arousing suspicion. The diversity score is used to evaluate the diversity between attack samples to avoid overfitting of the defense system due to overly similar attacks.

[0093] For example, the fitness value of an initial vector is calculated by the following formula (1): Fitness = ω1×manipulation ability score + ω2×(1-concealment score) + ω3×diversity score; (1)

[0094] Among them, ω1, ω2, and ω3 are weight factors, which can be adjusted according to the specific requirements of the attack scenario.

[0095] By calculating the fitness value of each initial vector, an initial vector that can be used as an attack vector is found in the initial vector set, and an attack vector set is formed. The attack vector set includes multiple attack prompts that may effectively attack the LLM model.

[0096] The embodiment of the present application can automatically generate an attack vector set, simulate possible attack scenarios through the attack vectors in the attack vector set, and thus test whether the defense strategy can effectively defend against the attack, realize adversarial training of the defense module and the attack module, and improve the robustness and reliability of the defense strategy of the LLM model.

[0097] In some embodiments of the present application, optionally, the training method includes:

[0098] Determine a third attack vector based on statistical data, where the third attack vector is a vector that effectively attacks the LLM model;

[0099] The vector feature of the third attack vector and the processing strategy corresponding to the third attack vector are added to the defense strategy. The processing strategy is used to indicate the processing method of the LLM model to perform defense processing on the third attack vector.

[0100] In an embodiment of the present application, during the testing and adversarial training of the LLM model, defense network model, and defense strategy, the evaluation network records each attack vector and its corresponding attack result and defense result, and forms statistical data.

[0101] As the training iterates, the defense network model continuously receives statistical data sent by the evaluation network model and finds successful attack cases in the statistical data, namely the third attack vector that successfully passed the defense network model and attacked the LLM model.

[0102] The defense network model generates a targeted processing strategy based on the vulnerabilities or weaknesses of the LLM model targeted by the third attack vector. Based on this processing strategy, the third attack vector can be effectively defended, that is, harmlessly processed. The input vector obtained after processing the third attack vector through the corresponding processing strategy will not cause the LLM model to produce abnormal output.

[0103] After obtaining the corresponding processing strategy for each third attack vector that successfully attacks the LLM model, the vector features of the third attack vector and the corresponding processing strategy are associated and added to the defense strategy. When the LLM model, or the defense network model therein, detects the input third attack vector again, the updated defense strategy can be used to find a processing strategy that effectively defends against the third attack vector, preventing the third attack vector from effectively attacking the LLM model.

[0104] Specifically, the defense network model receives attack samples and data on their impact on the LLM from the attack network model. It analyzes the attack vectors generated by the attack network model and identifies specific patterns that cause LLM output failures or anomalies. Based on feedback from preliminary testing, it adjusts the defense strategy to improve accuracy and reduce false positives.

[0105] Implement dynamic defense mechanisms, such as updating defense strategies in real time based on changes in attack patterns. Adopt adaptive algorithms that allow defense network models to quickly learn and adjust defense measures when encountering new types of attacks.

[0106] During testing, the defense network model designed a multi-role decision-making scheme. The system simulated different users and usage scenarios to evaluate the effectiveness of defense strategies in diverse environments. Through continuous simulation and testing, the optimal strategy combination was found to adapt to changing attack scenarios.

[0107] LLMs are leveraged to generate synthetic attack samples to test and improve the effectiveness of defense strategies. These synthetic samples can simulate unknown or unseen attacks, helping defense network models prepare countermeasures. By exposing the LLM to a variety of attack samples, it learns how to identify and defend against them through adversarial training. Through iterative training, the LLM's defense mechanisms are continuously improved, enabling it to identify increasingly complex attack patterns.

[0108] Deploy tested and validated defense strategies to LLMs in the production environment. Set up monitoring points for defense strategies to ensure they work properly in the production environment and adjust them as needed.

[0109] The embodiment of the present application records the attack vectors of successful attacks, generates targeted processing strategies based on successful attack cases, and updates the defense strategy based on the characteristics of the attack vectors in the successful attack cases and the effective processing strategies, so that the defense strategy can effectively defend against multiple possible attack methods and improve the security of the LLM model.

[0110] In some embodiments of the present application, adjusting at least one of the network parameters and defense strategies of the LLM model based on statistical data includes:

[0111] Determine a third attack vector based on statistical data, where the third attack vector is a vector that effectively attacks the LLM model;

[0112] Inputting the third attack vector into the LLM model to obtain first output data output by the LLM model;

[0113] When the first output data is abnormal output data, the network parameters of the LLM model are adjusted to obtain a trained LLM model. The trained LLM model outputs second output data for the third attack vector, and the second output data is normal output data.

[0114] In an embodiment of the present application, during the testing and adversarial training of the defense network model and defense strategy of the LLM model, the evaluation network records each attack vector and its corresponding attack result and defense result, and forms statistical data.

[0115] As the training iterates, the defense network model continuously receives statistical data sent by the evaluation network model and finds successful attack cases in the statistical data, namely the third attack vector that successfully passes through the defense network model and attacks the LLM model.

[0116] After accumulating a sufficient number of third attack vectors, a new training dataset is generated based on these third attack vectors and used to retrain the LLM model. During training, the network parameters of the LLM model are continuously adjusted. The model output data for each third attack vector after the adjusted parameters is evaluated to determine whether this data is anomalous.

[0117] If the LLM model outputs abnormal first output data in response to the third attack vector, the network data of the LLM model is adjusted to change the output of the LLM model. If the LLM model outputs normal second output data in response to the third attack vector, it is determined that the LLM model no longer has vulnerabilities or weaknesses that can be exploited by the third attack vector. At this point, the trained LLM model will no longer be vulnerable to the third attack vector, and is therefore immune to the third attack vector.

[0118] For example, during adversarial testing, the attack network model generates new attack prompts based on the current attack strategy. The generated attack prompts are applied to the LLM under test, and the model's response is observed. Model response data is collected, including the model's output and how the model handles the prompts.

[0119] The evaluation network model analyzes the response of the LLM to be tested to determine whether the attack is successful. If the attack is successful, the successful attack pattern is recorded and fed back to the attack network model and the defense network model.

[0120] After receiving a successful attack pattern, the defense network model analyzes the attack characteristics and the LLM's vulnerabilities. It then designs or updates defense strategies, such as adjusting filtering rules, enhancing adversarial training samples, or optimizing self-protection mechanisms. The updated defense strategies are then deployed to the LLM under test to prepare for the next round of attacks.

[0121] The attack network model adjusts or innovates its attack strategy based on updates to the defense network model to find new attack paths. The defense network model simultaneously monitors the effectiveness of the defense and prepares to update the defense strategy again based on changes in the attack network model.

[0122] This iterative process of attack and defense is repeated multiple times, with each iteration refining the attack pattern and defense mechanism. The evaluation network model continuously monitors the results of each attack and defense round to ensure the objectivity and accuracy of the test.

[0123] The evaluation network model regularly assesses the effectiveness of defense strategies, including reducing the frequency of successful attacks and improving the quality of LLM responses. Based on the evaluation results, it determines whether further iterations or fundamental adjustments to the strategy are necessary. As adversarial testing deepens, the system accumulates effective attack and defense knowledge, forming a library of attack and defense strategies. Through data analysis and machine learning methods, it continuously optimizes the identification of attack patterns and the implementation of defense strategies.

[0124] The embodiments of this application improve the security of the LLM model through an anti-evolution mechanism, enabling it to effectively identify and resist a variety of attacks. This mechanism not only responds to existing attack strategies, but also prevents unknown or unseen attacks, enhancing the robustness and reliability of the LLM model.

[0125] The training method of the model provided in the embodiment of the present application can be executed by a training device for the model. In the embodiment of the present application, the training method of the model is executed by the training device for the model as an example to illustrate the training device for the model provided in the embodiment of the present application.

[0126] In some embodiments of the present application, a model training device is provided. FIG3 shows a structural block diagram of the model training device in some embodiments of the present application. As shown in FIG3 , the training device 300 includes:

[0127] Attack module 302, configured to control the attack network model to output an attack vector to the LLM model according to the determined attack strategy, wherein the attack vector is used to simulate attack behavior against the LLM model;

[0128] An acquisition module 304 is used to obtain response data of the LLM model to the attack vector;

[0129] A training module 306 is used to control the defense network to train based on the attack vector and response data to obtain an updated defense network model;

[0130] The updating module 308 is configured to update the LLM model according to the updated defense network model.

[0131] The embodiment of the present application controls the attack network model to output an attack vector to the LLM model according to a determined attack strategy, and the attack vector is used to simulate the attack behavior against the LLM model; obtains the response data of the LLM model to the attack vector; controls the defense network model to be trained based on the attack vector and the response data to obtain an updated defense network model; and updates the LLM model according to the updated defense network model, so that the updated LLM model can effectively defend against attacks and improve the robustness and security of the LLM model.

[0132] In some embodiments of the present application, the update module is further specifically used to update at least one of the attack network model, the updated defense network model, and the updated LLM model by evaluating the network model.

[0133] The embodiment of the present application sets up a judgment network model to objectively evaluate the attacker and defender, thereby guiding the training of the attacker, defender, and LLM model, that is, the attack network model, the defense network model, and the LLM model are subjected to adversarial training, which can improve the training effect, make the updated LLM model more immune to attacks, and improve the reliability, security, and robustness of the LLM model.

[0134] In some embodiments of the present application, the update module is further used to update at least one of the attack network model, the updated defense network model, and the updated LLM model based on the attack vector and the response data by evaluating the network model.

[0135] The embodiments of the present application simulate possible attacks against the LLM model to automatically test and conduct adversarial training on the defense strategy of the LLM model, so that the defense strategy of the LLM model can effectively target known or unknown possible attacks, thereby improving the robustness and security of the LLM model.

[0136] In some embodiments of the present application, the acquisition module is further configured to acquire an initial attack vector set, the initial attack vector set including multiple initial attack vectors; and acquire fitness values ​​corresponding to the initial attack vectors;

[0137] The training device also includes:

[0138] An adding module, configured to add a target initial attack vector having a fitness value greater than a threshold to a target attack vector set;

[0139] The attack vector is a vector in the target attack vector set.

[0140] In the embodiment of the present application, known attack vectors are mutated and expanded to obtain an attack vector set that includes both known and unknown attack methods. The attack module tests and conducts adversarial training on the defense module and defense strategy through the attack vector set, so that the trained LLM model has the ability to resist known and unknown attack methods, thereby improving the robustness of the defense strategy.

[0141] In some embodiments of the present application, the acquisition module is further specifically configured to acquire a fitness value corresponding to the initial attack vector according to the manipulation capability score, concealment score, and diversity score of the initial attack vector.

[0142] The embodiment of the present application can automatically generate an attack vector set, simulate possible attack scenarios through the attack vectors in the attack vector set, and thus test whether the defense strategy can effectively defend against the attack, realize adversarial training of the defense module and the attack module, and improve the robustness and reliability of the defense strategy of the LLM model.

[0143] The training device of the model in the embodiments of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiments of the present application do not specifically limit it.

[0144] The training device of the model in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0145] The model training device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be described here.

[0146] Optionally, an embodiment of the present application also provides an electronic device. Figure 4 shows a structural block diagram of an electronic device according to an embodiment of the present application. As shown in Figure 4, the electronic device 400 includes a processor 402, a memory 404, and a program or instruction stored in the memory 404 and executable on the processor 402. When the program or instruction is executed by the processor 402, each process of the above-mentioned method embodiment is implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0147] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0148] FIG5 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0149] The electronic device 500 includes but is not limited to components such as a radio frequency unit 501 , a network module 502 , an audio output unit 503 , an input unit 504 , a sensor 505 , a display unit 506 , a user input unit 507 , an interface unit 508 , a memory 509 , and a processor 510 .

[0150] Those skilled in the art will appreciate that the electronic device 500 may further include a power source (e.g., a battery) to power various components. The power source may be logically connected to the processor 510 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG5 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or arrange the components differently, which will not be described in detail here.

[0151] Among them, the processor 510 is used to control the attack network model to output an attack vector to the LLM model according to the determined attack strategy, and the attack vector is used to simulate the attack behavior against the LLM model; obtain the response data of the LLM model to the attack vector; control the defense network model to train based on the attack vector and response data to obtain an updated defense network model; and update the LLM model according to the updated defense network model.

[0152] The embodiment of the present application controls the attack network model to output an attack vector to the LLM model according to a determined attack strategy, and the attack vector is used to simulate the attack behavior against the LLM model; obtains the response data of the LLM model to the attack vector; controls the defense network model to be trained based on the attack vector and the response data to obtain an updated defense network model; and updates the LLM model according to the updated defense network model, so that the updated LLM model can effectively defend against attacks and improve the robustness and security of the LLM model.

[0153] Optionally, the processor 510 is further configured to update at least one of the attack network model, the updated defense network model, and the updated LLM model by evaluating the network model.

[0154] The embodiment of the present application sets up a judgment network model to objectively evaluate the attacker and defender, thereby guiding the training of the attacker, defender, and LLM model, that is, the attack network model, the defense network model, and the LLM model are subjected to adversarial training, which can improve the training effect, make the updated LLM model more immune to attacks, and improve the reliability, security, and robustness of the LLM model.

[0155] Optionally, the processor 510 is further configured to update at least one of the attack network model, the updated defense network model, and the updated LLM model based on the attack vector and the response data by evaluating the network model.

[0156] The embodiments of the present application simulate possible attacks against the LLM model to automatically test and conduct adversarial training on the defense strategy of the LLM model, so that the defense strategy of the LLM model can effectively target known or unknown possible attacks, thereby improving the robustness and security of the LLM model.

[0157] Optionally, the processor 510 is also used to obtain an initial attack vector set, which includes multiple initial attack vectors; obtain a fitness value corresponding to the initial attack vector; add a target initial attack vector whose fitness value is greater than a threshold to the target attack vector set; wherein the attack vector is a vector in the target attack vector set.

[0158] In the embodiment of the present application, known attack vectors are mutated and expanded to obtain an attack vector set that includes both known and unknown attack methods. The attack module tests and conducts adversarial training on the defense module and defense strategy through the attack vector set, so that the trained LLM model has the ability to resist known and unknown attack methods, thereby improving the robustness of the defense strategy.

[0159] Optionally, the processor 510 is further configured to obtain a fitness value corresponding to the initial attack vector according to the manipulation capability score value, the concealment score value, and the diversity score value of the initial attack vector.

[0160] The embodiment of the present application can automatically generate an attack vector set, simulate possible attack scenarios through the attack vectors in the attack vector set, and thus test whether the defense strategy can effectively defend against the attack, realize adversarial training of the defense module and the attack module, and improve the robustness and reliability of the defense strategy of the LLM model.

[0161] It should be understood that in an embodiment of the present application, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and at least one of other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0162] The memory 509 can be used to store software programs and various data. The memory 509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 509 may include a volatile memory or a non-volatile memory, or the memory 509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0163] Processor 510 may include one or more processing units. Optionally, processor 510 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 510.

[0164] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0165] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0166] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0167] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0168] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0169] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0171] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A model training method, the method comprising: Controlling the attack network model to output an attack vector to the LLM model according to the determined attack strategy, wherein the attack vector is used to simulate an attack behavior against the LLM model; Obtaining response data of the LLM model to the attack vector; Controlling the defense network model to be trained based on the attack vector and the response data to obtain an updated defense network model; The LLM model is updated according to the updated defense network model.

2. The training method according to claim 1, wherein: The method further comprises: At least one of the attack network model, the updated defense network model, and the updated LLM model is updated by evaluating the network model.

3. The training method according to claim 2, wherein: The updating of at least one of the attack network model, the updated defense network model, and the updated LLM model by evaluating the network model includes: At least one of the attack network model, the updated defense network model, and the updated LLM model is updated based on the attack vector and the response data by evaluating the network model.

4. The training method according to claim 1, wherein: Before controlling the attack network model to output the attack vector to the LLM model according to the determined attack strategy, the method further includes: Acquire an initial attack vector set, where the initial attack vector set includes multiple initial attack vectors; Obtaining a fitness value corresponding to the initial attack vector; Add the target initial attack vector whose fitness value is greater than the threshold to the target attack vector set; The attack vector is a vector in the target attack vector set.

5. The training method according to claim 4, wherein: The obtaining of the fitness value corresponding to the initial attack vector includes: A fitness value corresponding to the initial attack vector is obtained according to the manipulation ability score value, the concealment score value, and the diversity score value of the initial attack vector.

6. A model training device, comprising: An attack module, configured to control the attack network model to output an attack vector to the LLM model according to a determined attack strategy, wherein the attack vector is used to simulate an attack behavior against the LLM model; An acquisition module, configured to acquire response data of the LLM model to the attack vector; A training module, configured to control the defense network model to be trained based on the attack vector and the response data to obtain an updated defense network model; An updating module is used to update the LLM model according to the updated defense network model.

7. The training device according to claim 6, wherein: The updating module is further configured to update at least one of the attack network model, the updated defense network model, and the updated LLM model by evaluating the network model.

8. The training device according to claim 7, wherein: The updating module is further specifically configured to update at least one of the attack network model, the updated defense network model, and the updated LLM model based on the attack vector and the response data by evaluating the network model.

9. The training device according to claim 6, wherein: The acquisition module is further configured to acquire an initial attack vector set, wherein the initial attack vector set includes a plurality of initial attack vectors; as well as Obtaining a fitness value corresponding to the initial attack vector; The training device further comprises: An adding module, configured to add a target initial attack vector having a fitness value greater than a threshold to a target attack vector set; The attack vector is a vector in the target attack vector set.

10. The training device according to claim 9, wherein: The acquisition module is further specifically configured to acquire a fitness value corresponding to the initial attack vector according to the manipulation capability score value, concealment score value, and diversity score value of the initial attack vector.

11. An electronic device comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium storing a program or instruction, wherein the program or instruction is executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

13. A chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps of the method according to any one of claims 1 to 5.

14. A computer program product, wherein the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 5.

Images