Large model alignment method and device based on knowledge forgetting

Abstract

The invention provides a large model alignment method and device based on knowledge forgetting, and relates to the technical field of natural language processing. The method comprises the following steps: obtaining each parameter weight in a large language model, calculating an importance score for each neuron, sorting the neuron, and constructing a binary knowledge forgetting mask; determining a knowledge forgetting layer and a forgetting module according to a sorting result; obtaining a harmful instruction and a harmful response, and constructing a harmful knowledge data set; and setting an optimization target, and performing harmful knowledge forgetting training on the knowledge forgetting layer and the forgetting module according to the binary knowledge forgetting mask, the harmful knowledge data set, the optimization target and a limited gradient ascending algorithm to obtain a trained large language model. According to the CKU method provided by the invention, by adopting a constraint optimization technology, a security alignment task is converted into a limited knowledge forgetting task, unnecessary or harmful knowledge in the generative large language model is accurately removed, and meanwhile, the overall performance and efficiency of the model are ensured to be kept.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a large model alignment method and device based on knowledge forgetting. Background Art

[0002] Since the great success of ChatGPT (Chat Generative Pre-trained Transformer), generative LLMs (Large Language Models) have been widely applied in various aspects of people's daily lives. From personal assistants, intelligent customer service to content creation, automatic translation, large language models have shown great potential in all walks of life. Their powerful language understanding and generation capabilities enable these models to not only perform simple tasks such as question answering, sentiment analysis, and language translation, but also complete more complex tasks such as writing code, generating creative content, and solving multidisciplinary problems. At the same time, the application of LLM (Large Language Model) in fields such as medicine, education, and finance is gradually deepening, and the potential to assist decision-making and improve service quality has also been fully exploited. More importantly, the emergence of generative language models has greatly promoted the efficiency of information exchange and knowledge dissemination. By deeply learning a large amount of text data, LLM can understand and generate natural language text, simulating human language processing capabilities, thus providing unprecedented innovative solutions in many traditional tasks. However, despite the many conveniences brought by LLMs, their security and ethical issues remain the focus of attention in society and academia. With the continuous development of these technologies, how to ensure that they provide powerful functions while avoiding being misused or causing adverse consequences remains a key direction for future research.

[0003] Adopting diverse alignment strategies to optimize the output of generative large language models can significantly enhance the security and reliability of the models in practical applications, while effectively reducing outputs that do not conform to human values or ethical standards. Through this alignment mechanism, the model can not only better understand and follow user intentions, but also maintain consistency with human expectations when generating content, avoiding biases, misleading, or inappropriate behaviors. This alignment is not only reflected in the accuracy and compliance of the content, but also to a certain extent avoids the model from producing out-of-control or harmful outputs in specific situations, ensuring that it meets the overall moral and legal requirements of society. The application of alignment technology enhances the adaptability of the model in different situations, enabling it to provide outputs that are more in line with ethical and legal norms in various complex tasks, thereby increasing the social value and sustainability of the model. Especially in sensitive fields or high-risk applications, such as medical diagnosis, legal consultation, etc., ensuring that the behavior of the model is consistent with human ethical values can significantly improve the controllability and trust of the model, promoting the wide application and popularization of generative large language models in natural language processing and generation. The continuous development and improvement of this technology will enable future generative language models to not only possess powerful technical capabilities, but also have a higher sense of social responsibility and ethical level.

[0004] Large language models have demonstrated powerful generation capabilities in many natural language processing tasks, being able to generate fluent and somewhat creative text. However, with the diversification of the application scenarios of these models, the issues of controllability and consistency of their generated results have gradually emerged. Since the generation process of large language models is based on a large amount of pre-trained data and complex statistical relationships, the generated content often lacks a clear goal orientation and sometimes even deviates from user expectations. Therefore, how to make the output of the model more in line with human needs and expectations has become an important research topic in the field of natural language processing. To achieve this goal, more and more researchers have started to explore various technical paths, including methods such as introducing reinforcement learning and user feedback mechanisms. Reinforcement learning optimizes the generation strategy through interaction with the environment, enabling the model to gradually adjust the output according to the preset goal, improving the quality and relevance of the results. The user feedback mechanism adjusts the behavior of the model by collecting real-time user evaluations of the generated content, making it more in line with user needs.

[0005] RLHF (Reinforcement Learning with Human Feedback) is a technical framework that combines reinforcement learning with human feedback, aiming to improve the model's decision-making ability and generation quality. The core technologies of RLHF include: (1) guiding the model's behavior through human feedback to ensure that the model output meets the actual requirements; (2) using reinforcement learning algorithms to further optimize the model's strategy with reward signals; (3) training the model on large-scale human feedback data to enable it to effectively self-improve in various tasks. In the implementation of the RLHF framework, (1) a human feedback loop is designed to convert human ratings and evaluations into reward signals for reinforcement learning, thereby guiding the model to generate outputs that meet user needs. (2) A reinforcement learning method based on policy optimization is adopted to continuously adjust the model's generation behavior to better conform to the task objectives. (3) A reward model based on human feedback is constructed to provide real-time feedback on the model's generation quality during training. The overall framework guides the model's generation process through the reward signal of human feedback, optimizes its behavior strategy, and ultimately achieves the adaptive generation of high-quality content. During the training process of RLHF, there are mainly three key strategies: (1) In the initial stage, the model is trained with human-annotated data to enable it to generate outputs that meet the initial requirements; (2) In the reinforcement learning stage, reinforcement learning training is carried out based on human feedback and reward signals to continuously adjust the model's behavior; (3) In the continuous optimization stage, by continuously collecting new user feedback, the reward model is updated to adapt to different task requirements. Through the combination of this heterogeneous supervision and reinforcement learning, RLHF can significantly improve the model's generation quality and has better task adaptability and flexibility.

[0006] DPO (Direct Preference Optimization) is a framework that directly optimizes the model output preference. Through fine-grained policy adjustment, the model can optimize its generation behavior according to human feedback. Different from traditional reinforcement learning methods, DPO directly uses human preference feedback rather than reward signals to guide the model's learning process. This framework adjusts the output of the generation model by converting the user's preference into an optimizable objective, making it more in line with the user's expectations. The key idea of DPO is to regard the generation task as a multi-option decision problem. The model directly learns how to generate content that is more in line with human preferences by comparing the relative advantages and disadvantages of different candidate outputs. Specifically, DPO optimizes the model's parameters by collecting preference data from users, enabling it to more accurately predict and generate high-quality outputs. Compared with the traditional reward model method, DPO avoids the need to construct a complex reward model and provides a more concise and efficient optimization strategy.

[0007] The above framework is implemented by controlling the model to select the response with stronger security in paired answers. Research shows that methods such as RLHF and DPO enhance the model's ability to reject harmful inputs and identify harmful content through additional training. Although these methods have achieved certain results, they have not effectively solved the problem that harmful knowledge still exists in the model, and this harmful knowledge is the root cause of generating harmful responses.

[0008] Although existing technical means have improved the security of large language models to a certain extent, the latest research reveals that even LLMs that have been finely aligned and optimized for security may still generate harmful content in specific "jailbreak" scenarios. Such "jailbreak" attacks usually bypass security protection through special input forms, causing the model to produce responses that violate security policies. To improve the security of LLMs in jailbreak attack scenarios, the most effective method at present is incremental training, that is, through further training steps to make the model reject harmful instructions. However, existing incremental training methods have exposed multiple deficiencies in practical applications: (1) Harmful information or knowledge still remains inside the model, posing potential security risks; (2) The general ability of the model decreases during the incremental training process, resulting in its performance on general problems not meeting expectations. This is because existing incremental training methods have not specifically removed harmful content, but only continuously superimposed new security response patterns, and have not fundamentally solved the harmful information stored inside the model. Therefore, in order to significantly improve the response security of LLMs when facing harmful instructions and problems (especially jailbreak attacks), there is an urgent need for a solution that can accurately remove harmful knowledge in the model without negatively affecting its general knowledge and normal task processing capabilities. This solution needs to be able to effectively eliminate potential harmful content in the model and avoid generating responses that are harmful to users and society. In this way, not only can harmful content inside the model be specifically removed, but its strong functionality and stability can also be ensured when processing diverse tasks, achieving a dual improvement in security and general ability. Summary of the Invention

[0009] To solve the technical problem of how to accurately remove harmful knowledge in the model in the existing technology without negatively affecting its general knowledge and normal task processing capabilities, an embodiment of the present invention provides a large model alignment method and device based on knowledge forgetting. The technical solution is as follows:

[0010] On the one hand, a large model alignment method based on knowledge forgetting is provided. This method is implemented by a large model alignment device based on knowledge forgetting, and the method includes:

[0011] S1. Obtain the respective parameter weights inside the large language model, calculate the importance scores for each neuron of the respective parameter weights, sort the neurons according to the importance scores, and construct a binary knowledge forgetting mask based on the sorting result.

[0012] S2. Determine the knowledge forgetting layer and forgetting module according to the sorting result.

[0013] S3. Obtain harmful instructions and the corresponding harmful responses to the harmful instructions, and construct a harmful knowledge dataset.

[0014] S4. Set the forgetting loss based on the harmful knowledge dataset, and perform harmful knowledge forgetting training on the knowledge forgetting layer and forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm to obtain a trained large language model.

[0015] Optionally, the calculation of the importance score in S1 is as shown in the following formula (1):

[0016]

[0017] Among them,

[0018] L(x) = -logp(y|x) (2)

[0019] In the formula, I(W,x) represents the importance score, W represents the weight matrix, x represents the input sample, represents the weight gradient, L(x) represents the loss function, p(y|x) represents the conditional probability that the model predicts the target output y given the input sample x, and y represents the target output of the input sample.

[0020] Optionally, constructing the binary knowledge forgetting mask in S1 includes:

[0021] Divide the neurons into key knowledge neurons and forgetting neurons according to the sorting result and a preset threshold, and construct a binary knowledge forgetting mask to mark the forgetting neurons.

[0022] Optionally, the forgetting loss based on the harmful knowledge dataset in S4 is as shown in the following formula (3):

[0023]

[0024] In the formula, L f represents the forgetting loss based on the harmful knowledge dataset, D f = {(x,y)|x∈X f ,y∈Y f} represents the harmful knowledge dataset, x represents the input sample, y represents the target output of the input sample, X fIndicates a harmful instruction, Y f Indicates a harmful response corresponding to the harmful instruction, p(y i |x,y <i ) represents that given the input sample x and the previously generated output y <i the model generates the i-th output y i under the condition of, y i represents the output at the i-th position.

[0025] Optionally, the harmful knowledge forgetting training of the knowledge forgetting layer and the forgetting module according to the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm in S4 includes:

[0026] S41. Set the gradient update of the key knowledge neurons to zero.

[0027] S42. For the forgetting neurons marked with the binarized knowledge forgetting mask, use the restricted gradient ascent algorithm to enable the large language model to perform harmful knowledge forgetting training based on the forgetting loss on the harmful knowledge dataset.

[0028] Optionally, the harmful knowledge forgetting training in S42 further includes:

[0029] Add a constraint parameter to the forgetting loss based on the harmful knowledge dataset, as shown in the following formula (4):

[0030] L = max(0, λ + L f ) (4)

[0031] In the formula, L represents the final loss function, λ represents the forgetting constraint parameter, and L f represents the forgetting loss based on the harmful knowledge dataset.

[0032] On the other hand, a large model alignment device based on knowledge forgetting is provided. The device is applied to the large model alignment method based on knowledge forgetting, and the device includes:

[0033] A knowledge forgetting mask construction module, configured to obtain each parameter weight inside the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binarized knowledge forgetting mask according to the sorting result.

[0034] A forgetting layer and forgetting module selection module, configured to determine the knowledge forgetting layer and the forgetting module according to the sorting result.

[0035] A dataset construction module, configured to obtain harmful instructions and harmful responses corresponding to the harmful instructions, and construct a harmful knowledge dataset.

[0036] An output module, configured to set a forgetting loss based on a harmful knowledge dataset, and perform harmful knowledge forgetting training on a knowledge forgetting layer and a forgetting module according to a binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and a constrained gradient ascent algorithm, so as to obtain a trained large language model.

[0037] Optionally, calculate an importance score as shown in the following formula (1):

[0038]

[0039] where

[0040] L(x) = -logp(y|x) (2)

[0041] In the formula, I(W,x) represents the importance score, W represents the weight matrix, x represents the input sample, represents the weight gradient, L(x) represents the loss function, p(y|x) represents the conditional probability that the model predicts the target output y given the input sample x, and y represents the target output of the input sample.

[0042] Optionally, a knowledge forgetting mask construction module, further configured to:

[0043] According to the sorting result and a preset threshold, divide the neurons into key knowledge neurons and forgetting neurons, and construct a binary knowledge forgetting mask for the forgetting neurons for marking.

[0044] Optionally, the forgetting loss based on the harmful dataset is as shown in the following formula (3):

[0045]

[0046] In the formula, L f represents the forgetting loss based on the harmful dataset, D f = {(x,y)|x ∈ X f , y ∈ Y f} represents the harmful knowledge dataset, x represents the input sample, y represents the target output of the input sample, X f represents the harmful instruction, Y f represents the harmful response corresponding to the harmful instruction, p(y i |x,y <i ) represents the conditional probability that the model generates the i-th output y <i given the input sample x and the previously generated output y i , and y i represents the output at the i-th position.

[0047] Optionally, the output module, further configured to:

[0048] S41. Set the gradient update of the key knowledge neurons to zero.

[0049] S42. For the forgetting neurons marked with the binary knowledge forgetting mask, use the restricted gradient ascent algorithm to enable the large language model to perform harmful knowledge forgetting training based on the forgetting loss on the harmful knowledge dataset.

[0050] Optionally, the harmful knowledge forgetting training further includes:

[0051] Add a constraint parameter to the forgetting loss based on the harmful knowledge dataset, as shown in the following formula (4):

[0052] L = max(0, λ + L f ) (4)

[0053] In the formula, L represents the final loss function, λ represents the forgetting constraint parameter, and L f represents the forgetting loss based on the harmful knowledge dataset.

[0054] On the other hand, a large model alignment device based on knowledge forgetting is provided. The large model alignment device based on knowledge forgetting includes: a processor; a memory, and computer-readable instructions are stored on the memory. When the computer-readable instructions are executed by the processor, any one of the methods in the above-mentioned large model alignment method based on knowledge forgetting is implemented.

[0055] On the other hand, a computer-readable storage medium is provided. At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by the processor to implement any one of the methods in the above-mentioned large model alignment method based on knowledge forgetting.

[0056] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0057] In the present invention, the proposed CKU method transforms the security alignment task into a restricted knowledge forgetting task by using constraint optimization technology, precisely removes unnecessary or harmful knowledge in the generative large language model, and at the same time ensures that the overall performance and efficiency of the model are maintained.

[0058] Flexible forgetting layer selection: The CKU method enables users to selectively perform forgetting training on specific MLP layers of the LLM according to specific application requirements through a flexible forgetting layer selection mechanism. This customized optimization method enables the model to effectively control risks in different security scenarios while ensuring the efficiency and applicability of the model.

[0059] Enhancing the robustness of the model: By fixing a part of the neurons in the MLP layer, the CKU method improves the stability and anti-interference ability of the LLM when facing external attacks or malicious inputs. This optimization increases the long-term reliability of the LLM in complex application environments and provides strong support for security in practical applications.

[0060] Minimizing the impact on the general capabilities of the model: CKU adopts refined gradient pruning technology to minimize the damage to general capabilities when removing harmful knowledge from the LLM, enabling the LLM to maintain high accuracy and practicality while improving security. This technology ensures the stable performance of the model in various tasks.

[0061] Superior balance between security and practicality: The CKU method achieves an ideal balance between security and practicality. By accurately identifying and pruning neurons related to harmful knowledge, it ensures that the LLM can operate efficiently in most application scenarios while providing effective protection against security threats. Description of the Drawings

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0063] Figure 1 is a flowchart of a large model alignment method based on knowledge forgetting provided by an embodiment of the present invention;

[0064] Figure 2 is a schematic diagram of a large model alignment method based on knowledge forgetting provided by an embodiment of the present invention;

[0065] Figure 3 is a block diagram of a large model alignment device based on knowledge forgetting provided by an embodiment of the present invention;

[0066] Figure 4 is a schematic structural diagram of a large model alignment device based on knowledge forgetting provided by an embodiment of the present invention. Detailed Embodiments

[0067] The technical solutions in the present invention will be described below with reference to the drawings.

[0068] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to give examples, illustrations or explanations. Any embodiment or design solution described as an "example" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of the word "example" is intended to present concepts in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two.

[0069] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meaning they express is the same. "(of)", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meaning they express is the same.

[0070] In the embodiments of the present invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0071] To make the technical problems to be solved, technical solutions and advantages of the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0072] The embodiments of the present invention provide a large model alignment method based on knowledge forgetting, aiming to solve the security problem in generative language models. This method can be implemented by a large model alignment device based on knowledge forgetting, and the large model alignment device based on knowledge forgetting can be a terminal or a server. As Figure 1 、 Figure 2 shown in the flowchart of the large model alignment method based on knowledge forgetting, the processing flow of this method can include the following steps:

[0073] S1. Obtain each parameter weight inside the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask according to the sorting result.

[0074] In a feasible implementation, in order to identify and retain the key knowledge in the model, a forgetting knowledge mask is constructed. Through the neuron importance evaluation method, the contribution degree of each neuron to different categories of knowledge is calculated, and whether to perform gradient clipping is determined based on its influence.

[0075] Furthermore, for various parameter weights inside the LLM, given the input sample x and its target output y, the loss function is defined as:

[0076] L(x) = -logp(y|x) (1)

[0077] Then, the importance score of each neuron \(W\) in the weight matrix \(W\) ij is defined as:

[0078]

[0079] To globally evaluate the importance of neurons, the mean value is calculated for the general dataset \(D\):

[0080]

[0081] In the formula, \(I(W, x)\) represents the importance score, \(W\) represents the weight matrix, represents the weight gradient, \(L(x)\) represents the loss function, \(p(y|x)\) represents the conditional probability that the model predicts the target output \(y\) given the input sample \(x\), and \(y\) represents the target output of the input sample.

[0082] Furthermore, based on the neuron importance scores, the weight matrix \(W\) is sorted, a fixed threshold \(p\) is set, and the top \(p\%\) neurons are selected as key knowledge neurons, and a binary knowledge forgetting mask \(M\) is constructed.

[0083] The present invention measures the contribution of neurons to different categories of knowledge, evaluates the influence of neurons on general knowledge, ensures the retention of key knowledge, assigns appropriate forgetting weights to it, and constructs a forgetting knowledge mask.

[0084] S2. Determine the knowledge forgetting layer and forgetting module according to the sorting result.

[0085] In a feasible implementation, by experimentally analyzing the contribution of neurons in different layers and different modules to the model security and general knowledge, a neuron screening method based on importance scores is adopted to accurately identify the key knowledge storage area, and the optimal knowledge forgetting layer and forgetting module are determined. During the experiment, for the multi-layer decoder structure of the model, the influence of forgetting training at different levels on the model security and general ability is systematically tested. Then, the influence of different neuron locking rates on security and the retention of general knowledge is analyzed through experiments and knowledge forgetting training is performed.

[0086] The present invention designs the optimal forgetting layer and forgetting strategy by experimentally analyzing the influence of different levels and different modules on knowledge forgetting, so as to maximize the effect of forgetting harmful knowledge while minimizing side effects.

[0087] S3. Obtain harmful instructions and corresponding harmful responses, and construct a harmful knowledge dataset.

[0088] S4. Set the forgetting loss based on the harmful knowledge dataset, and perform harmful knowledge forgetting training on the knowledge forgetting layer and the forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm to obtain a trained large language model.

[0089] In a feasible implementation, the method mainly includes key steps such as constructing a harmful knowledge dataset, performing gradient ascent training, and forgetting regularization strategies. Construct a harmful knowledge dataset D f , which is used to guide the harmful knowledge forgetting training of the model to ensure that the model can effectively shield or reject harmful requests. Its specific format is a pair of (prompt, response):

[0090] D f = {(x, y)|x ∈ X f , y ∈ Y f} (4)

[0091] where X f is the attack input and Y f is the harmful response. Define the optimization objective:

[0092]

[0093] In the formula, p(y i |x, y <i ) represents the conditional probability that the model generates the i-th output y <i given the input sample x and the previously generated output y i , and y i represents the output at the i-th position.

[0094] Optionally, the harmful knowledge forgetting training of the knowledge forgetting layer and the forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm in S4 may include the following steps S41 - S42:

[0095] S41. Set the gradient update of the key knowledge neurons to zero.

[0096] S42. For the forgetting neurons marked with the binary knowledge forgetting mask, use the restricted gradient ascent algorithm to make the large language model perform harmful knowledge forgetting training based on the forgetting loss on the harmful knowledge dataset.

[0097] In a feasible implementation, during the optimization process, the present invention adopts a restricted gradient update mechanism, which needs to lock the gradient update of the key neurons to ensure the retention of the core knowledge of the model; in addition, it is necessary to perform gradient ascent only at the neurons to be forgotten to weaken the model's dependence on harmful knowledge.

[0098] Specifically, assume that M is the forgotten knowledge mask matrix, which is used to mark the neurons for which gradient clipping should be performed. Then:

[0099]

[0100] Furthermore, during the process of harmful knowledge forgetting, in order to ensure that the model can retain core knowledge, for the locked key neurons, their gradient updates are forced to be set to zero, that is:

[0101]

[0102] Furthermore, when performing harmful knowledge forgetting, the present invention adopts the gradient ascent technique to make the model perform reverse optimization on the harmful data set D f to reduce its dependence on this type of data. Specifically:

[0103]

[0104] where θ represents the model parameters, η is the learning rate, and L unlearn (θ) is the forgetting loss function, whose goal is to maximize the loss of the model on the data set D f to weaken the model's mastery of harmful knowledge. Gradient ascent is performed at the forgotten neurons to increase the value of the loss function, thereby prompting the model to gradually "forget" harmful knowledge. Prevent gradient adjustment from affecting the core knowledge storage area to ensure the integrity of general capabilities.

[0105] Although harmful knowledge forgetting training can effectively improve the security of the model, excessive forgetting may lead to a decline in the overall ability of the model, affecting its normal reasoning and task execution capabilities. Therefore, the present invention introduces a forgetting regularization mechanism to balance the forgetting intensity and the general performance of the model.

[0106] Furthermore, during the forgetting process, the present invention sets the loss function constraint parameter λ to control the degree of forgetting and prevent overtraining. The final loss function is defined as follows:

[0107] L = max(0, λ + L f ) (9)

[0108] In the formula, λ is the forgetting constraint parameter, which controls the upper limit of forgetting. L f is the forgetting loss based on the harmful data set. When the degree of forgetting reaches the set threshold, the optimization process automatically stops to prevent the model from overforgetting key knowledge. Among them, when L f exceeds the set threshold λ, the training automatically stops to prevent the decline of the general ability of the model.

[0109] During the forgetting training process of the present invention, by clipping the gradients at the positions of the forgotten knowledge masks, the gradient direction and magnitude are constrained to prevent general knowledge from being accidentally damaged during the forgetting process.

[0110] The present invention provides a secure alignment method based on CKU (Constrained Knowledge Unlearning), which is used to effectively delete harmful knowledge in large language models while retaining general knowledge. This solution constructs the secure alignment task as a knowledge forgetting task, including three processes: knowledge forgetting mask construction, forgetting layer and forgetting module selection, and constrained gradient ascent algorithm, to achieve precise suppression of harmful content and enhance the security and robustness of the model.

[0111] Previous secure alignment methods for large models only trained through simple rules or reward models, without fully considering the impact of harmful knowledge inside the model on the responses. The present invention adopts the basic framework of knowledge forgetting, specifically forgetting harmful knowledge and retaining general knowledge during the forgetting process, ensuring that while the security of the model is improved, its ability to handle normal tasks is maintained. Before the training phase, by calculating the correlation between each neuron and general knowledge, the neurons related to harmful content are identified and precisely calibrated. Then, during the training process, through the gradient pruning mechanism, the influence of highly correlated neurons on the model update is specifically reduced, thereby effectively weakening the contribution of neurons related to harmful knowledge to the final output. This method enables the large model to retain general knowledge and valid information related to the task during the knowledge forgetting process, while avoiding over-forgetting or deleting non-harmful parts that may have a negative impact on the model performance, ensuring that the general ability does not decline while the security is improved. In this way, potential harmful risks can be effectively reduced without sacrificing the diversity and generality of the model, thereby improving the security and reliability of the model in complex environments.

[0112] In the embodiments of the present invention, the proposed CKU method transforms the secure alignment task into a constrained knowledge forgetting task by adopting constraint optimization techniques, precisely removing unnecessary or harmful knowledge in generative large language models while ensuring that the overall performance and efficiency of the model are maintained.

[0113] Flexible forgetting layer selection: The CKU method enables users to selectively perform forgetting training on specific MLP layers of the LLM according to specific application requirements through a flexible forgetting layer selection mechanism. This customized optimization method enables the model to effectively control risks in different security scenarios while ensuring the efficiency and applicability of the model.

[0114] Enhancing the Robustness of the Model: By fixing a portion of neurons in the MLP layer, the CKU method improves the stability and anti-interference ability of the LLM when facing external attacks or malicious inputs. This optimization increases the long-term reliability of the LLM in complex application environments and provides strong support for security in practical applications.

[0115] Minimizing the Impact on the General Capabilities of the Model: CKU adopts refined gradient pruning techniques to minimize the damage to general capabilities when removing harmful knowledge from the LLM, enabling the LLM to maintain high accuracy and practicality while enhancing security. This technique ensures the stable performance of the model in various tasks.

[0116] Superior Balance between Security and Practicality: The CKU method achieves an ideal balance between security and practicality. By precisely identifying and pruning neurons related to harmful knowledge, it ensures that the LLM can operate efficiently in most application scenarios while providing effective protection against security threats.

[0117] Figure 3 is a block diagram of a large model alignment device based on knowledge forgetting shown according to an exemplary embodiment. This device is used for the large model alignment method based on knowledge forgetting. Refer to Figure 3 This device includes a knowledge forgetting mask construction module 310, a forgetting layer and forgetting module selection module 320, a dataset construction module 330, and an output module 340. Among them:

[0118] The knowledge forgetting mask construction module 310 is used to obtain the respective parameter weights inside the large language model, calculate the importance scores for each neuron of the respective parameter weights, sort the neurons according to the importance scores, and construct a binary knowledge forgetting mask according to the sorting results.

[0119] The forgetting layer and forgetting module selection module 320 is used to determine the knowledge forgetting layer and forgetting module according to the sorting results.

[0120] The dataset construction module 330 is used to obtain harmful instructions and the corresponding harmful responses to the harmful instructions, and construct a harmful knowledge dataset.

[0121] The output module 340 is used to set a forgetting loss based on the harmful knowledge dataset, and perform harmful knowledge forgetting training on the knowledge forgetting layer and forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm to obtain a trained large language model.

[0122] In the embodiments of the present invention, the proposed CKU method converts the secure alignment task into a restricted knowledge forgetting task by adopting constraint optimization techniques, precisely removing unnecessary or harmful knowledge in the generative large language model while ensuring the overall performance and efficiency of the model are maintained.

[0123] Flexible forgetting layer selection: The CKU method enables users to selectively perform forgetting training on specific MLP layers of the LLM according to specific application requirements through a flexible forgetting layer selection mechanism. This customized optimization method allows the model to effectively control risks in different security scenarios while ensuring the efficiency and applicability of the model.

[0124] Enhancing the robustness of the model: By fixing a part of the neurons in the MLP layer, the CKU method improves the stability and anti-interference ability of the LLM when facing external attacks or malicious inputs. This optimization increases the long-term reliability of the LLM in complex application environments and provides strong support for security in practical applications.

[0125] Minimizing the impact on the general capabilities of the model: CKU adopts refined gradient pruning techniques to minimize the damage to general capabilities when removing harmful knowledge from the LLM, enabling the LLM to maintain high accuracy and practicality while improving security. This technique ensures the stable performance of the model in various tasks.

[0126] Superior balance between security and practicality: The CKU method achieves an ideal balance between security and practicality. By precisely identifying and pruning neurons related to harmful knowledge, it ensures that the LLM can operate efficiently in most application scenarios while providing effective protection against security threats.

[0127] Figure 4 FIG. is a schematic structural diagram of a large model alignment device based on knowledge forgetting provided by an embodiment of the present invention. As Figure 4 shown, the large model alignment device based on knowledge forgetting may include the above-mentioned Figure 3 shown large model alignment device based on knowledge forgetting. Optionally, the large model alignment device 410 based on knowledge forgetting may include a first processor 2001.

[0128] Optionally, the large model alignment device 410 based on knowledge forgetting may further include a memory 2002 and a transceiver 2003.

[0129] Wherein, the first processor 2001, the memory 2002, and the transceiver 2003 may be connected through a communication bus, for example.

[0130] Next, the specific components of the large model alignment device 410 based on knowledge forgetting will be introduced in conjunction with Figure 4 :

[0131] Among them, the first processor 2001 is the control center of the large model alignment device 410 based on knowledge forgetting, which can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or can be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. For example: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0132] Optionally, the first processor 2001 can execute various functions of the large model alignment device 410 based on knowledge forgetting by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0133] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 the CPU0 and CPU1 shown in

[0134] In a specific implementation, as an embodiment, the large model alignment device 410 based on knowledge forgetting may also include multiple processors, such as Figure 4 the first processor 2001 and the second processor 2004 shown in

[0135] Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, the processor can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0136] Optionally, the memory 2002 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled to the first processor 2001 through an interface circuit of the large model alignment device 410 based on knowledge forgetting ( Figure 4 not shown in the figure), and the embodiments of the present invention do not make specific limitations in this regard.

[0137] The transceiver 2003 is used to communicate with a network device or with a terminal device.

[0138] Optionally, the transceiver 2003 can include a receiver and a transmitter ( Figure 4 not shown separately in the figure). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0139] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently and be coupled to the first processor 2001 through an interface circuit of the large model alignment device 410 based on knowledge forgetting ( Figure 4 not shown in the figure), and the embodiments of the present invention do not make specific limitations in this regard.

[0140] It should be noted that Figure 4 the structure of the large model alignment device 410 based on knowledge forgetting shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0141] In addition, the technical effects of the large model alignment device 410 based on knowledge forgetting can refer to the technical effects of the large model alignment method based on knowledge forgetting described in the above method embodiments, and will not be elaborated here.

[0142] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0143] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0144] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0145] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. Additionally, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be understood specifically by referring to the context before and after.

[0146] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0147] It should be understood that in various embodiments of the present invention, the magnitudes of the sequence numbers of the above processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0148] Those of ordinary skill in the art will realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0149] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices, apparatuses, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0150] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0153] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0154] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A large model alignment method based on knowledge forgetting, characterized in that, The method includes: S1. Obtain each parameter weight inside the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask according to the sorting result; S2. Determine the knowledge forgetting layer and forgetting module according to the sorting result; S3. Obtain harmful instructions and corresponding harmful responses to the harmful instructions, and construct a harmful knowledge dataset; S4. Set a forgetting loss based on the harmful knowledge dataset, and perform harmful knowledge forgetting training on the knowledge forgetting layer and forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm to obtain a trained large language model.

2. The large model alignment method based on knowledge forgetting according to claim 1, wherein The calculation of the importance score in S1 is as shown in the following formula (1): Where L(x) = -logp(y|x) (2) Wherein, I(W, x) represents the importance score, W represents the weight matrix, x represents the input sample, represents the weight gradient, L(x) represents the loss function, p(y|x) represents the conditional probability that the model predicts the target output y given the input sample x, and y represents the target output of the input sample.

3. The large model alignment method based on knowledge forgetting according to claim 1, characterized in that The construction of the binary knowledge forgetting mask according to the sorting result in S1 includes: Divide the neurons into key knowledge neurons and forgetting neurons according to the sorting result and a preset threshold, and construct a binary knowledge forgetting mask to mark the forgetting neurons.

4. The large model alignment method based on knowledge forgetting according to claim 1, wherein The forgetting loss based on the harmful knowledge dataset in S4 is as shown in the following formula (3): where L f represents the forgetting loss based on the harmful dataset, D f ={(x, y)|x ∈ X f , y ∈ Y f} represents the harmful knowledge dataset, x represents the input sample, y represents the target output of the input sample, X f represents the harmful instruction, Y f represents the harmful response corresponding to the harmful instruction, p(y i |x, y <i ) represents the conditional probability that the model generates the i-th output y <i given the input sample x and the previously generated output y i , and y i represents the output at the i-th position.

5. The large model alignment method based on knowledge forgetting according to claim 1, characterized in that The performing of harmful knowledge forgetting training on the knowledge forgetting layer and forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm in S4 includes: S41. Set the gradient update of the key knowledge neurons to zero; S42. For the forgetting neurons marked with the binary knowledge forgetting mask, use the restricted gradient ascent algorithm to perform harmful knowledge forgetting training on the large language model based on the forgetting loss on the harmful knowledge dataset.

6. The method for aligning large models based on knowledge forgetting according to claim 5, wherein The harmful knowledge forgetting training in S42 further includes: Add a constraint parameter to the forgetting loss based on the harmful knowledge dataset, as shown in the following formula (4): L = max(0, λ + L f ) (4) Wherein, L represents the final loss function, λ represents the forgetting constraint parameter, and L f represents the forgetting loss based on the harmful knowledge dataset.

7. A large model alignment device based on knowledge forgetting, the large model alignment device based on knowledge forgetting is used to implement the large model alignment method based on knowledge forgetting according to any one of claims 1-6, characterized in that, The device includes: A knowledge forgetting mask construction module, configured to obtain each parameter weight inside the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask according to the sorting result; A forgetting layer and forgetting module selection module, configured to determine the knowledge forgetting layer and forgetting module according to the sorting result; A dataset construction module, configured to obtain harmful instructions and corresponding harmful responses to the harmful instructions, and construct a harmful knowledge dataset; An output module, configured to set a forgetting loss based on the harmful knowledge dataset, and perform harmful knowledge forgetting training on the knowledge forgetting layer and forgetting module according to the binary knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm to obtain a trained large language model.

8. The large model alignment device based on knowledge forgetting according to claim 7, wherein The construction of the binary knowledge forgetting mask according to the sorting result includes: Divide the neurons into key knowledge neurons and forgetting neurons according to the sorting result and a preset threshold, and construct a binary knowledge forgetting mask to mark the forgetting neurons.

9. A large model alignment device based on knowledge forgetting, characterized in that, The large model alignment device based on knowledge forgetting includes: A processor; A memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that, Program code is stored in the computer-readable storage medium, and the program code can be called by the processor to execute the method according to any one of claims 1 to 6.

Images