A method and device for aligning large models based on knowledge forgetting

Abstract

The application provides a kind of big model alignment method and device based on knowledge forgetting, it is related to natural language processing technical field.The method comprises: obtaining each parameter weight inside big language model, calculating importance score for each neuron, sorting neuron, and constructing binary knowledge forgetting mask;According to the sorting result, determine the knowledge forgetting layer and the forgetting module;Obtain harmful instruction and harmful response, and construct harmful knowledge dataset;Set optimization goal, and according to binary knowledge forgetting mask, harmful knowledge dataset, optimization goal and restricted gradient ascent algorithm, harmful knowledge forgetting training is carried out on knowledge forgetting layer and forgetting module, and a trained big language model is obtained.The CKU method proposed in the application converts the safe alignment task into a restricted knowledge forgetting task by using constraint optimization technology, accurately removes unnecessary or harmful knowledge in the generative big language model, while ensuring that the overall performance and efficiency of the model are maintained.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a large model alignment method and apparatus based on knowledge forgetting. Background Technology

[0002] Since the tremendous success of ChatGPT's Chat Generative Pre-trained Transformer, generative LLMs (Large Language Models) have been widely applied to all aspects of daily life. From personal assistants and intelligent customer service to content creation and automatic translation, large language models have demonstrated immense potential across various industries. Their powerful language understanding and generation capabilities enable these models not only to perform simple tasks such as question answering, sentiment analysis, and language translation, but also to accomplish more complex tasks such as coding, generating creative content, and solving interdisciplinary problems. Meanwhile, the application of LLMs in fields such as medicine, education, and finance is deepening, and their potential for assisting decision-making and improving service quality is being fully realized. More importantly, the emergence of generative language models has greatly promoted the efficiency of information exchange and knowledge dissemination. By deeply learning from large amounts of text data, LLMs can understand and generate natural language text, simulating human language processing capabilities, thus providing unprecedented innovative solutions to many traditional tasks. However, despite the numerous conveniences brought by LLMs, their safety and ethical issues remain a focus of attention for society and academia. As these technologies continue to develop, ensuring that they provide powerful functionality while avoiding misuse or adverse consequences remains a key direction for future research.

[0003] Employing diverse alignment strategies to optimize the output of generative large-scale language models can significantly improve the model's safety and reliability in practical applications, while effectively reducing outputs that do not conform to human values ​​or ethical standards. Through this alignment mechanism, the model can not only better understand and follow user intent but also maintain consistency with human expectations when generating content, avoiding bias, misleading, or inappropriate behavior. This alignment is not only reflected in the accuracy and compliance of content but also, to a certain extent, prevents the model from producing uncontrolled or harmful outputs in specific situations, ensuring its compliance with overall social moral and legal requirements. The application of alignment technology enhances the model's adaptability in different contexts, enabling it to provide more ethical and legally compliant outputs in various complex tasks, thereby improving the model's social value and sustainability. Especially in sensitive areas or high-risk applications, such as medical diagnosis and legal consultation, ensuring that the model's behavior is consistent with human ethical values ​​can significantly improve the model's controllability and trustworthiness, promoting the widespread application and popularization of generative large-scale language models in natural language processing and generation. The continuous development and improvement of this technology will ensure that future generative language models not only possess powerful technical capabilities but also a higher level of social responsibility and ethical standards.

[0004] Large language models have demonstrated powerful generative capabilities in many natural language processing tasks, producing fluent and creative text. However, with the diversification of these models' applications, issues regarding the controllability and consistency of their generated results have become increasingly apparent. Because the generation process of large language models is based on large amounts of pre-training data and complex statistical relationships, the generated content often lacks a clear goal orientation and may even deviate from user expectations. Therefore, how to make the model's output more in line with human needs and expectations has become an important research topic in the field of natural language processing. To achieve this goal, more and more researchers are exploring various technical approaches, including the introduction of reinforcement learning and user feedback mechanisms. Reinforcement learning optimizes the generation strategy through interaction with the environment, enabling the model to gradually adjust its output according to preset goals, improving the quality and relevance of the results. User feedback mechanisms, on the other hand, collect user evaluations of the generated content in real time and adjust the model's behavior to better meet user needs.

[0005] RLHF (Reinforcement Learning with Human Feedback) is a technical framework that combines reinforcement learning with human feedback to improve the decision-making ability and generation quality of models. The core technologies of RLHF include: (1) guiding model behavior through human feedback to ensure that model output meets actual needs; (2) further optimizing model policies using reward signals through reinforcement learning algorithms; and (3) training the model on large-scale human feedback data to enable effective self-improvement across various tasks. In the implementation of the RLHF framework, (1) a human feedback loop is designed to convert human ratings and evaluations into reward signals for reinforcement learning, thereby guiding the model to generate outputs that meet user needs; (2) a policy optimization-based reinforcement learning method is adopted to continuously adjust the model's generation behavior to better align with task objectives; and (3) a reward model based on human feedback is constructed to provide real-time feedback on the model's generation quality during training. The overall framework guides the model's generation process through reward signals from human feedback, optimizes its behavioral strategies, and ultimately achieves adaptive generation of high-quality content. The training process of RLHF mainly includes three key strategies: (1) the initial stage, which trains the model using human-labeled data to generate outputs that meet the initial requirements; (2) the reinforcement learning stage, which trains the model based on human feedback and reward signals to continuously adjust the model behavior; and (3) the continuous optimization stage, which updates the reward model by continuously collecting new user feedback to adapt it to different task requirements. Through this combination of heterogeneous supervision and reinforcement learning, RLHF can significantly improve the generation quality of the model and has better task adaptability and flexibility.

[0006] Direct Preference Optimization (DPO) is a framework for directly optimizing model output preferences. Through fine-tuning of policies, it enables the model to optimize its generative behavior based on human feedback. Unlike traditional reinforcement learning methods, DPO directly utilizes human preference feedback rather than reward signals to guide the model's learning process. This framework adjusts the generative model's output to better align with user expectations by transforming user preferences into an optimizable objective. The key idea of ​​DPO is to treat the generation task as a multi-option decision problem. The model learns directly how to generate content that better suits human preferences by comparing the relative merits of different candidate outputs. Specifically, DPO collects preference data from users to optimize the model's parameters, enabling it to more accurately predict and generate high-quality outputs. Compared to traditional reward model methods, DPO avoids the need to build complex reward models, providing a simpler and more efficient optimization strategy.

[0007] The aforementioned framework controls the model's selection of the safer answer from paired responses. Research indicates that methods such as RLHF and DPO enhance the model's ability to reject harmful inputs and identify harmful content through additional training. While these methods have achieved some success, they do not effectively address the problem of harmful knowledge remaining within the model, which is the root cause of harmful responses.

[0008] Although existing technologies have improved the security of large language models to some extent, recent research reveals that even finely aligned and security-optimized LLMs can still generate harmful content in specific "jailbreak" scenarios. Such "jailbreak" attacks typically bypass security protections through special input methods, causing the model to produce responses that violate security policies. To improve the security of LLMs in jailbreak attack scenarios, the most effective method at present is incremental training, which involves further training steps to make the model reject harmful instructions. However, existing incremental training methods have several shortcomings in practical applications: (1) harmful information or knowledge is still retained within the model, posing potential security risks; (2) the model's general capabilities decrease during incremental training, resulting in performance on general problems falling short of expectations. This is because existing incremental training methods fail to specifically remove harmful content, but only continuously add new security response patterns, failing to fundamentally address the harmful information stored within the model. Therefore, to significantly improve the security of LLM responses to harmful instructions and problems (especially jailbreak attacks), a solution is urgently needed that can accurately remove harmful knowledge from the model without negatively impacting its general knowledge and normal task processing capabilities. The solution needs to effectively eliminate potentially harmful content from the model, preventing the generation of responses harmful to users and society. This approach not only allows for targeted removal of harmful content within the model but also ensures robust functionality and stability when handling diverse tasks, achieving a dual improvement in security and versatility. Summary of the Invention

[0009] To address the technical problem of how to accurately remove harmful knowledge from models without negatively impacting their general knowledge and normal task processing capabilities, this invention provides a method and apparatus for aligning large models based on knowledge forgetting. The technical solution is as follows:

[0010] On the one hand, a large model alignment method based on knowledge forgetting is provided, which is implemented by a large model alignment device based on knowledge forgetting. The method includes:

[0011] S1. Obtain the weights of each parameter within the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask based on the sorting results.

[0012] S2. Determine the knowledge forgetting layer and forgetting module based on the sorting results.

[0013] S3. Obtain harmful commands and their corresponding harmful responses to construct a harmful knowledge dataset.

[0014] S4. Set a forgetting loss based on the harmful knowledge dataset, and train the knowledge forgetting layer and forgetting module for harmful knowledge forgetting according to the binarized knowledge forgetting mask, harmful knowledge dataset, forgetting loss and restricted gradient ascent algorithm to obtain a trained large language model.

[0015] Optionally, the importance score in S1 is calculated as shown in equation (1) below:

[0016]

[0017] in,

[0018] L(x)=-logp(y|x) (2)

[0019] In the formula, I(W,x) represents the importance score, W represents the weight matrix, and x represents the input sample. Let represent the weight gradient, L(x) represent the loss function, and p(y|x) represent the conditional probability of the model predicting the target output y given the input sample x, where y represents the target output of the input sample.

[0020] Optionally, S1 constructs a binary knowledge forgetting mask based on the sorting results, including:

[0021] Based on the sorting results and the preset threshold, neurons are divided into key knowledge neurons and forgotten neurons, and a binary knowledge forgetting mask is constructed and labeled for the forgotten neurons.

[0022] Optionally, the forgetting loss in S4 based on the harmful knowledge dataset is as shown in equation (3):

[0023]

[0024] In the formula, L f D represents the forgetting loss based on a harmful knowledge dataset. f ={(x,y)|x∈X f ,y∈Y f Let} represent the harmful knowledge dataset, x represent the input sample, and y represent the target output of the input sample. fIndicates a harmful instruction, Y f p(y) represents the harmful response corresponding to the harmful instruction. i |x,y <i ) represents the output y generated given the input sample x and the previous input sample x. <i In this case, the model generates the i-th output y. i The conditional probability, y i This represents the output at the i-th position.

[0025] Optionally, in S4, the knowledge forgetting layer and forgetting module are trained for harmful knowledge forgetting based on the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm, including:

[0026] S41. Set the gradient update of the key knowledge neurons to zero.

[0027] S42. For the forgetting neurons labeled with binary knowledge forgetting mask, the restricted gradient ascent algorithm is used to enable the large language model to be trained on the harmful knowledge dataset based on the forgetting loss.

[0028] Optionally, the harmful knowledge forgetting training in S42 also includes:

[0029] The following constraint parameter is added to the forgetting loss based on the harmful knowledge dataset, as shown in equation (4):

[0030] L=max(0,λ+L f (4)

[0031] In the formula, L represents the final loss function, λ represents the forgetting constraint parameter, and L f This represents the forgetting loss based on a harmful knowledge dataset.

[0032] On the other hand, a large model alignment device based on knowledge forgetting is provided, which is applied to a large model alignment method based on knowledge forgetting. The device includes:

[0033] The knowledge forgetting mask construction module is used to obtain the weights of each parameter within the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask based on the sorting result.

[0034] The forgetting layer and forgetting module selection module is used to determine the knowledge forgetting layer and forgetting module based on the sorting results.

[0035] The dataset construction module is used to obtain harmful commands and their corresponding harmful responses to build a harmful knowledge dataset.

[0036] The output module is used to set the forgetting loss based on the harmful knowledge dataset. Based on the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm, the knowledge forgetting layer and the forgetting module are trained to perform harmful knowledge forgetting to obtain a trained large language model.

[0037] Optionally, the importance score is calculated as shown in equation (1):

[0038]

[0039] in,

[0040] L(x)=-logp(y|x) (2)

[0041] In the formula, I(W,x) represents the importance score, W represents the weight matrix, and x represents the input sample. Let represent the weight gradient, L(x) represent the loss function, and p(y|x) represent the conditional probability of the model predicting the target output y given the input sample x, where y represents the target output of the input sample.

[0042] Optionally, the knowledge forgetting mask building module is further used for:

[0043] Based on the sorting results and the preset threshold, neurons are divided into key knowledge neurons and forgotten neurons, and a binary knowledge forgetting mask is constructed and labeled for the forgotten neurons.

[0044] Alternatively, the forgetting loss based on the harmful dataset is as shown in equation (3):

[0045]

[0046] In the formula, L f D represents the forgetting loss based on harmful datasets. f ={(x,y)|x∈X f ,y∈Y f Let} represent the harmful knowledge dataset, x represent the input sample, and y represent the target output of the input sample. f Indicates a harmful instruction, Y f p(y) represents the harmful response corresponding to the harmful instruction. i |x,y <i ) represents the output y generated given the input sample x and the previous input sample x. <i In this case, the model generates the i-th output y. i The conditional probability, y i This represents the output at the i-th position.

[0047] Optionally, the output module is further used for:

[0048] S41. Set the gradient update of the key knowledge neurons to zero.

[0049] S42. For the forgetting neurons labeled with binary knowledge forgetting mask, the restricted gradient ascent algorithm is used to enable the large language model to be trained on the harmful knowledge dataset based on the forgetting loss.

[0050] Optionally, training for the forgetting of harmful knowledge also includes:

[0051] The following constraint parameter is added to the forgetting loss based on the harmful knowledge dataset, as shown in equation (4):

[0052] L=max(0,λ+L f (4)

[0053] In the formula, L represents the final loss function, λ represents the forgetting constraint parameter, and L f This represents the forgetting loss based on a harmful knowledge dataset.

[0054] On the other hand, a large model alignment device based on knowledge forgetting is provided, the large model alignment device based on knowledge forgetting includes: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods in the large model alignment method based on knowledge forgetting described above.

[0055] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described large model alignment methods based on knowledge forgetting.

[0056] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0057] In this invention, the proposed CKU method transforms the safe alignment task into a restricted knowledge forgetting task by employing constrained optimization techniques. This accurately removes unnecessary or harmful knowledge from generative large language models while ensuring that the overall performance and efficiency of the model are maintained.

[0058] Flexible Forget Layer Selection: The CKU method employs a flexible forget layer selection mechanism, allowing users to selectively perform forgetting training on specific MLP layers of the LLM based on specific application requirements. This customized optimization approach enables the model to effectively control risks in different security scenarios while ensuring the model's efficiency and applicability.

[0059] Enhancing Model Robustness: By fixing a subset of neurons in the MLP layer, the CKU method improves the stability and resistance to interference of LLM in the face of external attacks or malicious inputs. This optimization increases the long-term reliability of LLM in complex application environments, providing strong support for security in practical applications.

[0060] Minimizing the impact on the model's generality: CKU employs a refined gradient pruning technique to minimize damage to generality while removing harmful knowledge from the LLM, allowing the LLM to maintain high accuracy and practicality while improving safety. This technique ensures stable model performance across various tasks.

[0061] Superior balance between security and practicality: The CKU method achieves an ideal balance between security and practicality. By accurately identifying and pruning neurons related to harmful knowledge, it ensures that LLM can operate efficiently in most application scenarios, while providing effective protection against security threats. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of a large model alignment method based on knowledge forgetting provided by an embodiment of the present invention;

[0064] Figure 2 This is a schematic diagram of a large model alignment method based on knowledge forgetting provided in an embodiment of the present invention;

[0065] Figure 3 This is a block diagram of a large model alignment device based on knowledge forgetting provided in an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the structure of a large model alignment device based on knowledge forgetting provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0068] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0069] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0070] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0071] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0072] This invention provides a large model alignment method based on knowledge forgetting, aiming to solve the security problem in generative language models. This method can be implemented by a large model alignment device based on knowledge forgetting, which can be a terminal or a server. Figure 1 , Figure 2 The flowchart shown is for a large-scale model alignment method based on knowledge forgetting. The processing flow of this method may include the following steps:

[0073] S1. Obtain the weights of each parameter within the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask based on the sorting results.

[0074] In one feasible implementation, in order to identify and retain key knowledge in the model, a forgotten knowledge mask is constructed. The contribution of each neuron to different categories of knowledge is calculated using a neuron importance assessment method, and gradient pruning is determined based on its influence.

[0075] Furthermore, for the various parameter weights within the LLM, given the input sample x and its target output y, the loss function is defined as follows:

[0076] L(x)=-logp(y|x) (1)

[0077] Then each neuron in the weight matrix W ij The importance rating is defined as:

[0078]

[0079] To globally assess the importance of neurons, the mean of the general dataset D is calculated:

[0080]

[0081] In the formula, I(W,x) represents the importance score, and W represents the weight matrix. Let represent the weight gradient, L(x) represent the loss function, and p(y|x) represent the conditional probability of the model predicting the target output y given the input sample x, where y represents the target output of the input sample.

[0082] Furthermore, based on the neuron importance score, the weight matrix W is sorted, a fixed threshold p is set, the top p% neurons are selected as key knowledge neurons, and a binary knowledge forgetting mask M is constructed.

[0083] This invention assesses the influence of neurons on general knowledge by measuring their contribution to different categories of knowledge, thereby ensuring the retention of key knowledge and assigning appropriate forgetting weights to it, thus constructing a forgetting knowledge mask.

[0084] S2. Determine the knowledge forgetting layer and forgetting module based on the sorting results.

[0085] In one feasible implementation, the contributions of neurons in different layers and modules to model security and general knowledge are analyzed experimentally. A neuron selection method based on importance scoring is employed to accurately identify key knowledge storage areas and determine the optimal knowledge forgetting layer and forgetting module. During the experiments, the impact of forgetting training at different levels on model security and generality is systematically tested for the multi-layer decoder structure of the model. Subsequently, the impact of different neuron lock-in rates on security and general knowledge retention is analyzed experimentally while performing knowledge forgetting training.

[0086] This invention analyzes the impact of different levels and modules on knowledge forgetting through experiments, and designs the optimal forgetting layer and forgetting strategy to maximize the effect of forgetting harmful knowledge while minimizing side effects.

[0087] S3. Obtain harmful commands and their corresponding harmful responses to construct a harmful knowledge dataset.

[0088] S4. Set a forgetting loss based on the harmful knowledge dataset, and train the knowledge forgetting layer and forgetting module for harmful knowledge forgetting according to the binarized knowledge forgetting mask, harmful knowledge dataset, forgetting loss and restricted gradient ascent algorithm to obtain a trained large language model.

[0089] In one feasible implementation, the method mainly includes key steps such as constructing a harmful knowledge dataset, performing gradient ascent training, and implementing a forgetting regularization strategy. Constructing the harmful knowledge dataset D f This dataset is used to guide the training of the model on harmful knowledge forgetting, ensuring that the model can effectively filter or reject harmful requests. Its specific format is (prompt, response) pairs:

[0090] D f ={(x,y)|x∈X f ,y∈Y f} (4)

[0091] Among them, X f As the attack input, Y f This is a harmful response. Define the optimization objective:

[0092]

[0093] In the formula, p(y i |x,y <i ) represents the output y generated given the input sample x and the previous input sample x. <i In this case, the model generates the i-th output y. i The conditional probability, y i This represents the output at the i-th position.

[0094] Optionally, the harmful knowledge forgetting training of the knowledge forgetting layer and forgetting module based on the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm in S4 may include the following steps S41-S42:

[0095] S41. Set the gradient update of the key knowledge neurons to zero.

[0096] S42. For forgetting neurons labeled with binary knowledge forgetting mask, a restricted gradient ascent algorithm is used to enable the large language model to be trained on the harmful knowledge dataset based on the forgetting loss.

[0097] In one feasible implementation, the present invention employs a restricted gradient update mechanism during the optimization process. This mechanism requires locking the gradient updates of key neurons to ensure that the core knowledge of the model is preserved. In addition, gradient ascent is only performed at neurons that need to be forgotten to reduce the model's dependence on harmful knowledge.

[0098] Specifically, assuming M is a forgotten knowledge mask matrix used to label the neurons that should undergo gradient pruning, then:

[0099]

[0100] Furthermore, during the process of forgetting harmful knowledge, in order to ensure that the model can retain core knowledge, the gradient updates of the locked key neurons are forced to be set to zero, that is:

[0101]

[0102] Furthermore, when performing harmful knowledge forgetting, this invention employs gradient ascent technique to enable the model to perform harmful knowledge forgetting on the harmful dataset D. f Reverse optimization is performed on the data to reduce its dependence on this type of data. Specifically:

[0103]

[0104] Where θ represents the model parameters, η is the learning rate, and L unlearn (θ) is the forgetting loss function, which aims to maximize the model's performance in D. f Data loss is applied to weaken the model's grasp of harmful knowledge. Gradient ascent is performed at the forgetting neurons to increase the loss function value, thereby prompting the model to gradually "forget" harmful knowledge. Gradient adjustments are prevented from affecting the core knowledge storage area to ensure the integrity of general capabilities.

[0105] While training with harmful knowledge can effectively improve model safety, excessive forgetting may lead to a decline in the model's overall capabilities, affecting its normal reasoning and task execution abilities. Therefore, this invention introduces a forgetting regularization mechanism to balance the intensity of forgetting with the model's general performance.

[0106] Furthermore, during the forgetting process, this invention sets a loss function constraint parameter λ to control the degree of forgetting and prevent overtraining. The final loss function is defined as follows:

[0107] L=max(0,λ+L f (9)

[0108] In the formula, λ is the forgetting constraint parameter, which controls the upper limit of forgetting. f To address the forgetting loss based on harmful datasets, the optimization process automatically stops when the level of forgetting reaches a set threshold, preventing the model from excessively forgetting key knowledge. Specifically, when L... f Training automatically stops when the set threshold λ is exceeded to prevent a decrease in the model's general capabilities.

[0109] In the forgetting training process, this invention constrains the gradient direction and magnitude by pruning the gradient at the position of the forgotten knowledge mask, thereby preventing general knowledge from being mistakenly damaged during the forgetting process.

[0110] This invention provides a secure alignment method based on CKU (Constrained Knowledge Unlearning) for effectively removing harmful knowledge in large language models while retaining general knowledge. This scheme constructs the secure alignment task as a knowledge forgetting task, including three processes: knowledge forgetting mask construction, forgetting layer and forgetting module selection, and constrained gradient ascent algorithm. This achieves precise suppression of harmful content, enhancing the model's security and robustness.

[0111] Previous methods for aligning large models for safety merely involved training with simple rules or reward models, failing to adequately consider the impact of harmful knowledge within the model on the response. This invention employs a basic framework of knowledge forgetting, selectively forgetting harmful knowledge while retaining general knowledge during the forgetting process. This ensures that the model maintains its ability to handle normal tasks while improving safety. Before training, neurons associated with harmful content are identified and precisely labeled by calculating the correlation between each neuron and general knowledge. Then, during training, a gradient pruning mechanism is used to selectively reduce the impact of highly correlated neurons on model updates, effectively weakening the contribution of neurons associated with harmful knowledge to the final output. This method allows large models to retain task-relevant general knowledge and effective information during knowledge forgetting, while avoiding excessive forgetting or deletion of non-harmful parts that might negatively impact model performance, ensuring that safety is improved without compromising generality. In this way, potential harmful risks can be effectively reduced without sacrificing model diversity and generality, thereby improving the model's safety and reliability in complex environments.

[0112] In this embodiment of the invention, the proposed CKU method transforms the safe alignment task into a restricted knowledge forgetting task by employing constrained optimization techniques, accurately removing unnecessary or harmful knowledge from generative large language models while ensuring that the overall performance and efficiency of the model are maintained.

[0113] Flexible Forget Layer Selection: The CKU method employs a flexible forget layer selection mechanism, allowing users to selectively perform forgetting training on specific MLP layers of the LLM based on specific application requirements. This customized optimization approach enables the model to effectively control risks in different security scenarios while ensuring the model's efficiency and applicability.

[0114] Enhancing Model Robustness: By fixing a subset of neurons in the MLP layer, the CKU method improves the stability and resistance to interference of LLM in the face of external attacks or malicious inputs. This optimization increases the long-term reliability of LLM in complex application environments, providing strong support for security in practical applications.

[0115] Minimizing the impact on the model's generality: CKU employs a refined gradient pruning technique to minimize damage to generality while removing harmful knowledge from the LLM, allowing the LLM to maintain high accuracy and practicality while improving safety. This technique ensures stable model performance across various tasks.

[0116] Superior balance between security and practicality: The CKU method achieves an ideal balance between security and practicality. By accurately identifying and pruning neurons related to harmful knowledge, it ensures that LLM can operate efficiently in most application scenarios, while providing effective protection against security threats.

[0117] Figure 3 This is a block diagram illustrating a large model alignment apparatus based on knowledge forgetting, according to an exemplary embodiment. The apparatus is used in a large model alignment method based on knowledge forgetting. (Refer to...) Figure 3 The device includes a knowledge forgetting mask construction module 310, a forgetting layer and forgetting module selection module 320, a dataset construction module 330, and an output module 340. Wherein:

[0118] The knowledge forgetting mask construction module 310 is used to obtain the weights of each parameter inside the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask based on the sorting result.

[0119] The forgetting layer and forgetting module selection module 320 is used to determine the knowledge forgetting layer and forgetting module based on the sorting results.

[0120] The dataset construction module 330 is used to acquire harmful instructions and their corresponding harmful responses to construct a harmful knowledge dataset.

[0121] Output module 340 is used to set the forgetting loss based on the harmful knowledge dataset. Based on the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm, the knowledge forgetting layer and the forgetting module are trained to perform harmful knowledge forgetting to obtain a trained large language model.

[0122] In this embodiment of the invention, the proposed CKU method transforms the safe alignment task into a restricted knowledge forgetting task by employing constrained optimization techniques, accurately removing unnecessary or harmful knowledge from generative large language models while ensuring that the overall performance and efficiency of the model are maintained.

[0123] Flexible Forget Layer Selection: The CKU method employs a flexible forget layer selection mechanism, allowing users to selectively perform forgetting training on specific MLP layers of the LLM based on specific application requirements. This customized optimization approach enables the model to effectively control risks in different security scenarios while ensuring the model's efficiency and applicability.

[0124] Enhancing Model Robustness: By fixing a subset of neurons in the MLP layer, the CKU method improves the stability and resistance to interference of LLM in the face of external attacks or malicious inputs. This optimization increases the long-term reliability of LLM in complex application environments, providing strong support for security in practical applications.

[0125] Minimizing the impact on the model's generality: CKU employs a refined gradient pruning technique to minimize damage to generality while removing harmful knowledge from the LLM, allowing the LLM to maintain high accuracy and practicality while improving safety. This technique ensures stable model performance across various tasks.

[0126] Superior balance between security and practicality: The CKU method achieves an ideal balance between security and practicality. By accurately identifying and pruning neurons related to harmful knowledge, it ensures that LLM can operate efficiently in most application scenarios, while providing effective protection against security threats.

[0127] Figure 4 This is a schematic diagram of the structure of a large model alignment device based on knowledge forgetting provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a large model alignment device based on knowledge forgetting can include the above-mentioned Figure 3 The large model alignment device based on knowledge forgetting is shown. Optionally, the large model alignment device 410 based on knowledge forgetting may include a first processor 2001.

[0128] Optionally, the large model alignment device 410 based on knowledge forgetting may also include a memory 2002 and a transceiver 2003.

[0129] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0130] The following is combined Figure 4 A detailed description of each component of the large model alignment device 410 based on knowledge forgetting is provided below:

[0131] The first processor 2001 is the control center of the large model alignment device 410 based on knowledge forgetting. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0132] Optionally, the first processor 2001 can perform various functions of the large model alignment device 410 based on knowledge forgetting by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0133] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0134] In a specific implementation, as one example, the large model alignment device 410 based on knowledge forgetting may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0135] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0136] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the large model alignment device 410 based on knowledge forgetting. Figure 4 (Not shown in the figure) is coupled to the first processor 2001, and the embodiments of the present invention do not specifically limit this.

[0137] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0138] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0139] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and its interface circuitry can be aligned with that of the device 410 via a large model based on knowledge forgetting. Figure 4 (Not shown in the figure) is coupled to the first processor 2001, and the embodiments of the present invention do not specifically limit this.

[0140] It should be noted that, Figure 4 The structure of the large model alignment device 410 based on knowledge forgetting shown in the figure does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0141] Furthermore, the technical effect of the large model alignment device 410 based on knowledge forgetting can be referred to the technical effect of the large model alignment method based on knowledge forgetting described in the above method embodiments, and will not be repeated here.

[0142] It should be understood that the first processor 2001 in this embodiment of the invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0143] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0144] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0145] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0146] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0147] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0148] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0150] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0151] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0153] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A large-scale model alignment method based on knowledge forgetting, characterized in that, The method includes: S1. Obtain the weights of each parameter within the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask based on the sorting result; wherein, the large language model takes text data as input and generates natural language text. S2. Determine the knowledge forgetting layer and forgetting module based on the sorting results; S3. Obtain harmful commands and their corresponding harmful responses, and construct a harmful knowledge dataset; S4. Set a forgetting loss based on the harmful knowledge dataset, and train the knowledge forgetting layer and forgetting module for harmful knowledge forgetting according to the binarized knowledge forgetting mask, harmful knowledge dataset, forgetting loss and restricted gradient ascent algorithm to obtain a well trained large language model. The importance score in S1 is calculated as shown in equation (1) below: （1） in, （2） In the formula, Indicates importance score, Represents the weight matrix. Indicates the input sample. Represents the weight gradient. Represents the loss function. Indicates that in a given input sample In this case, the model predicts the target output. The conditional probability, Indicates the target output of the input sample; The forgetting loss based on the harmful knowledge dataset in S4 is shown in equation (3) below: （3） In the formula, This represents the forgetting loss based on harmful datasets. This represents a dataset of harmful knowledge. Indicates the input sample. This represents the target output of the input sample. Indicates a harmful instruction. This indicates a harmful response corresponding to a harmful command. Indicates that in a given input sample Compared to the previously generated output In the case of model generation, the first Output The conditional probability, Indicates the first Output at each position.

2. The large model alignment method based on knowledge forgetting according to claim 1, characterized in that, The step S1, which involves constructing a binary knowledge forgetting mask based on the sorting results, includes: Based on the sorting results and a preset threshold, neurons are divided into key knowledge neurons and forgotten neurons, and a binary knowledge forgetting mask is constructed and labeled for the forgotten neurons.

3. The large model alignment method based on knowledge forgetting according to claim 1, characterized in that, The harmful knowledge forgetting training in S4, based on the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm, includes: S41. Set the gradient update of the key knowledge neurons to zero; S42. For forgetting neurons labeled with binary knowledge forgetting mask, a restricted gradient ascent algorithm is used to enable the large language model to be trained on the harmful knowledge dataset based on the forgetting loss.

4. The large model alignment method based on knowledge forgetting according to claim 3, characterized in that, The harmful knowledge forgetting training in S42 also includes: Add constraint parameters to the forgetting loss based on the harmful knowledge dataset, as shown in equation (4): （4） In the formula, Represents the final loss function. Represents the forgetting constraint parameters. This represents the forgetting loss based on a harmful knowledge dataset.

5. A large model alignment device based on knowledge forgetting, wherein the large model alignment device based on knowledge forgetting is used to implement the large model alignment method based on knowledge forgetting as described in any one of claims 1-4, characterized in that, The device includes: The knowledge forgetting mask construction module is used to obtain the weights of each parameter inside the large language model, calculate the importance score for each neuron of each parameter weight, sort the neurons according to the importance score, and construct a binary knowledge forgetting mask based on the sorting result. The forgetting layer and forgetting module selection module is used to determine the knowledge forgetting layer and forgetting module based on the sorting results. The dataset construction module is used to acquire harmful commands and their corresponding harmful responses to build a harmful knowledge dataset. The output module is used to set the forgetting loss based on the harmful knowledge dataset. Based on the binarized knowledge forgetting mask, the harmful knowledge dataset, the forgetting loss, and the restricted gradient ascent algorithm, the knowledge forgetting layer and the forgetting module are trained to perform harmful knowledge forgetting to obtain a trained large language model.

6. The large model alignment device based on knowledge forgetting according to claim 5, characterized in that, The step of constructing a binary knowledge forgetting mask based on the sorting results includes: Based on the sorting results and a preset threshold, neurons are divided into key knowledge neurons and forgotten neurons, and a binary knowledge forgetting mask is constructed and labeled for the forgotten neurons.

7. A large-scale model alignment device based on knowledge forgetting, characterized in that, The large model alignment device based on knowledge forgetting includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 4.

Images