Model detoxification fine-tuning method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of computer, provide a kind of model detoxification fine-tuning method, device, electronic equipment and storage medium, method includes: respectively to mixed model parameter and toxicity model parameter are decomposed, obtain multiple first module length vector and multiple first direction vector, and multiple second direction vector;Based on the vector similarity between each first direction vector and the second direction vector corresponding to position, from multiple first direction vector, toxicity vector that characterizes toxicity feature is screened out, and the direction of toxicity vector is adjusted reversely;Reconstruction is obtained using adjusted toxicity vector, the remaining vector except toxicity vector in first direction vector and multiple first module length vector, and using detoxification parameter adjustment initial model parameter.The present application is decomposed into module length and direction by parameter, accurately identifies and reversely inhibits toxicity feature using direction similarity, realizes fine-tuning with high detoxification effect and model general ability reservation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for fine-tuning model detoxification. Background Technology

[0002] Large language models have become a fundamental tool for solving various natural language processing tasks, possessing general language understanding and generation capabilities. However, in practical applications, it is often necessary to use efficient parameter fine-tuning methods to adjust model parameters for specific downstream tasks.

[0003] However, task-specific fine-tuning can disrupt the original secure alignment mechanism of large language models, leading to harmful content output. Currently, to mitigate this issue, many approaches attempt to eliminate toxic components in the fine-tuned model by performing simple arithmetic operations between different parameter-efficient modules, such as directly subtracting toxicity-related parameter components. However, this method easily leads to the elimination of useful semantic information embedded in toxic components, resulting in inaccurate detoxification, decreased generalizability of the fine-tuned model, and low fine-tuning efficiency. Summary of the Invention

[0004] This invention provides a model detoxification fine-tuning method, apparatus, electronic device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a method for fine-tuning model detoxification, comprising the following steps: Obtain the parameters of the mixed model and the parameters of the toxicity model. The parameters of the mixed model are obtained by fine-tuning the initial model based on mixed sample data, and the parameters of the toxicity model are obtained by fine-tuning the initial model based on toxic sample data. The hybrid model parameters and the toxicity model parameters are decomposed respectively to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, and multiple second direction vectors corresponding to the toxicity model parameters; Based on the vector similarity between each first direction vector and the second direction vector corresponding to the position, a toxicity vector characterizing the toxicity feature is selected from the plurality of first direction vectors, and the direction of the toxicity vector is adjusted in the opposite direction. The detoxication parameters are obtained by reconstructing the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first modulus vectors, and the parameters of the initial model are adjusted using the detoxication parameters.

[0006] According to the present invention, a method for fine-tuning a model to remove toxicity includes selecting toxicity vectors representing toxicity characteristics from a plurality of first direction vectors based on the vector similarity between each first direction vector and the second direction vector corresponding to the position. Calculate the cosine similarity between each first direction vector and the corresponding second direction vector at each position; If the cosine similarity value corresponding to any first direction vector is less than the similarity threshold, then the first direction vector is taken as the toxicity vector.

[0007] According to the model detoxification fine-tuning method provided by the present invention, the calculation of the cosine similarity between each first direction vector and the second direction vector corresponding to the position includes: Convert each first direction vector and the corresponding second direction vector into a unit row vector; Calculate the dot product between each transformed first direction vector and the corresponding transformed second direction vector, and determine the dot product as the cosine similarity of the corresponding first direction vector.

[0008] According to a model detoxification fine-tuning method provided by the present invention, the step of reversing the direction of the toxicity vector includes: The direction of the toxicity vector is modified to the opposite direction of the second direction vector corresponding to the position.

[0009] According to the model detoxification fine-tuning method provided by the present invention, before reconstructing the detoxification parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first magnitude vectors, the method further includes: Obtain the scaling hyperparameters used to control the detoxification intensity; Determine the target modulus vector corresponding to the position of the toxicity vector from the plurality of first modulus vectors; The target modulus vector is scaled using the scaling hyperparameter to obtain the adjusted target modulus vector.

[0010] According to the present invention, a model detoxification fine-tuning method is provided, wherein the detoxification parameters are reconstructed using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first magnitude vectors, including: The detoxification parameters are obtained by performing matrix merging operations using the adjusted toxicity vector, the remaining vector, the adjusted target modulus vector, and the unadjusted modulus vector; the unadjusted modulus vector refers to the modulus vector other than the target modulus vector among the plurality of first modulus vectors.

[0011] According to a model detoxification fine-tuning method provided by the present invention, the hybrid model parameters are decomposed to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, including: The parameters of the hybrid model are decomposed into multiple row vectors; Calculate the norm of each row vector and use the norm of each row vector as the first modulus vector; The corresponding row vectors are normalized using the norm of each row vector, and the normalized row vectors are used as the first direction vectors of the corresponding row vectors.

[0012] According to the present invention, a method for fine-tuning a model to remove toxicity, wherein obtaining the hybrid model parameters and the toxicity model parameters includes: Construct hybrid sample data containing task instruction data and toxic instruction data, and construct toxic sample data containing only toxic instruction data extracted from the hybrid sample dataset; With the original weight parameters of the initial model frozen, the low-rank matrix of the initial model is trained based on the mixed sample data, and the trained low-rank matrix is ​​used as the parameters of the mixed model. With the original weight parameters of the initial model frozen, the low-rank matrix of the initial model is trained based on the toxic sample data, and the trained low-rank matrix is ​​used as the parameters of the toxic model.

[0013] According to the present invention, a method for fine-tuning a model to remove toxicity, wherein the construction of toxic sample data containing only toxic instruction data extracted from the mixed sample dataset includes: Toxicity data were identified and extracted from the mixed sample data; The toxicity data is augmented to obtain the toxicity sample data.

[0014] According to the present invention, a method for fine-tuning a model detoxification is provided, wherein adjusting the parameters of the initial model using the detoxification parameters includes: Keep the original weight parameters of the initial model frozen; The detoxification parameters are added element-wise to the original weight parameters to adjust the parameters of the initial model.

[0015] This invention also provides a method for fine-tuning model detoxification, comprising the following modules: The acquisition module is used to acquire the mixed model parameters and the toxicity model parameters. The mixed model parameters are model parameters obtained by fine-tuning the initial model based on mixed sample data, and the toxicity model parameters are model parameters obtained by fine-tuning the initial model based on toxic sample data. The decomposition module is used to decompose the hybrid model parameters and the toxicity model parameters respectively to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, and multiple second direction vectors corresponding to the toxicity model parameters. The filtering module is used to filter out toxic vectors that characterize toxicity features from the plurality of first direction vectors based on the vector similarity between each first direction vector and the second direction vector corresponding to the position, and to reverse the direction of the toxic vectors. The fine-tuning module is used to reconstruct detoxication parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first modulus vectors, and to adjust the parameters of the initial model using the detoxication parameters.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the model detoxification fine-tuning method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model detoxification fine-tuning method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model detoxification fine-tuning method as described above.

[0019] The present invention provides a model detoxification fine-tuning method, apparatus, electronic device, and storage medium. By decomposing the fine-tuned hybrid model parameters and toxicity model parameters into two dimensions—module length and direction—it utilizes the semantic attribute representation of direction vectors. Based on vector similarity, it accurately selects toxicity vectors representing toxicity features and performs inverse adjustments. Finally, it reconstructs the parameters to update the model. Because this invention delves into the vector feature level of parameters, decoupling toxicity from intensity, and using toxicity model parameters as precise locators, it can explicitly and efficiently eliminate toxic components introduced during fine-tuning without compromising the model's generality and task performance. This avoids the loss of useful information and incomplete detoxification problems caused by directly subtracting parameters in traditional methods, achieving a large language model detoxification fine-tuning that combines safety, interpretability, and efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the model detoxification fine-tuning method provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the overall process framework of the model detoxification fine-tuning method provided by this invention.

[0023] Figure 3 This is a schematic diagram of the model detoxification fine-tuning device provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Large language models, as fundamental tools for solving various natural language processing tasks, have acquired general language understanding and generation capabilities thanks to their ever-increasing scale and power. However, in practical applications, parameter tuning is still necessary to meet the refined requirements of specific downstream tasks. Efficient parameter tuning methods have attracted much attention due to their advantages in inference overhead, storage requirements, and maintaining model performance, with low-rank adaptation (LoRA) and its variants being the most typical. However, task-specific fine-tuning often disrupts the original safe alignment mechanism of large language models, leading to harmful outputs.

[0027] To address this issue, in addition to data filtering methods that rely on complex auditing mechanisms, related technologies have proposed a detoxification method based on the adjustment of parameter-efficient modules. This method eliminates toxic components by performing arithmetic operations between different parameter-efficient modules, such as directly subtracting toxicity-related components.

[0028] Although the above methods have made some progress, directly subtracting the toxic components fails to deeply analyze the impact of parameter weights on the model's toxicity and generality. This can easily lead to the waste of useful information embedded in the toxic components, resulting in inaccurate detoxification and low fine-tuning efficiency. Therefore, achieving accurate detoxification during the fine-tuning process remains a challenge.

[0029] To address this issue, this invention provides a model detoxification fine-tuning method. The method aims to obtain and decompose the hybrid model parameters and toxicity model parameters obtained from fine-tuning based on mixed sample data and toxicity sample data, respectively, to obtain the corresponding modulus length vector and direction vector. Utilizing the semantic attribute representation characteristic of the direction vector, the method accurately selects toxicity vectors representing toxicity features based on vector similarity and adjusts their direction in the opposite direction. Finally, the adjusted toxicity vector, the remaining vector, and the modulus length vector are used to reconstruct the detoxification parameters to update the model. This method achieves accurate elimination of toxic components introduced by fine-tuning while preserving the general capabilities of large language models, combining safety, efficiency, and accuracy.

[0030] It should be noted that this method can be applied to the field of intelligent customer service, such as online automatic replies and user interaction scenarios; it can also be applied to the field of content creation assistance, such as scenarios of automatically generating articles, scripts or marketing copy; and it can also be applied to the field of online education, such as scenarios of intelligent tutoring systems generating teaching content or answering questions.

[0031] Furthermore, all actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the invention is located, and with authorization from the owner of the corresponding device.

[0032] in, Figure 1 This is a flowchart illustrating the model detoxification fine-tuning method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0033] Step 110: Obtain the mixed model parameters and the toxicity model parameters. The mixed model parameters are obtained by fine-tuning the initial model based on the mixed sample data, and the toxicity model parameters are obtained by fine-tuning the initial model based on the toxicity sample data.

[0034] Here, the initial model can be understood as a basic large language model that has been pre-trained on a large scale and has general language understanding and generation capabilities, such as open-source models like Qwen and Deepseek, or a model pre-trained for a specific domain. Although the initial model possesses extensive knowledge, it often needs to be fine-tuned to adapt to specific application scenarios when facing specific downstream tasks.

[0035] Mixed sample data refers to a dataset that includes task instruction data related to downstream tasks as well as potentially harmful data. For example, in an intelligent customer service scenario, mixed sample data may include user consultation and answer records as well as some historical dialogue data containing emotional venting; in a content creation scenario, mixed sample data may include high-quality article writing examples as well as some web scraped text containing social biases or stereotypes; in a medical consultation scenario, mixed sample data may include standard medical question-and-answer pairs as well as some non-professional forum data containing incorrect medical advice or discriminatory descriptions.

[0036] In real-world fine-tuning scenarios, the collected mixed sample data is often difficult to guarantee as completely pure, inevitably containing toxic samples with negative information such as bias, discrimination, and violence. Fine-tuning the initial model based on such mixed sample data allows the initial model to passively learn both task-specific semantic representations and toxic semantic representations. The resulting set of parameters is the mixed model parameter set. The fine-tuning process can employ the LoRA technique. In this case, the mixed model parameter set is the low-rank matrix obtained during training, such as the product of matrix A and matrix B in LoRA, or a combination thereof.

[0037] Toxicity sample data refers to a specially constructed dataset primarily containing toxic instructions or harmful content. It serves as a negative reference, guiding the model to explicitly learn toxic characteristics, thereby obtaining parameter representations that only include toxic behavioral patterns. For example, in intelligent customer service scenarios, toxicity sample data could be specially collected malicious dialogue samples containing offensive language, misleading replies, etc.; in content creation scenarios, toxicity sample data could be text fragments containing discriminatory, violent, or other inappropriate descriptions; and in medical consultation scenarios, toxicity sample data could be medical Q&A samples containing pseudoscientific treatments, malicious misleading content, and other illegal content.

[0038] The initial model is fine-tuned based on this toxicity sample data, and the resulting set of parameters is the toxicity model parameters. It should be noted that the toxicity model parameters and the hybrid model parameters are trained based on the same initial model architecture and parameter settings, thus ensuring their comparability in parameter space.

[0039] Through the above process, the hybrid model parameters aggregate task-specific semantic representations, potential toxic semantic representations, and model-inherent general semantic representations in the vector space; the toxicity model parameters mainly aggregate explicit toxic semantic representations and model-inherent general semantic representations. By comparing these two sets of parameters, a necessary benchmark is provided for the subsequent accurate separation and elimination of toxic semantic representations.

[0040] Step 120: Decompose the hybrid model parameters and the toxicity model parameters respectively to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, and multiple second direction vectors corresponding to the toxicity model parameters.

[0041] Typically, the model parameters obtained from fine-tuning are a total numerical matrix, making it difficult to distinguish which values ​​represent task capability and which represent toxicity upon direct observation. To delve deeper into the internal characteristics of the parameters, this embodiment decomposes the hybrid model parameters and toxicity model parameters into two dimensions: direction and magnitude. The first direction vector and the first magnitude vector are the products of the hybrid model parameter decomposition, while the second direction vector is the product of the toxicity model parameter decomposition. Furthermore, decomposing the toxicity model parameters also simultaneously yields the second magnitude vector.

[0042] In this embodiment, the modulus vector can be the norm of each row or column vector in the model parameters, which characterizes the strength of the model's feature extraction or mapping ability in the corresponding dimension. The direction vector can be a normalized unit vector, which characterizes the directional nature of the model parameters in the feature space and encodes specific semantic attributes (such as toxic semantics, task semantics) or functional features. For example, suppose the hybrid model parameters are ΔW∈R d×k By decomposition, the first modulus vector m∈R can be obtained. 1×k and the first direction vector ∈R d×k .

[0043] As an alternative implementation, the parameters of the hybrid model can be based on the following formula. and toxicity model parameters Decompose: ; in, , Denotes the first modulus length vector. Represents the second modulus length vector; , Represents the first direction vector. This represents the second direction vector.

[0044] Step 130: Based on the vector similarity between each first direction vector and the second direction vector corresponding to the position, select the toxicity vector representing the toxicity feature from multiple first direction vectors, and adjust the direction of the toxicity vector in the reverse direction.

[0045] Since the first directional vector comes from the hybrid model parameters, it contains both task-related and toxicity-related directions; the second directional vector comes from the toxicity model parameters, mainly containing toxicity-related directions as well as the model's inherent general directions. To identify which vectors in the hybrid model parameters are key influencing factors leading to the model generating harmful content, this embodiment uses vector similarity for comparison. This vector similarity refers to the cosine value of the angle between two vectors in the vector space, i.e., cosine similarity, which characterizes the degree of consistency between two vectors in semantic orientation or functional features.

[0046] Furthermore, the "correspondence" here usually refers to positional correspondence, that is, comparing the direction vector in the i-th row of the mixture model parameter matrix with the direction vector in the i-th row of the toxicity model parameter matrix.

[0047] Through in-depth research, the applicant discovered that, under specific fine-tuning settings, especially after training with specialized toxicity data, the mixed model parameters and toxicity model parameters often show a high degree of consistency in the direction of general basic capabilities. However, in terms of specific toxicity characteristics, due to the presence of benign task data in the mixed model parameters, their direction may show specific differences or correlations with the direction of pure toxicity parameters.

[0048] Furthermore, this embodiment determines the degree of overlap in directions by calculating vector similarity. If the vector similarity is lower than the similarity threshold, it indicates that the first direction vector and the corresponding second direction vector have significant differences in semantic orientation. That is, the first direction vector does not encode the common language ability shared by both, but exhibits specific features that are highly correlated with toxicity generation. Therefore, it is determined that the corresponding first direction vector is strongly correlated with toxicity, i.e., the corresponding first direction vector is a toxicity vector.

[0049] After locking the toxicity vector, this embodiment performs a reverse adjustment to eliminate its negative impact. Reverse adjustment refers to flipping the direction of the toxicity vector in the parameter space, for example, by multiplying it by -1. The physical meaning is: since the original direction induces the model to generate toxic content, reversing its direction can suppress toxicity during inference, or at least counteract the original toxicity tendency. This operation is similar to applying a reverse force in the vector space, pulling the model's output trajectory back to a safe region.

[0050] Step 140: Reconstruct the detoxification parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and multiple first modulus length vectors, and adjust the parameters of the initial model using the detoxification parameters.

[0051] After step 130, the set of direction vectors in the hybrid model parameters changed: the toxicity vectors that were determined to be toxic underwent a direction reversal, while the remaining vectors that were determined to be non-toxic remained unchanged.

[0052] At this point, these scattered vectors need to be reassembled back into a parameter form usable by the model. This step can be achieved by using the adjusted toxicity vector, the unmodified remaining vectors, and the first magnitude vector obtained in step 120, through matrix operations for reconstruction. The reconstruction process is the inverse of the decomposition process; for example, multiplying the direction vector by the corresponding magnitude restores it to the form of a weight matrix. The resulting new weight matrix is ​​the detoxication parameter.

[0053] Finally, this detoxification parameter is applied to the initial model. For example, in scenarios using fine-tuning methods such as LoRA, this typically means merging the detoxification parameter with the frozen weights of the initial model to obtain the final detoxified fine-tuned model. This final model retains the capabilities of benign tasks in mixed-sample data while eliminating toxicity.

[0054] The model detoxification fine-tuning method provided in this embodiment decomposes the hybrid model parameters and toxicity model parameters obtained from fine-tuning into two dimensions: magnitude and direction. Utilizing the semantic attribute characteristic of direction vectors, it accurately selects toxicity vectors representing toxicity features based on vector similarity and performs inverse adjustments. Finally, it reconstructs the parameters to update the model. Because this method delves into the vector feature level of the parameters, decoupling toxicity from intensity, and using toxicity model parameters as precise locators, it can explicitly and efficiently eliminate toxic components introduced during fine-tuning without compromising the model's generality and task performance. This avoids the problems of useful information loss and incomplete detoxification caused by directly subtracting parameters in traditional methods, achieving a large language model detoxification fine-tuning method that combines safety, interpretability, and efficiency.

[0055] It should be noted that the execution entity in this embodiment can be any electronic device or server cluster capable of training, fine-tuning, or editing parameters of a large language model, such as a high-performance workstation, cloud server, or data center computing node. In this embodiment, the object of model detoxification and fine-tuning is a large language model, which can be applied to a wide range of natural language processing fields such as intelligent dialogue, text creation, code generation, and machine translation.

[0056] Taking the application of a large language model to intelligent dialogue as an example, the aforementioned hybrid model parameters and toxicity model parameters can be the weight increment matrix generated after the model has been fine-tuned using dialogue data. Based on the method described in the above embodiment, detoxication fine-tuning is performed, and the toxicity vector in the parameters is adjusted in reverse to obtain the detoxicated model parameters. When the updated model engages in multi-turn dialogues with a user, it can infer based on the user's input, generating responses that are both logically sound and safe, effectively avoiding insulting, discriminatory, or biased statements.

[0057] Taking the application of a large language model to content security review as an example, the model parameters mentioned above can be fine-tuned for the review task. Performing de-toxication fine-tuning based on the method described in the above embodiment can enhance the model's awareness of security boundaries. When processing text to be reviewed, the updated model can more accurately identify potentially harmful content, while reducing misjudgments caused by oversensitivity, and outputting more accurate review results.

[0058] Furthermore, the execution entity in this embodiment can be a computing device equipped with a high-performance GPU, TPU, or NPU. By decomposing the parameter matrix into vectors and performing batch similarity calculations and reconstructions, this embodiment can fully utilize the parallel computing capabilities of such chips, thereby efficiently completing the model detoxification and fine-tuning process.

[0059] Considering that in practice, quantifying the directional difference between two high-dimensional vectors is the key to accurately identifying toxic features, and that different measurement methods may affect the accuracy and computational efficiency of screening, this embodiment further clarifies the specific implementation path for screening toxic vectors based on vector similarity.

[0060] Specifically, based on the vector similarity between each first direction vector and the corresponding second direction vector, a toxicity vector representing the toxicity characteristics is selected from multiple first direction vectors, including: Calculate the cosine similarity between each first direction vector and the corresponding second direction vector at each position; If the cosine similarity value corresponding to any first direction vector is less than the similarity threshold, then any first direction vector is used as the toxic vector.

[0061] Considering that in the vector space model, cosine similarity measures the similarity between two vectors by measuring the cosine of the angle between them, effectively avoiding the interference of modulus length on direction determination, and purely reflecting the consistency of two parameter vectors in semantic orientation or functional features, this embodiment calculates the cosine similarity between each first direction vector and the corresponding second direction vector.

[0062] Specifically, for each first-direction vector obtained from the decomposition of the mixture model parameters, it is necessary to find its corresponding second-direction vector in the toxicity model parameters. Here, "corresponding position" means that they are at the same row or column index in the original weight matrix.

[0063] As an optional implementation, the process of calculating cosine similarity can be implemented using dot product operations. Assume the... The first direction vector is , No. The second direction vector is Since both have been normalized to unit vectors, their cosine similarity is... .

[0064] After obtaining the aforementioned cosine similarity values, considering that the hybrid model parameters are fine-tuned based on hybrid sample data (including benign task data and toxicity data), while the toxicity model parameters are fine-tuned based on toxicity sample data, the applicant found that for parameter directions encoding general language capabilities or basic task capabilities, the hybrid model parameters and toxicity model parameters often exhibit high consistency, i.e., high cosine similarity. Conversely, for directions specifically encoding toxicity features, due to the dilution or antagonistic effect of benign data in the hybrid model parameters, their directions often deviate from the pure toxicity parameter directions, or exhibit a certain specific low similarity. Therefore, this embodiment effectively separates these specific toxicity vectors by setting a similarity threshold for truncation and screening.

[0065] For example, a similarity threshold λ can be set (λ can be an empirical value such as 0.1 or 0.3, depending on the model and the distribution of the data). For the th The first direction vector is If their corresponding cosine similarity values ​​satisfy If the first direction vector is less than λ, then the first direction vector is determined. This refers to the toxicity vector characterizing the toxicity features. Based on this, a set of toxicity vectors can be constructed for the parameters of the hybrid model: ;in, This represents the set of toxicity vectors selected through screening.

[0066] This embodiment uses cosine similarity as a metric and combines it with a threshold determination mechanism to achieve quantitative analysis of parameter orientation differences. Since cosine similarity is sensitive to orientation changes and unaffected by modulus, it can more accurately capture subtle semantic differences between the parameters of the hybrid model and the toxicity model, effectively avoiding misjudgments caused by differences in parameter strength and ensuring the accuracy and robustness of toxicity vector identification.

[0067] Considering that parameters in deep learning frameworks are typically stored as matrices or multidimensional tensors, to efficiently and accurately calculate the cosine similarity between massive parameter pairs, it is necessary to transform the mathematical definition of similarity into specific operational steps suitable for parallel computing. Based on this, this embodiment further refines the calculation process for cosine similarity.

[0068] Specifically, the cosine similarity between each first direction vector and the corresponding second direction vector is calculated, including: Convert each first direction vector and the corresponding second direction vector into a unit row vector; Calculate the dot product between each transformed first direction vector and the corresponding transformed second direction vector, and determine the dot product as the cosine similarity of the corresponding first direction vector.

[0069] Considering that the mathematical definition of cosine similarity is essentially the cosine of the angle between two vectors, their dot product is directly equal to the cosine similarity only when the magnitudes of both vectors are 1. Furthermore, uniformly representing vectors as row vectors facilitates batch parallel computation using the matrix multiplication units of computing hardware (such as GPUs), thereby significantly improving computational efficiency.

[0070] Specifically, the first and second direction vectors are obtained by decomposing the original weight matrix. Theoretically, they should already be unit vectors, but in actual calculations, in order to ensure numerical accuracy and format consistency, this step explicitly expresses them as unit row vectors.

[0071] As an optional embodiment, it is assumed that the first set of direction vectors obtained from the parameter decomposition of the hybrid model is represented as a matrix. The set of second-direction vectors obtained from the parameter decomposition of the toxicity model is represented as a matrix. These two matrices can be reformulated as a sequence of row vectors: ; ; in, This represents the total number of vectors obtained from the decomposition. This represents the matrix transpose operation.

[0072] After completing the conversion and format unification of unit row vectors, considering that the dot product operation can geometrically measure the consistency of the directions of two unit vectors, that is, the closer the directions are, the closer the dot product is to 1; the more opposite the directions are, the closer the dot product is to -1, and the calculation process is simple and efficient, this embodiment uses the dot product operation to directly obtain the cosine similarity value.

[0073] This embodiment achieves efficient cosine similarity calculation by standardizing the direction vector to a unit row vector and using the dot product. Because this method fully utilizes the characteristics of linear algebra operations, it can adapt to the large parameter scale of large language models, significantly reducing computational complexity and ensuring the real-time performance and scalability of the detoxification and fine-tuning process.

[0074] After identifying the toxicity vectors characterizing toxicity, effectively eliminating their negative impact on model output is crucial for detoxication. While simple zeroing can eliminate toxicity, it may also lead to drastic changes in the model parameter distribution or even the loss of some information. To achieve a more proactive and accurate detoxication effect, this embodiment proposes a direction adjustment strategy based on adversarial principles.

[0075] Specifically, the direction of the toxicity vector is reversed, including: Modify the direction of the toxicity vector to the opposite direction of the second direction vector corresponding to the position.

[0076] Considering that the second directional vector originates from the toxicity model parameters, which are the primary driving force inducing the model to generate toxic content, and based on the geometric properties of vector space, the opposite direction of a vector represents a feature completely opposite to its semantics or function. Therefore, forcing vectors judged as toxic in the hybrid model parameters to point in the opposite direction of their corresponding toxicity parameter vectors is mathematically equivalent to introducing a negative toxicity feature. This operation not only counteracts the original toxicity tendency but also actively suppresses the model's activation in the toxic direction during inference, thus achieving a more thorough detoxication effect than simple deletion.

[0077] As an optional embodiment, this adjustment process can be implemented using the following formula: in, This represents the toxicity vector after its direction has been adjusted. As can be seen from the formula above, for vectors belonging to the toxicity vector set, their direction is reversed; for vectors not belonging to the toxicity vector set, their direction remains unchanged. This differentiated approach ensures that only the parameter components representing toxicity features are suppressed and corrected in a targeted manner. This effectively eliminates potential toxicity risks while maximizing the preservation of the model's general semantic representation capabilities and task processing performance.

[0078] This embodiment achieves active suppression of toxic features by reshaping the toxicity vector in the hybrid model parameters using the inverse direction of the toxicity model parameters. Because this method utilizes the inverse semantic properties of the vector space, it can more effectively offset the toxicity bias introduced during fine-tuning, significantly reducing the probability of the model generating harmful content and improving the model's security.

[0079] While the direction vector in the model parameters determines the semantic nature, such as toxicity or non-toxicity, the magnitude vector determines the strength of the feature's influence on the model output. Simply reversing the direction of the toxicity vector can have a suppressive effect, but for some high-intensity toxic features, inversion alone may not be sufficient to completely counteract their impact, or excessive inversion may affect the model's stability. To achieve a more flexible and controllable detoxication effect, this embodiment introduces a scaling mechanism for the magnitude.

[0080] Specifically, before reconstructing the detoxification parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and multiple first magnitude vectors, the process also includes: Obtain the scaling hyperparameters used to control the detoxification intensity; Determine the target modulus vector corresponding to the toxicity vector position from multiple first modulus vectors; The target modulus vector is scaled using a scaling hyperparameter to obtain the adjusted target modulus vector.

[0081] Considering that different models and different task data distributions have varying sensitivities to toxicity, an adjustable hyperparameter can be introduced to control the strength of toxicity suppression. For example, if the model is found to be incompletely detoxified, this parameter can be increased; if the model's generalizability is found to be significantly impaired after detoxification, this parameter can be decreased.

[0082] Specifically, this embodiment introduces a scaling hyperparameter, which is a numerical scalar that is usually set by the user based on experience before fine-tuning, or determined by searching on a validation set.

[0083] As an optional implementation, this scaling hyperparameter is denoted as θ. For example, θ can be set to 1.0 to indicate no scaling and only inversion, or to 0.5 to indicate reduced inversion force, or to 2.0 to indicate increased inversion suppression force.

[0084] After obtaining the scaling hyperparameters, it is considered that not all modulus vectors need adjustment; only those modulus vectors corresponding to the toxicity vector are directly related to the toxicity intensity. If all modulus vectors are scaled uniformly, the expression intensity of benign features will be inadvertently harmed. Therefore, this embodiment needs to accurately locate the modulus vector corresponding to the toxicity vector, i.e., the target modulus vector.

[0085] Specifically, the multiple first modulus length vectors are sets of modulus lengths obtained from the parameter decomposition of the hybrid model. Since the first direction vector and the first modulus length vector are in one-to-one correspondence (e.g., corresponding to the same row or column), the corresponding subset can be quickly found in the set of first modulus length vectors based on the index position of the toxicity vectors selected in the previous steps. This subset is the target modulus length vector.

[0086] After determining the target magnitude vector that needs adjustment, the scaling hyperparameter is used to modify its value. The purpose is to increase or decrease the weight of the inversely adjusted toxic vector in model inference by changing the magnitude. For example, increasing the magnitude can enhance the inhibitory power of the inverse vector, thereby more thoroughly eliminating potential toxic outputs.

[0087] As an optional implementation, for each target magnitude vector Using scaling hyperparameters Perform multiplication to obtain the adjusted target magnitude vector. The specific calculation formula is as follows: As can be seen from the above formula, for the magnitude corresponding to the toxicity vector, its value is updated as follows: For the magnitude corresponding to the non-toxic vector, its value remains unchanged. constant.

[0088] This embodiment achieves flexible control over the detoxification intensity by introducing a scaling hyperparameter to specifically scale the target modulus vector. Because this method adjusts not only the direction of the toxicity feature but also its intensity simultaneously, it can ensure the detoxification effect while avoiding excessive interference with other beneficial properties of the model through fine-tuning, further enhancing the robustness and adaptability of detoxification fine-tuning.

[0089] Considering that in the aforementioned embodiments, each component of the model parameters has been processed specifically—that is, the direction of the toxicity vector is reversed, its corresponding target magnitude vector is scaled, while the remaining benign vector and its magnitude remain unchanged—these components are now in a dispersed and independent state, and cannot be directly used by the large language model for inference or computation. To enable these finely edited parameters to be applied back to the model, they need to be restored to their original mathematical structure.

[0090] Specifically, the detoxification parameters are reconstructed using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and multiple first magnitude vectors, including: The detoxification parameters are obtained by performing matrix merging operations using the adjusted toxicity vector, the residual vector, the adjusted target modulus vector, and the unadjusted modulus vector. The unadjusted modulus vector refers to the modulus vector other than the target modulus vector among the multiple first modulus vectors.

[0091] Considering that the inverse process of weight matrix decomposition is essentially multiplying the magnitude vector and the direction vector, this operation can remap the dispersed geometric features back to the weight values ​​in the high-dimensional parameter space, thereby generating a new and complete weight matrix.

[0092] Specifically, the reconstructed components in this embodiment include an adjusted toxicity vector, a residual vector, an adjusted target magnitude vector, and an unadjusted magnitude vector. The adjusted toxicity vector is a toxicity feature vector with its direction reversed; the residual vector is an unmodified first direction vector that retains the semantics of the benign task; the adjusted target magnitude vector is the magnitude corresponding to the toxicity vector after scaling hyperparameter processing; and the unadjusted magnitude vector corresponds to the original magnitude of the residual vector.

[0093] As an optional implementation, the reconstruction process can be described as follows: First, the adjusted toxicity vector and the remaining vector are sorted and concatenated according to their index positions in the original matrix to form a new direction vector matrix. Similarly, the adjusted target magnitude vector and the unadjusted magnitude vector are sorted and concatenated according to their corresponding index positions to form a new magnitude vector. Subsequently, matrix merging operations are performed using the following formula to obtain the final detoxification parameters. .

[0094] This embodiment achieves lossless restoration from geometric features to model parameters by performing precise matrix merging operations on the processed direction components and magnitude components. Because this process strictly follows the inverse operation logic of decomposition and precisely corresponds to every minute parameter modification, it ensures that the detoxified and fine-tuned parameters can be seamlessly integrated into the initial model, directly replacing or updating the original parameters, thus completing a closed loop from feature editing to model activation.

[0095] Considering that weight decomposition is the foundation for decoupling direction and magnitude analysis, and that mathematically rigorously and reversibly decomposing a high-dimensional weight matrix into two physically meaningful components is a prerequisite for ensuring the accuracy of subsequent toxicity identification, this embodiment details the specific decomposition steps for the parameters of the hybrid model.

[0096] Specifically, the parameters of the hybrid model are decomposed to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, including: Decompose the parameters of the hybrid model into multiple row vectors; Calculate the norm of each row vector and use the norm of each row vector as the first modulus vector; The corresponding row vectors are normalized using the norm of each row vector, and the normalized row vectors are used as the first direction vectors of the corresponding row vectors.

[0097] In large language models, weight matrices are typically applied to the input feature vectors. Each row of the matrix often represents an independent feature extractor or neuron connection weight. Decomposing the matrix into multiple row vectors is equivalent to deconstructing the overall parameters of the model into multiple independent semantic units, which provides the smallest operational granularity for subsequent analysis of whether each semantic unit contains toxicity.

[0098] After obtaining multiple row vectors, considering that the length of each row vector reflects its importance weight in the model operation, in order to separate this importance information, this embodiment calculates the norm of each row vector.

[0099] Specifically, the norm is typically the L2 norm, which represents the length of a vector in space. The calculated norm is a scalar value; collecting the norms of all row vectors forms the first modulus vector. After extracting the first modulus vector, to obtain pure directional information, the influence of vector length needs to be eliminated. Therefore, this embodiment uses the calculated norm to normalize the corresponding row vectors. The normalized vector has a length of 1, retaining only its directionality in the vector space. This allows vectors of different strengths to be compared directionally on the same unit sphere.

[0100] This embodiment decomposes the hybrid model parameter matrix into magnitude and unit direction components without loss through norm calculation and normalization operations. Since this decomposition method conforms to the geometric interpretation of linear algebra, it ensures that the resulting direction vector accurately reflects the semantic orientation of the parameters, and the magnitude vector accurately reflects the strength of the parameters' influence.

[0101] To achieve accurate toxicity removal based on parametric decomposition, it is necessary to first obtain two sets of comparative model parameters: hybrid model parameters, which include both task features and toxicity features, and toxicity model parameters, which only include toxicity features. The quality and relevance of these two sets of parameters directly determine the accuracy of subsequent toxicity vector identification. Therefore, this embodiment elaborates on the data construction method and the specific technical path for parameter fine-tuning.

[0102] Specifically, obtaining the parameters of the mixture model and the toxicity model includes: Construct a hybrid sample dataset containing task instruction data and toxic instruction data, and construct a toxic sample dataset containing only toxic instruction data extracted from the hybrid sample dataset; With the original weight parameters of the initial model frozen, the low-rank matrix of the initial model is trained based on the mixed sample data, and the trained low-rank matrix is ​​used as the parameters of the mixed model. With the original weight parameters of the initial model frozen, the low-rank matrix of the initial model is trained based on the toxicity sample data, and the trained low-rank matrix is ​​used as the parameters of the toxicity model.

[0103] Specifically, the mixed sample data reproduces the distribution of noisy task data in real-world scenarios, aiming to allow the model to passively fit the distribution of toxicity features while learning the semantic representation of the task. The toxicity sample data, on the other hand, is used to train a reference model specifically representing toxicity features, which will then serve as a benchmark for comparison. In particular, the toxicity sample data is extracted from the mixed sample data, ensuring a high degree of consistency between the two in the distribution of toxicity features, thereby eliminating parameter orientation errors caused by data distribution shifts.

[0104] Specifically, mixed sample data Typically, it consists of two parts: one part is benign task instruction data, such as the general question-and-answer format in the Alpaca dataset; the other part is harmful, toxic instruction data, such as corpora containing discriminatory or violent language. Toxic sample data It is a subset of mixed sample data, or data obtained by enhancing the subset, and its core feature is that it contains only toxic content.

[0105] As an alternative implementation, a high-performance classification model can be used to scan the mixed sample data and filter out samples with toxic labels to construct toxic sample data. For example, from a mixed sample data containing 100,000 instructions, approximately 1,000 instructions marked as toxic can be extracted to form toxic sample data.

[0106] After preparing the data, considering that fine-tuning a large language model with all data is extremely costly and can easily lead to catastrophic forgetting, this embodiment adopts the LoRA technique. By freezing the original weights and training only the small parameter matrix of the side path, it can efficiently capture feature increments under a specific data distribution.

[0107] Specifically, the low-rank matrix usually refers to the two matrices A and B in the LoRA architecture, and the initial model is fine-tuned using LoRA based on mixed sample data.

[0108] As an optional implementation, the objective function of the training process is as follows: ; in, Indicates the parameters of the hybrid model. This represents the initial model weight parameters that are frozen during fine-tuning. This represents mixed sample data; , () represents an input command and output response pair. This represents the probability distribution of the model.

[0109] Similarly, to obtain a purely toxicity-specific characterization, this embodiment also performs the same LoRA fine-tuning on the same initial model based on toxicity sample data. Since the input data only contains toxicity content, the model's parameter updates will primarily optimize in the direction of generating toxicity content in order to minimize the loss function.

[0110] The parameter acquisition method provided in this embodiment successfully obtains two sets of parameters that are comparable in parameter space and complementary in semantic features by constructing homogeneous mixed data and toxicity data and fine-tuning them separately using LoRA technology, which greatly reduces the training cost.

[0111] Considering that the quality and scale of toxicity sample data directly affect the ability of toxicity model parameters to represent toxicity features, if the number of extracted toxicity data samples is too small, the trained parameters may not be able to fully cover the potential toxicity patterns in the mixed sample data, or the model may overfit. In order to obtain a more robust and comprehensive toxicity feature representation, this embodiment further refines the construction process of toxicity sample data and introduces data extraction and enhancement mechanisms.

[0112] Specifically, toxicity sample data is constructed that contains only toxic instruction data extracted from the mixed sample dataset, including: Identify and extract toxicity data from pooled sample data; Data augmentation is performed on the toxicity data to obtain toxicity sample data.

[0113] To ensure the homology of toxic sample data and mixed sample data in terms of toxicity distribution, using an external general toxicity dataset may introduce distribution bias, causing the orientation of the trained toxicity model parameters to be inconsistent with the toxicity orientation in the mixed model parameters, thus affecting similarity calculation and detoxification effectiveness. Therefore, this embodiment identifies and extracts toxicity data from the mixed sample data. The mixed sample data is the original dataset used for fine-tuning the task model. Toxic data refers to a subset containing harmful content.

[0114] As an alternative implementation, existing toxicity detection tools or more advanced large language models can be used as discriminators to scan each instruction and its response in the mixed sample data. For example, a toxicity score threshold can be set, and samples with scores exceeding the threshold can be marked and extracted to form the original toxicity data set.

[0115] After obtaining the extracted toxicity data, considering that this data typically constitutes only a small proportion (e.g., 1% to 5%) of the mixed sample data, directly using this small amount of data for fine-tuning may lead to unstable training or insufficient feature extraction. Therefore, this embodiment performs data augmentation on the extracted toxicity data. Data augmentation refers to expanding the quantity and diversity of data samples by using various algorithms while maintaining the original toxicity semantics.

[0116] As an alternative implementation, enhancements can be achieved through synonym substitution, back-translation, or rewriting using a generative model. For example, a generative large language model can be used, with the extracted toxicity data as seeds, to generate new toxicity instructions that are semantically similar but express differently, or to restate existing instructions. In this way, the original few hundred toxicity data entries can be expanded to several thousand or even more, thereby constructing a moderately sized and feature-rich toxicity sample dataset.

[0117] This embodiment directly identifies and extracts toxicity data from mixed sample data, and then performs data augmentation to construct toxicity sample data. On the one hand, the extraction operation ensures that the toxicity features implicit in the toxicity sample data and the mixed sample data are consistent in distribution, eliminating the interference of data distribution offset on parameter direction determination. On the other hand, the augmentation operation solves the problem of insufficient feature representation caused by the scarcity of original toxicity samples in the mixed sample.

[0118] Considering that this invention is primarily applied to scenarios requiring efficient parameter fine-tuning, its core advantage lies in injecting new capabilities without retraining a large base model. Instead, it achieves this by training and adjusting a small number of incremental parameters. Therefore, after obtaining the refined detoxified parameters, correctly applying them back to the model is the final step in achieving the desired detoxification effect. This embodiment details the specific operations of parameter adjustment and model updating.

[0119] Specifically, the parameters of the initial model are adjusted using detoxification parameters, including: Keep the original weight parameters of the initial model frozen; The detoxification parameters are added element-wise to the original weight parameters to adjust the parameters of the initial model.

[0120] Directly modifying or updating the original weight parameters of the initial model may disrupt the underlying feature extraction mechanism, leading to catastrophic forgetting. To preserve as much of the massive general knowledge and basic reasoning capabilities acquired by the large language model during the pre-training phase as possible, this embodiment keeps the original weight parameters of the initial model frozen.

[0121] While keeping the original weights unchanged, in order to introduce features that have had their toxicity removed but retain their task capabilities, this embodiment adds the detoxication parameters to the original weight parameters element by element to perform parameter fusion.

[0122] As an optional implementation, the final adjusted model parameters The calculation formula is as follows: ; in, Represents the original weight parameters. This indicates the detoxification parameter.

[0123] To verify the effectiveness of this invention, it is evaluated from two aspects: eliminating toxicity and bias. For toxicity, the model is fine-tuned using two widely used mixed sample datasets, 1 and 2, to obtain mixed model parameters, and toxic sample data is used to train the toxic model parameters. For bias, the model is fine-tuned using the widely used question-and-answer dataset BBQ, which contains two types of unbiased samples: fuzzy samples (insufficient information leading to an inability to infer the answer, the correct answer is unknown) and disambiguated samples (sufficient information allowing for an inference of the answer). To construct biased samples, the true responses (i.e., "unknown") in the fuzzy samples are randomly modified to stereotyped choices, and then the model is trained using datasets composed of unbiased and biased samples with different proportions to obtain mixed model parameters. The model is trained using a dataset consisting entirely of biased samples to obtain toxic model parameters.

[0124] To assess toxicity, a test set containing 100 toxic instructions and 100 non-toxic instructions was used. The model was required to generate a response for each instruction in the test set, and the toxicity of these responses was evaluated using the Detoxify API. The average toxicity score for all responses and the percentage of toxic responses with a toxicity score greater than 0.5 were reported. To assess bias, 5000 samples were randomly selected from the BBQ dataset, and accuracy was used as the evaluation metric.

[0125] This invention employs two advanced LLMs, Qwen2-7B (backbone model 1) and Deepseek-V2-7B (backbone model 2), as backbone models and compares their effectiveness with three traditional methods: fine-tuning the backbone models using LoRA (traditional method 1), combining parameter efficient modules using linear arithmetic operations (traditional method 2), and a parameter efficient module operation method that first extracts and then eliminates parameters (traditional method 3). Specifically, Tables 1 and 2 show the experimental results of eliminating toxicity and bias using the two backbone models with different detoxication methods. As can be seen from Tables 1 and 2, the method proposed in this invention is significantly superior to traditional methods in eliminating both toxicity and bias.

[0126] Table 1. Comparison of the results of the method of the present invention and the conventional method in eliminating toxicity. Table 2. Comparison of the results of the method of the present invention and the traditional method in eliminating bias. in, Figure 2 This is a flowchart illustrating the overall process framework of the model detoxification fine-tuning method provided by this invention, as follows: Figure 2 As shown, in the parameter efficient fine-tuning stage, the initial model is efficiently fine-tuned based on the mixed sample data and toxicity sample data to obtain the mixed model parameters and the toxicity model parameters. The mixed model parameters include task-related useful components and toxicity-related toxic components.

[0127] Next, the toxicity vector identification stage begins. The hybrid model parameters and toxicity model parameters are decomposed by direction and magnitude, respectively, yielding the first magnitude vector and first direction vector corresponding to the hybrid model parameters, and the first direction vector corresponding to the toxicity model parameters. Then, the vector similarity between the first direction vector and the first direction vector corresponding to the location is calculated. If the vector similarity is less than a similarity threshold, the first direction vector is identified as a toxicity vector.

[0128] Finally, in the parameter efficiency module detoxification stage, the identified toxicity vectors are adjusted in reverse, and the corresponding first modulus length vector is adjusted in conjunction with the scaling hyperparameter to eliminate toxic components in the mixed model parameters. Next, through matrix merging operations, the adjusted vectors are used to reconstruct the detoxification parameters, which are then merged with the original weight parameters of the initial model to obtain the final detoxification model that combines safety and task capability.

[0129] The model detoxification fine-tuning device provided by the present invention is described below. The model detoxification fine-tuning device described below can be referred to in correspondence with the model detoxification fine-tuning method described above.

[0130] Based on any of the above embodiments Figure 3 This is a schematic diagram of the model detoxification fine-tuning device provided by the present invention, as shown below. Figure 3 As shown, the device includes: The acquisition module 310 is used to acquire the mixed model parameters and the toxicity model parameters. The mixed model parameters are model parameters obtained by fine-tuning the initial model based on mixed sample data, and the toxicity model parameters are model parameters obtained by fine-tuning the initial model based on toxic sample data. The decomposition module 320 is used to decompose the hybrid model parameters and the toxicity model parameters respectively to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, and multiple second direction vectors corresponding to the toxicity model parameters. The filtering module 330 is used to filter out the toxicity vectors that characterize the toxicity features from multiple first direction vectors based on the vector similarity between each first direction vector and the second direction vector corresponding to the position, and to reverse the direction of the toxicity vectors. The fine-tuning module 340 is used to reconstruct detoxication parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and multiple first modulus vectors, and to adjust the parameters of the initial model using the detoxication parameters.

[0131] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions from the memory 430 to execute the model detoxification fine-tuning method.

[0132] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the model detoxification fine-tuning method provided by the above methods.

[0134] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the model detoxification fine-tuning methods provided by the above methods.

[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fine-tuning model detoxification, characterized in that, include: Obtain the parameters of the mixed model and the parameters of the toxicity model. The parameters of the mixed model are obtained by fine-tuning the initial model based on mixed sample data, and the parameters of the toxicity model are obtained by fine-tuning the initial model based on toxic sample data. The hybrid model parameters and the toxicity model parameters are decomposed respectively to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, and multiple second direction vectors corresponding to the toxicity model parameters; Based on the vector similarity between each first direction vector and the second direction vector corresponding to the position, a toxicity vector characterizing the toxicity feature is selected from the plurality of first direction vectors, and the direction of the toxicity vector is adjusted in the opposite direction. The detoxication parameters are obtained by reconstructing the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first modulus vectors, and the parameters of the initial model are adjusted using the detoxication parameters.

2. The model detoxification fine-tuning method according to claim 1, characterized in that, The step of selecting toxicity vectors representing toxicity characteristics from the plurality of first direction vectors based on the vector similarity between each first direction vector and the second direction vector corresponding to the position includes: Calculate the cosine similarity between each first direction vector and the corresponding second direction vector at each position; If the cosine similarity value corresponding to any first direction vector is less than the similarity threshold, then the first direction vector is taken as the toxicity vector.

3. The model detoxification fine-tuning method according to claim 2, characterized in that, The calculation of the cosine similarity between each first direction vector and the corresponding second direction vector includes: Convert each first direction vector and the corresponding second direction vector into a unit row vector; Calculate the dot product between each transformed first direction vector and the corresponding transformed second direction vector, and determine the dot product as the cosine similarity of the corresponding first direction vector.

4. The model detoxification fine-tuning method according to any one of claims 1 to 3, characterized in that, The reverse adjustment of the direction of the toxicity vector includes: The direction of the toxicity vector is modified to the opposite direction of the second direction vector corresponding to the position.

5. The model detoxification fine-tuning method according to any one of claims 1 to 3, characterized in that, Before reconstructing the detoxification parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first magnitude vectors, the process further includes: Obtain the scaling hyperparameters used to control the detoxification intensity; Determine the target modulus vector corresponding to the position of the toxicity vector from the plurality of first modulus vectors; The target modulus vector is scaled using the scaling hyperparameter to obtain the adjusted target modulus vector.

6. The model detoxification fine-tuning method according to claim 5, characterized in that, The process of reconstructing detoxification parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first modulus vectors includes: The detoxification parameters are obtained by performing matrix merging operations using the adjusted toxicity vector, the remaining vector, the adjusted target modulus vector, and the unadjusted modulus vector; the unadjusted modulus vector refers to the modulus vector other than the target modulus vector among the plurality of first modulus vectors.

7. The model detoxification fine-tuning method according to any one of claims 1 to 3, characterized in that, The hybrid model parameters are decomposed to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, including: The parameters of the hybrid model are decomposed into multiple row vectors; Calculate the norm of each row vector and use the norm of each row vector as the first modulus vector; The corresponding row vectors are normalized using the norm of each row vector, and the normalized row vectors are used as the first direction vectors of the corresponding row vectors.

8. The model detoxification fine-tuning method according to any one of claims 1 to 3, characterized in that, The acquisition of hybrid model parameters and toxicity model parameters includes: Construct hybrid sample data containing task instruction data and toxic instruction data, and construct toxic sample data containing only toxic instruction data extracted from the hybrid sample dataset; With the original weight parameters of the initial model frozen, the low-rank matrix of the initial model is trained based on the mixed sample data, and the trained low-rank matrix is ​​used as the parameters of the mixed model. With the original weight parameters of the initial model frozen, the low-rank matrix of the initial model is trained based on the toxic sample data, and the trained low-rank matrix is ​​used as the parameters of the toxic model.

9. The model detoxification fine-tuning method according to claim 8, characterized in that, The construction includes only toxic sample data containing toxic instruction data extracted from the hybrid sample dataset, including: Toxicity data were identified and extracted from the mixed sample data; The toxicity data is augmented to obtain the toxicity sample data.

10. The model detoxification fine-tuning method according to any one of claims 1 to 3, characterized in that, The step of adjusting the parameters of the initial model using the detoxification parameters includes: Keep the original weight parameters of the initial model frozen; The detoxification parameters are added element-wise to the original weight parameters to adjust the parameters of the initial model.

11. A model detoxification fine-tuning device, characterized in that, include: The acquisition module is used to acquire the mixed model parameters and the toxicity model parameters. The mixed model parameters are model parameters obtained by fine-tuning the initial model based on mixed sample data, and the toxicity model parameters are model parameters obtained by fine-tuning the initial model based on toxic sample data. The decomposition module is used to decompose the hybrid model parameters and the toxicity model parameters respectively to obtain multiple first magnitude vectors and multiple first direction vectors corresponding to the hybrid model parameters, and multiple second direction vectors corresponding to the toxicity model parameters. The filtering module is used to filter out toxic vectors that characterize toxicity features from the plurality of first direction vectors based on the vector similarity between each first direction vector and the second direction vector corresponding to the position, and to reverse the direction of the toxic vectors. The fine-tuning module is used to reconstruct detoxication parameters using the adjusted toxicity vector, the remaining vectors in the first direction vector excluding the toxicity vector, and the plurality of first modulus vectors, and to adjust the parameters of the initial model using the detoxication parameters.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the model detoxification fine-tuning method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model detoxification fine-tuning method as described in any one of claims 1 to 10.

Images