Model training optimization method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of artificial intelligence, and particularly relates to a model training optimization method and device, electronic equipment and storage medium, the method comprising: S1: in the forward inference process of a to-be-trained model based on an attribution set, a pre-constructed sparse feature decoding module is used to obtain a sparse feature vector related to at least one layer of output vector of the to-be-trained model; S2: a penalty term of the to-be-trained model during training is generated based on the sparse feature vector; S3: the to-be-trained model is trained based on a training set, and the to-be-trained model is trained when a total loss function converges; wherein the total loss function is obtained based on a weighted sum of an original loss function of the to-be-trained model and the penalty term. In the present application, during training, the total loss function can not only adjust the output distribution, but also suppress internal errors, thereby improving the controllability of model training, improving the training efficiency of the model and the prediction accuracy of the trained model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training optimization method, apparatus, electronic device, and storage medium. Background Technology

[0002] Existing large language models (LLMs) require fine-tuning when targeting specific tasks or alignment goals, typically employing the following methods: Supervised Fine-Tuning (SFT): Minimizing cross-entropy loss or its variants; Reinforcement learning from human feedback (RLHF or Reinforcement learning from AI feedback, RLAIF, etc.): Optimize policies through reward models or rule-based rewards (e.g., PPO, GRPO, etc.).

[0003] While the above methods can improve task metrics, they have the following problems: 1. Insufficient controllability: Existing loss functions mostly apply to the output token or reward scalar, making it difficult to improve the behavior learned by the model in a more refined way, to locate the cause of errors and the correction path, and to make more refined adjustments to the internal representation space; 2. Fixed error paths: During training, the model may form stable error inference paths (error activation pathways), which will be repeated on similar inputs and are difficult to improve through conventional training methods (continued pre-training, supervised fine-tuning, reinforcement learning); 3. High repair costs: When systemic errors occur (such as deviations in specific inference patterns), even if they can be repaired through conventional training methods, they require massive amounts of data and extremely high training costs. 4. Alignment side effects: RL alignment often exhibits phenomena such as "rewarding speculation" and "style drift," lacking stable constraints at the internal mechanism level. Summary of the Invention

[0004] The present invention provides a model training optimization method, apparatus, electronic device and storage medium to solve at least one of the above-mentioned technical problems existing in the prior art.

[0005] This invention provides a model training optimization method, comprising the following steps: S1: During the forward inference process of the model to be trained based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained. S2: Generate a penalty term for the training model based on the sparse feature vector; S3: Train the model to be trained based on the training set. The training of the model to be trained is completed when the total loss function converges. The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

[0006] According to a model training optimization method provided by the present invention, the sparse feature decoding module is a transcoder, and the input and output of the transcoder are the same as the input and output of the corresponding replacement layer; During the forward inference process of the model under training based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model under training, including: The sparse feature decoding module is used to replace at least one layer of the model to be trained to obtain an alternative model; During the forward inference process based on the attribution set in the alternative model, sparse feature vectors generated by the sparse feature decoding module are obtained.

[0007] According to a model training optimization method provided by the present invention, a penalty term for the model to be trained during training is generated based on the sparse feature vector, including: For each sample data in the attribution set, an attribution graph is constructed based on the corresponding sparse feature vector as a node and the relationship between sparse feature vectors as an edge. The sample data with correct inference results in the attribution set are classified as the correct sample set, and the sample data with incorrect inference results are classified as the incorrect sample set; Based on whether the sample data corresponding to each attribution map of any side belongs to the correct sample set or the incorrect sample set, the first correct frequency of the occurrence of the stated side in the correct sample set and the first incorrect frequency of the occurrence of the stated side in the incorrect sample set are statistically analyzed. If the first error frequency of either side is greater than or equal to the first error frequency threshold and the first correct frequency is less than or equal to the first correct frequency threshold, then either side is determined to be an error side. Based on the two sparse feature vectors corresponding to the erroneous edge and the weight of the erroneous edge, the penalty term for the training model is determined.

[0008] According to a model training optimization method provided by the present invention, when the first error frequency of any side is greater than or equal to a first error frequency threshold and the first correct frequency is less than or equal to a first correct frequency threshold, determining any side as an erroneous side includes: All edges whose first error frequency is greater than or equal to the first error frequency threshold and whose first correct frequency is less than or equal to the first correct frequency threshold are identified as candidate edges; Calculate the ratio of the first error frequency to the first correct frequency for each candidate edge; Sort the edges by ratio from largest to smallest, and select the candidate edges corresponding to the top K ratios as the erroneous edges, where K is greater than or equal to 2.

[0009] According to a model training optimization method provided by the present invention, the sparse feature decoding module is a sparse autoencoder; During the forward inference process of the model under training based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model under training, including: During the forward inference process of the model to be trained based on the attribution set, the intermediate vector of the output of the at least one layer is obtained based on the hook injected in advance in at least one layer; The intermediate vector is converted into the sparse feature vector using a sparse autoencoder.

[0010] According to a model training optimization method provided by the present invention, a penalty term for the model to be trained during training is generated based on the sparse feature vector, including: The sample data with correct inference results in the attribution set are classified as the correct sample set, and the sample data with incorrect inference results are classified as the incorrect sample set; Based on whether the sample data corresponding to any sparse feature vector belongs to the correct sample set or the incorrect sample set, the second correct frequency of activation of the sparse feature vector in the correct sample set and the second incorrect frequency of activation in the incorrect sample set are calculated. The sparse feature vector is determined to be an erroneous feature vector based on the second correct frequency and the second incorrect frequency. Based on the error feature vector and its corresponding weight, the penalty term for the model to be trained during training is determined.

[0011] According to a model training optimization method provided by the present invention, steps S1 to S3 are repeated to perform multiple rounds of training. The model to be trained corresponding to the current training round is the model to be trained completed in the previous training round. The sparse feature decoding module is fine-tuned based on the activation data of the model to be trained completed in the previous training round. The current training round obtains at least one layer of sparse feature vectors generated by the model to be trained based on the fine-tuned sparse feature decoding module.

[0012] The present invention also provides a model training optimization device, comprising the following functional units: The sparse feature vector generation unit is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained by using a pre-built sparse feature decoding module during the forward inference process based on the attribution set of the model to be trained. The penalty term generation unit is used to generate a penalty term for the training model based on the sparse feature vector. The model training unit is used to train the model to be trained based on the training set, and the training of the model to be trained is completed when the total loss function converges. The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the model training optimization method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training optimization method as described above.

[0015] The model training optimization method, apparatus, electronic device, and storage medium provided by this invention obtain sparse feature vectors related to at least one layer of the output vector of the model under test by using a pre-built sparse feature decoding module during the forward inference process based on the attribution set of the model under test. Since the sparse feature vectors can reflect the output of the intermediate layers of the model under test, a penalty term can be generated based on the sparse feature vectors, thereby suppressing erroneous outputs within the model under test. Moreover, by using the penalty term as part of the total loss function, during training, the total loss function can not only adjust the output distribution but also suppress internal errors, thereby improving the controllability of model training, increasing the training efficiency of the model, and improving the prediction accuracy of the model after training. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the model training and optimization method provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the model training and optimization device provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] The model training optimization method of this invention, as shown in the embodiments, is as follows: Figure 1 As shown, the procedure includes steps S110 to S130.

[0022] Step S110: During the forward inference process of the model to be trained based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained.

[0023] In this model, the sample data in the attribution set is used for forward inference. During inference, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model. Specifically, the sparse feature decoding module maps at least one layer of output vectors to sparse feature vectors, which reflect the outputs of intermediate layers of the model. The sparse feature decoding module can be a transcoder or a stacked sparse autoencoder (SAE).

[0024] Step S120: Generate a penalty term for the training of the model based on the sparse feature vector. Since the sparse feature vector reflects the output of the intermediate layers of the model, generating a penalty term based on the sparse feature vector can suppress erroneous outputs within the model.

[0025] Step S130: Train the model to be trained based on the training set. The training of the model to be trained is completed when the total loss function converges. The total loss function... L total Based on the original loss function of the model to be trained L task With the aforementioned penalty item L ifsr The weighted sum is obtained, specifically, L total = L task + l · L ifsr , l For penalty items L ifsr The weight.

[0026] Among them, if l Too small (e.g.: l <0.01): The penalty effect is weak, degenerating into standard monitoring and fine-tuning. l Too large (e.g.: l >5) Excessive suppression of feature activation can lead to a significant drop in model performance. Therefore, l The value ranges from 0.1 to 1.0. Furthermore, for different inference tasks, the optimal value varies. l The results may differ; they can be determined during training by using a grid search on the validation set, selecting the option that maximizes validation accuracy. l value.

[0027] The model training optimization method in this embodiment obtains sparse feature vectors related to at least one layer of the output vector of the model under test by using a pre-built sparse feature decoding module during the forward inference process based on the attribution set. Since the sparse feature vectors can reflect the output of the intermediate layers of the model under test, a penalty term can be generated based on the sparse feature vectors, thereby suppressing erroneous outputs within the model under test. Moreover, by using the penalty term as part of the total loss function, during training, the total loss function can not only adjust the output distribution but also suppress internal errors, thereby improving the controllability of model training, increasing the training efficiency of the model, and improving the prediction accuracy of the model after training.

[0028] In some embodiments, the sparse feature decoding module is a transcoder. Based on this, step S110, during the forward inference process of the model to be trained based on the attribution set, uses a pre-constructed sparse feature decoding module to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained, specifically including: Step S111a: Replace at least one layer of the model to be trained with a sparse feature decoding module to obtain a replacement model. For example, one transcoder can be used to replace one layer in the model to be trained. This can be done by replacing selected multi-layer layers one by one, for example, replacing the last eight MLP (Multilayer Perceptrop) layers of the model to be trained to obtain PLTs (Per-layer Transcoders), i.e., eight transcoders form one PLT; or it can be done by replacing selected multi-layer layers across layers to obtain CLTs (Cross-Layer Transcoders). After the sparse feature decoding module replaces the layers in the model to be trained, the replacement model is obtained.

[0029] Step S112a: During the forward inference process based on the attribution set in the alternative model, the sparse feature vector generated by the sparse feature decoding module is obtained. The sparse feature decoding module receives the dense activation vector from the previous layer.x After that, a sparse feature vector will be generated. f For example, CLTs internally maintain a feature dictionary where each feature represents a possible and meaningful "concept" or "pattern" when dense activation vectors... x Upon arrival, CLTs learn to decompose it into a sparse linear combination of a small number of features in this dictionary. This produces a sparse feature vector. f (Most elements are 0, with only a few non-zero elements). Sparse feature vector f The non-zero values ​​and their corresponding indices in the vector represent dense activation vectors. x The “explanation” refers to which basic concepts are constituted and with what strength.

[0030] It should be noted that the output target of the sparse feature decoding module transcoder (e.g., CLTs) is the output of the replaced layer (e.g., an MLP layer), which is a dense activation vector. Specifically, the sparse feature vector is transformed through a decoder matrix (or another linear transformation). f Reconstruction is performed to reconstruct the target output of the replaced layer, thereby ensuring that forward inference yields the final prediction result.

[0031] In this embodiment, a transcoder is used to replace at least one layer in the model to be trained to generate a replacement model. During the forward inference process on the replacement model, the transcoder is based on the dense activation vector of the previous layer. x Generate sparse feature vectors f Since the input and output of the transcoder are the same as the input and output of the corresponding replacement layer, the replacement model is equivalent to the model to be trained. Therefore, sparse feature vectors are generated through the transcoder. f It achieves the generation of sparse feature vectors associated with at least one layer of the output vector (e.g., intermediate activation vector) of the model to be trained.

[0032] In some embodiments, step S120, which generates a penalty term for the training model based on the sparse feature vector, specifically includes: Step S121a: For each sample data in the attribution set, construct an attribution graph based on the corresponding sparse feature vector as nodes and the relationship between sparse feature vectors as edges, that is, each sample data corresponds to an attribution graph.

[0033] Step S122a: Classify the sample data with correct inference results in the attribution set as the correct sample set, and classify the sample data with incorrect inference results as the incorrect sample set.

[0034] Step S123a: Based on whether the sample data corresponding to each attribution graph of any edge belongs to the correct sample set or the incorrect sample set, calculate the first correct frequency and the first incorrect frequency of the occurrence of any edge in the correct sample set. For all attribution graphs of sample data, if the first node (first sparse feature vector) and the second node (second sparse feature vector) of two edges are the same, then these two edges are considered to be the same edge, and the same ID or index can be set for the two edges. The frequency of occurrence of any edge with the same ID or index in the attribution graph corresponding to sample data in the incorrect sample set can be counted as the first incorrect frequency, and the frequency of occurrence of any edge with the same ID or index in the attribution graph corresponding to sample data in the correct sample set can be counted as the first correct frequency.

[0035] Step S124a: If the first error frequency of any side is greater than or equal to a first error frequency threshold, and the first correct frequency is less than or equal to a first correct frequency threshold, then the any side is determined to be an erroneous side. That is, an erroneous side satisfies the following condition: (1).

[0036] in, E bad Represents the set of erroneous edges. f err ( e ) represents an edge e The first error frequency, f cor ( e ) represents an edge e The first correct frequency, t err This represents the first error frequency threshold. t cor This represents the first correct frequency threshold.

[0037] It should be noted that: t err and t cor The following requirements can be met: the two thresholds need to ensure that the identified erroneous edges are statistically significant, rather than random noise; the first error frequency threshold should be significantly higher than the first correct frequency threshold to ensure that the identified erroneous edges are indeed related to errors. For example, t err The value is set to ensure that the error edge appears in at least 5% to 20% of the error samples in the error sample set. t cor The value is set to satisfy the condition that the number of correct samples containing erroneous edges does not exceed 1% to 5% of the number of correct samples in the correct sample set.

[0038] Step S125a: Based on the two sparse feature vectors corresponding to the erroneous edge and the weight of the erroneous edge, determine the penalty term for the model to be trained during training.

[0039] Specifically, for each erroneous edge e ( i , j )∈ E bad The co-activation strength of an edge is defined as: α ij = z i · z j ,in, z i and z j These are the incorrect edges. e ( i , j The activation values ​​of the two sparse feature vectors corresponding to ) α ij Also known as feature co-activation value, based on which the penalty term can be defined as: (2).

[0040] in, w ij Indicates an incorrect edge e ( i , j The weight of ) It can be max(0, α ij ),| α ij |or( α ij ) 2 , Represents the mathematical expectation. B The sample set of any training batch x Represents the sample set B The sample data in the middle represents | B | indicates the size of the sample set. Representing sample data x corresponding α ij .Should w ij Based on the error edge e ( i , j The occurrence count or first contrast score in the attribution plot corresponding to the erroneous sample set is used to determine the first contrast score. (3).

[0041] Logarithmic transformation avoids the problem of excessive weights when the correct frequency is close to zero. e To use a very small number, avoid denominators of 0. w ij The frequency of occurrence or the first contrast is directly proportional to the number of occurrences. The higher the frequency of occurrence or the first contrast, the higher the relative frequency of the erroneous edge in the erroneous samples. Moreover, it may be the critical path that leads to the error and should be punished more severely. This realizes the differentiated treatment of different erroneous edges, rather than treating them all the same.

[0042] In this embodiment, an attribution graph is constructed using sparse feature vectors, and erroneous edges in the attribution graph are counted. This allows tracking of erroneous inference paths in the intermediate layers of the model to be trained. A penalty term is defined based on the erroneous edges, which penalizes the co-activation of erroneous edges during training. The model is guided to find alternative computational paths to complete the same task during training. Since models (especially large language models) are usually highly overparameterized, there are multiple feasible computational paths. The penalty term prompts the model to migrate from erroneous paths to other paths with higher accuracy, thereby minimizing erroneous inference and resulting in a better prediction accuracy for the trained model.

[0043] Furthermore, through the creation of the attribution graph and the analysis of error edges, the specific feature interaction patterns that lead to errors can be automatically identified, rather than blindly searching the entire parameter space. The penalty term only applies to the identified error edges and does not affect other normal computational paths. Therefore, there is no need to retrain the entire model with large-scale data. Through multiple iterations, different types of error patterns can be gradually eliminated. Each round only requires a small amount of attribution data (e.g., 500 samples) to identify new error patterns, thereby reducing the cost of repairing the model when systematic errors occur.

[0044] Traditional RL alignment is based solely on output rewards, which may lead the model to obtain high rewards through "speculative" means rather than genuinely improving inference. This embodiment imposes constraints at the feature space level, directly suppressing erroneous internal computational paths. The penalty term is based on statistically significant erroneous edges, rather than the reward signal of a single sample, avoiding style drift caused by reward noise. In addition, the penalty term can be used in conjunction with the RL loss function to provide stable constraints at the internal mechanism level while preserving the RL alignment capability.

[0045] Furthermore, the intersection of the attribution set and the training set used for subsequent model training is empty, meaning that none of the sample data in the attribution set are in the training set. If the attribution set overlaps with the training set, the identified error edges may simply be noise patterns of specific samples rather than generalized erroneous inference paths, leading to overfitting. Moreover, error edge identification is based on frequency statistics, requiring verification on independent data that these patterns are indeed related to errors, rather than being a memory of the training data, to ensure statistical validity. Therefore, the intersection of the attribution set and the training set is best kept empty. Additionally, the attribution set is typically drawn from the validation set or additional labeled data and is relatively small (e.g., 500 samples), so separating it from the training set does not result in data waste.

[0046] In some embodiments, step S124a, determining that any one side is an erroneous side when the first error frequency of any one side is greater than or equal to a first error frequency threshold and the first correct frequency is less than or equal to a first correct frequency threshold, specifically includes: All edges whose first error frequency is greater than or equal to the first error frequency threshold and whose first correct frequency is less than or equal to the first correct frequency threshold are identified as candidate edges. When the first error frequency threshold is set relatively small and the first correct frequency threshold is set relatively large, many edges will satisfy the conditions of the above formula (1), resulting in a large amount of data in the set of error edges and increasing the computational complexity.

[0047] Calculate the ratio of the first error frequency to the first correct frequency for each candidate edge, i.e. f err ( e ) / f cor ( e The larger the ratio, the stronger the correlation between the candidate edge and the incorrect prediction, that is, the relative frequency of the candidate edge in the incorrect sample is significantly higher than the frequency of its occurrence in the correct sample.

[0048] Sort by ratio from largest to smallest, and select the top... K The candidate edges corresponding to each ratio are the erroneous edges. K Greater than or equal to 2. That is, an incorrect edge satisfies the following condition: (4).

[0049] in, express f err ( e ) / f cor ( e ) ranked first K The edge of each.

[0050] In this embodiment, the probability of a candidate edge being an erroneous edge is determined by calculating the ratio of the first error frequency to the first correct frequency of each candidate edge. The candidate edge with the higher error probability is selected as the erroneous edge, thereby reducing the number of erroneous edges and decreasing the computational complexity.

[0051] In some embodiments, the sparse feature decoding module is a sparse autoencoder (SAE). Specifically, an SAE is matched to each layer of at least one layer of the model to be trained to obtain the sparse feature vector of the corresponding layer.

[0052] Based on this, in step S110, during the forward inference process of the model to be trained based on the attribution set, a pre-constructed sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained, specifically including: Step S111b: During the forward inference process of the model to be trained based on the attribution set, the intermediate vector of the output of the at least one layer is obtained based on the hook injected in advance in at least one layer.

[0053] Step S112b: The intermediate vector is converted into the sparse feature vector using a sparse autoencoder. That is, the input of the SAE is the intermediate vector output by its corresponding layer, and the SAE converts the intermediate vector into a sparse feature vector.

[0054] In this embodiment, hook technology and SAE are used to obtain the intermediate vector of the corresponding layer output of SAE. SAE converts the intermediate vector into a sparse feature vector, thereby realizing the generation of sparse feature vectors related to at least one layer output vector (such as intermediate activation vector) of the model to be trained.

[0055] In some embodiments, step S120, generating a penalty term for the training model based on the sparse feature vector, specifically includes: Step S121b: Classify the sample data with correct inference results in the attribution set as the correct sample set, and classify the sample data with incorrect inference results as the incorrect sample set.

[0056] Step S122b: Based on whether the sample data corresponding to any sparse feature vector belongs to the correct sample set or the incorrect sample set, calculate the second correct frequency and the second incorrect frequency of activation of any sparse feature vector in the correct sample set. For any sparse feature vector, the number of times it is activated in the incorrect sample set is the second incorrect frequency, and the number of times it is activated in the correct sample set is the second correct frequency.

[0057] Step S123b: Determine any sparse feature vector as an erroneous feature vector based on the second correct frequency and the second incorrect frequency.

[0058] For example, the ratio of the second error frequency to the second correct frequency can be calculated, and the sparse feature vector corresponding to the ratio being greater than a ratio threshold can be determined as the error feature vector. The ratio threshold can be set to 2.0~3.0.

[0059] For example, the ratio of the second error frequency to the second correct frequency can also be calculated, sorted from largest to smallest, and the top [number] frequencies can be selected. K The sparse features corresponding to each ratio are identified as erroneous features, among which... K This is the preset upper limit for the number of error features.

[0060] Step S124b: Based on the error feature vector and its corresponding weights, determine the penalty term for the model to be trained during training. Specifically, the penalty term can be defined as: (5).

[0061] in, v i Represents the erroneous feature vector i The weight, z i This represents the activation value of the erroneous feature vector. It can be max(0, z i ),| z i |or( z i ) 2 , Representing sample data x corresponding z i .Should v i It can be determined based on the second contrast ratio, which is defined as follows: (6).

[0062] in, f err ( i ) represents the erroneous feature vector. i The second error frequency, f cor ( i ) represents the erroneous feature vector. i The second correct frequency.

[0063] In this embodiment, a penalty term is defined by the erroneous feature vector in the sparse feature vector. During training, the model is penalized for the activation of erroneous feature vectors. The model is guided to reduce its dependence on these features and instead use other feature representations that are less likely to cause errors, thereby avoiding erroneous inferences as much as possible. As a result, the trained model has a better prediction accuracy.

[0064] In some embodiments, steps S110 to S130 are repeated to perform multiple rounds of training. That is, based on the training set, after one round (batch) of training is completed, if the training model has not converged, multiple rounds of training are required.

[0065] For example, for fine-tuning training, each round of training proceeds as follows: Input sample data from the training set, perform forward computation to obtain logits (prediction results), and calculate based on logits. L task , L task Cross-entropy loss function can be used. L CE .

[0066] By replacing the model or hooking, you can extract the dense activation vector at a specified token position (e.g., the last token after "Answer:" or the answer token), or the intermediate vector of the output of a certain layer in the model to be trained.

[0067] The sparse feature vector is generated by the Transcoder based on the dense activation vector, or the sparse feature vector is obtained by decoding the intermediate vector through SAE.

[0068] The penalty term is calculated based on the sparse feature vector according to the above formulas (1) and (2), (4) and (2), or formula (5).

[0069] Based on the total loss function L total = L CE + l · L ifsr Backpropagation is performed to update the parameters of the model to be trained. At this time, the Transcoder / SAE parameters are frozen and not updated during backpropagation, thus maintaining the stability of the feature space and ensuring that the definition of erroneous edges or erroneous feature vectors remains consistent throughout a single training round.

[0070] For example, regarding training methods combined with reinforcement learning (RL), L task For RL policy loss, for example, the PPO loss function can be used. L PPO The total loss function is: L total = L PPO + l · Lifsr .

[0071] Moreover, in this embodiment, the model to be trained corresponding to the current training round is the model to be trained completed in the previous training round, and the sparse feature decoding module is fine-tuned based on the activation data of the model to be trained completed in the previous training round. The current training round obtains at least one layer of sparse feature vectors generated by the model to be trained based on the fine-tuned sparse feature decoding module.

[0072] Specifically, the process of fine-tuning the sparse feature decoding module based on the activation data of the model to be trained in the previous training round is as follows: 1. Data preparation: Collect activation data of the model to be trained from the previous training round on general pre-training data and task-specific data.

[0073] 2. Fine-tuning objective: Minimize reconstruction error plus sparsity penalty: (7).

[0074] in, h This indicates the actual activation of the model to be trained. This represents the reconstructed output of the transcoder or SAE. This represents the reconstruction error term, ensuring that the transcoder or SAE can accurately reconstruct the original activation. f Represents sparse feature vectors. Representing the L1 norm encourages sparse feature vectors. f Most of the elements are zero. l sparse The sparsity penalty coefficient is used to balance reconstruction accuracy and feature sparsity. l sparse The larger the value, the sparser the generated feature vector. l sparse The smaller the value, the higher the reconstruction accuracy but the worse the sparsity. l sparse The value ranges from 0.001 to 0.1.

[0075] 3. Mixed data ratio: The ratio of general pre-training data to task data is usually 4:1 or higher to prevent overfitting to task-specific patterns.

[0076] 4. Training rounds: Fine-tuning can usually be completed in 1 to 3 epochs.

[0077] In this embodiment, after each round of training, that is, after training with the total loss function with a penalty term, the activation distribution of the model to be trained will change. The original transcoder or SAE may not be able to accurately reconstruct the new activation pattern. Fine-tuning allows the transcoder or SAE to readjust to the updated model, so that it can accurately identify new error patterns in the next round of training.

[0078] The model training optimization apparatus provided by the present invention is described below. The model training optimization apparatus described below can be referred to in correspondence with the model training optimization method described above.

[0079] The model training optimization device of this invention embodiment is as follows: Figure 2 As shown, it includes the following units: The sparse feature vector generation unit 210 is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained by using a pre-built sparse feature decoding module during the forward inference process based on the attribution set of the model to be trained.

[0080] The penalty term generation unit 220 is used to generate a penalty term for the model to be trained during training based on the sparse feature vector.

[0081] The model training unit 230 is used to train the model to be trained based on the training set, and the training of the model to be trained is completed when the total loss function converges.

[0082] The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

[0083] In some embodiments, the sparse feature decoding module is a transcoder, and the input and output of the transcoder are the same as the input and output of the corresponding replacement layer.

[0084] The sparse feature vector generation unit 210 specifically includes: A layer replacement unit is used to replace at least one layer of the model to be trained with a sparse feature decoding module to obtain a replacement model; The sparse feature vector acquisition unit is used to acquire the sparse feature vector generated by the sparse feature decoding module during the forward inference process based on the attribution set of the alternative model.

[0085] In some embodiments, the penalty term generation unit 220 specifically includes: The attribution graph construction unit is used to construct an attribution graph for each sample data in the attribution set, based on the corresponding sparse feature vector as nodes and the association relationship between sparse feature vectors as edges. The first attribution set classification unit is used to classify the sample data with correct inference results in the attribution set into the correct sample set, and the sample data with incorrect inference results into the incorrect sample set. The edge statistics unit is used to calculate the first correct frequency of occurrence of the edge in the correct sample set and the first incorrect frequency of occurrence in the incorrect sample set, based on whether the sample data corresponding to each attribution map of any edge belongs to the correct sample set or the incorrect sample set. The error edge determination unit is used to determine that any edge is an error edge when the first error frequency of any edge is greater than or equal to the first error frequency threshold and the first correct frequency is less than or equal to the first correct frequency threshold. The first penalty term determination unit is used to determine the penalty term of the model to be trained during training based on the two sparse feature vectors corresponding to the erroneous edge and the weight of the erroneous edge.

[0086] In some embodiments, the error edge determination unit includes: The candidate edge determination unit is used to determine all edges whose first error frequency is greater than or equal to the first error frequency threshold and whose first correct frequency is less than or equal to the first correct frequency threshold as candidate edges. A ratio calculation unit is used to calculate the ratio of the first error frequency to the first correct frequency for each candidate edge; The error edge selection unit is used to sort the edges by ratio from largest to smallest and select the candidate edges corresponding to the top K ratios as the error edges, where K is greater than or equal to 2.

[0087] In some embodiments, the sparse feature decoding module is a sparse autoencoder; The sparse feature vector generation unit 210 specifically includes: An intermediate vector acquisition unit is used to acquire intermediate vectors of the output of the at least one layer based on hooks pre-injected in at least one layer during the forward inference process of the model to be trained based on the attribution set. A sparse feature vector conversion unit is used to convert the intermediate vector into the sparse feature vector using a sparse autoencoder.

[0088] In some embodiments, the penalty term generation unit 220 specifically includes: The second attribution set classification unit is used to classify the sample data with correct inference results in the attribution set into the correct sample set, and the sample data with incorrect inference results into the incorrect sample set. The sparse feature vector statistics unit is used to calculate the second correct frequency of activation of any sparse feature vector in the correct sample set and the second incorrect frequency of activation in the incorrect sample set, based on whether the sample data corresponding to any sparse feature vector belongs to the correct sample set or the incorrect sample set. An error feature vector determination unit is used to determine any sparse feature vector as an error feature vector based on a second correct frequency and a second error frequency. The second penalty term determination unit is used to determine the penalty term for the model to be trained during training based on the error feature vector and the corresponding weight.

[0089] In some embodiments, the sparse feature vector generation unit 210 to the model training unit 230 are repeatedly executed to perform multiple rounds of training. The model to be trained corresponding to the current training round is the model to be trained completed in the previous training round. The sparse feature decoding module is fine-tuned based on the activation data of the model to be trained completed in the previous training round. The current training round obtains at least one layer of sparse feature vectors generated by the model to be trained based on the fine-tuned sparse feature decoding module.

[0090] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a model training optimization method, which includes: S1: During the forward inference process of the model to be trained based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained.

[0091] S2: Generate a penalty term for the model to be trained during training based on the sparse feature vector.

[0092] S3: Train the model to be trained based on the training set. The training of the model to be trained is completed when the total loss function converges.

[0093] The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

[0094] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the model training and optimization methods provided by the above methods, the method comprising: S1: During the forward inference process of the model to be trained based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained.

[0096] S2: Generate a penalty term for the model to be trained during training based on the sparse feature vector.

[0097] S3: Train the model to be trained based on the training set. The training of the model to be trained is completed when the total loss function converges.

[0098] The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

[0099] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the model training and optimization methods provided by the methods described above, the method comprising: S1: During the forward inference process of the model to be trained based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained.

[0100] S2: Generate a penalty term for the model to be trained during training based on the sparse feature vector.

[0101] S3: Train the model to be trained based on the training set. The training of the model to be trained is completed when the total loss function converges.

[0102] The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] All actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training optimization method, characterized in that, include: S1: During the forward inference process of the model to be trained based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained. S2: Generate a penalty term for the training model based on the sparse feature vector; S3: Train the model to be trained based on the training set. The training of the model to be trained is completed when the total loss function converges. The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

2. The model training optimization method according to claim 1, characterized in that, The sparse feature decoding module is a transcoder, and the input and output of the transcoder are the same as the input and output of the corresponding replacement layer; During the forward inference process of the model under training based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model under training, including: The sparse feature decoding module is used to replace at least one layer of the model to be trained to obtain an alternative model; During the forward inference process based on the attribution set in the alternative model, sparse feature vectors generated by the sparse feature decoding module are obtained.

3. The model training optimization method according to claim 2, characterized in that, Based on the sparse feature vectors, a penalty term is generated for the training model, including: For each sample data in the attribution set, an attribution graph is constructed based on the corresponding sparse feature vector as a node and the relationship between sparse feature vectors as an edge. The sample data with correct inference results in the attribution set are classified as the correct sample set, and the sample data with incorrect inference results are classified as the incorrect sample set; Based on whether the sample data corresponding to each attribution map of any side belongs to the correct sample set or the incorrect sample set, the first correct frequency of the occurrence of the stated side in the correct sample set and the first incorrect frequency of the occurrence of the stated side in the incorrect sample set are statistically analyzed. If the first error frequency of either side is greater than or equal to the first error frequency threshold and the first correct frequency is less than or equal to the first correct frequency threshold, then either side is determined to be an error side. Based on the two sparse feature vectors corresponding to the erroneous edge and the weight of the erroneous edge, the penalty term for the training model is determined.

4. The model training optimization method according to claim 3, characterized in that, Determining any edge as an erroneous edge when the first error frequency of either edge is greater than or equal to a first error frequency threshold and the first correct frequency is less than or equal to a first correct frequency threshold includes: All edges whose first error frequency is greater than or equal to the first error frequency threshold and whose first correct frequency is less than or equal to the first correct frequency threshold are identified as candidate edges; Calculate the ratio of the first error frequency to the first correct frequency for each candidate edge; Sort the edges by ratio from largest to smallest, and select the candidate edges corresponding to the top K ratios as the erroneous edges, where K is greater than or equal to 2.

5. The model training optimization method according to claim 1, characterized in that, The sparse feature decoding module is a sparse autoencoder; During the forward inference process of the model under training based on the attribution set, a pre-built sparse feature decoding module is used to obtain sparse feature vectors related to at least one layer of the output vector of the model under training, including: During the forward inference process of the model to be trained based on the attribution set, the intermediate vector of the output of the at least one layer is obtained based on the hook injected in advance in at least one layer; The intermediate vector is converted into the sparse feature vector using a sparse autoencoder.

6. The model training optimization method according to claim 5, characterized in that, Based on the sparse feature vectors, a penalty term is generated for the training model, including: The sample data with correct inference results in the attribution set are classified as the correct sample set, and the sample data with incorrect inference results are classified as the incorrect sample set; Based on whether the sample data corresponding to any sparse feature vector belongs to the correct sample set or the incorrect sample set, the second correct frequency of activation of the sparse feature vector in the correct sample set and the second incorrect frequency of activation in the incorrect sample set are calculated. The sparse feature vector is determined to be an erroneous feature vector based on the second correct frequency and the second incorrect frequency. Based on the error feature vector and its corresponding weight, the penalty term for the model to be trained during training is determined.

7. The model training optimization method according to any one of claims 1 to 6, characterized in that, Repeat steps S1 to S3 for multiple rounds of training. The model to be trained in the current training round is the model to be trained completed in the previous training round. The sparse feature decoding module is fine-tuned based on the activation data of the model to be trained completed in the previous training round. The current training round obtains at least one layer of sparse feature vectors generated by the model to be trained based on the fine-tuned sparse feature decoding module.

8. A model training optimization device, characterized in that, include: The sparse feature vector generation unit is used to obtain sparse feature vectors related to at least one layer of the output vector of the model to be trained by using a pre-built sparse feature decoding module during the forward inference process based on the attribution set of the model to be trained. The penalty term generation unit is used to generate a penalty term for the training model based on the sparse feature vector. The model training unit is used to train the model to be trained based on the training set, and the training of the model to be trained is completed when the total loss function converges. The total loss function is obtained by weighted summation of the original loss function of the model to be trained and the penalty term.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the model training optimization method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training optimization method as described in any one of claims 1 to 7.

Images