Frequency adaptive feature selection method and system based on sparse automatic encoder

Abstract

The invention discloses a frequency adaptive feature selection method and system based on a sparse automatic encoder. The method provided by the invention comprises the following steps of: dynamically tracking an activation frequency estimation value of each feature during a training period; a reverse bias value is generated for each feature according to the estimated value, the higher the frequency is, the negative the bias value is, and otherwise, the positive the bias value is; before Top-K feature selection, the bias is applied to the original activation value to adjust the selection probability. The core innovation of the invention lies in that the bias is only used for guiding feature selection, and the calculation for model reconstruction still uses the original activation value. The method effectively solves the problem of feature absorption, and helps to learn a more complete and more diversified single-sense feature dictionary on the premise of not sacrificing model reconstruction fidelity, thereby improving the accuracy and effectiveness of model interpretability analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and machine learning, more particularly, to the technical field of explainability of artificial intelligence models, and especially relates to a frequency adaptive feature selection method and system based on a sparse autoencoder (SAE). BACKGROUND

[0002] In recent years, artificial intelligence technology represented by large language models (LLMs) has made breakthrough progress, but the "black box" problem of its internal working mechanism has become increasingly prominent, posing a serious challenge to the reliability, security and trustworthiness of the model. In response to this challenge, mechanistic interpretability (MI) has emerged as a new research field, with the core goal of reverse engineering neural networks to understand their internal algorithms and data structures. In the study of mechanistic interpretability, a core challenge stems from the polysemanticity and superposition phenomena within neural networks, i.e., a single neuron may respond to multiple unrelated concepts, and the model will superimpose features far exceeding its neuron count in a linear combination in the same activation space, making traditional methods that analyze individual neurons ineffective.

[0003] To solve the above problems, sparse autoencoder (SAE) as a powerful unsupervised learning method has been proven to be one of the most promising core tools, which aims to decompose the dense and superimposed activations within the model into high-dimensional, sparse and unambiguous feature representations. However, in the training process of existing SAEs, especially for advanced architectures that use TopK activation mechanism (i.e., only the top K features with the largest activation values are selected at each step), researchers have found a ubiquitous technical bottleneck, "feature absorption". This problem is caused by the severe imbalance in the frequency of concepts in the data, leading to the continuous victory of a small number of features corresponding to high-frequency concepts in the TopK competition, and forming a positive feedback loop through gradient descent, thereby systematically suppressing a large number of features corresponding to low-frequency but equally important concepts. Ultimately, a large number of meaningful low-frequency features become "dead features" due to continuous defeat, resulting in an incomplete and biased feature dictionary learned, severely limiting the ability to conduct in-depth and comprehensive analysis of large language models, and constituting a key obstacle in current mechanistic interpretability research. SUMMARY

[0004] The present application aims to solve the problem of feature absorption caused by continuous inhibition of low-frequency features by high-frequency features in the training of existing sparse autoencoders, which seriously reduces the completeness and diversity of the learned feature dictionary, and provides a frequency adaptive feature selection method and system based on sparse autoencoders.

[0005] In order to achieve the above-mentioned application purposes, the present application specifically adopts the following technical solutions:

[0006] In a first aspect, the present application provides a frequency adaptive feature selection method based on sparse autoencoders, which includes the following steps:

[0007] S1, obtain a sparse autoencoder with Top-K activation mechanism as a text sparse autoencoder, embed the text sparse autoencoder before the attention sublayer in a Transformer block of a pre-trained large language model of the Transformer architecture to form a text processing model;

[0008] S2, input a batch of text data into the text processing model for one iteration training, in the forward propagation stage of the training, the text sparse autoencoder takes the activation vector containing the context information of the text data extracted by the Transformer block as input, the feature dictionary learned inside is related to the semantic of the text data, the latent features in the feature dictionary are used to perform sparse linear approximation on the activation vector, after continuously tracking the activation state of each latent feature, the exponential moving average method is used to update the activation frequency estimation value of each latent feature in the latent feature set in real time based on the activated latent feature set;

[0009]

[0009] S3, for each latent feature, calculate a frequency adaptive bias value according to its corresponding activation frequency estimation value, then form a bias vector with the same dimension as the feature dictionary from the calculated bias value; the encoder of the text sparse autoencoder performs linear transformation on the activation vector to generate a high-dimensional original pre-activation vector, each element in the original pre-activation vector represents the original activation value of the corresponding latent feature before the ReLU activation function processing;

[0010] S4, add the bias vector and the original pre-activation vector element by element to obtain a bias-adjusted pre-activation vector as an adjusted pre-activation vector, each element in the adjusted pre-activation vector represents the adjusted activation value of the corresponding latent feature;

[0011] S5, execute Top-K algorithm on the adjusted pre-activation vector to find K elements with the largest adjusted activation values as winning elements, and take the latent features corresponding to the winning elements as winning features, obtain the index of each winning feature in the feature dictionary to form an index set of the winning features;

[0012] S6, extract all original activation values corresponding to the index set from the original pre-activation vector, and apply the ReLU activation function to the extracted original activation values as the processed activation values, to obtain a sparse activation vector, which is used for the decoder of the text sparse auto-encoder to reconstruct;

[0013] S7, input the sparse activation vector into the decoder of the text sparse auto-encoder to complete the reconstruction calculation of the activation vector, and take the mean square error between the reconstruction result and the activation vector as the reconstruction loss, update the learnable parameters of the text sparse auto-encoder based on the minimum reconstruction loss through the back propagation algorithm, and the large language model parameters are fixed, so as to complete a complete iteration training;

[0014] S8, constantly iterate S2~S7, and obtain the trained text processing model when the iteration times reach the preset maximum iteration times.

[0015] On the basis of the above scheme, each step can be implemented in the following preferred specific manner.

[0016] As a preferred embodiment of the first aspect, in S3, the bias value of each latent feature is inversely proportional to the activation frequency estimation value of the latent feature.

[0017] As a preferred embodiment of the first aspect, in S3, the bias value of the i-th latent feature is obtained by multiplying the activation frequency difference value and the preset bias intensity hyperparameter, and the activation frequency difference value is the difference between the preset target activation frequency and the activation frequency estimation value of the latent feature; wherein the bias intensity hyperparameter is a positive value. As a preferred embodiment of the first aspect, the target activation frequency is set to the ratio of the number of latent features in the latent feature set to the total number of latent features.

[0018] As a preferred embodiment of the first aspect, in S6, the sparse activation vector only has non-negative values at all positions pointed to by the index set, and the values at the remaining positions are zero.

[0019] As a preferred embodiment of the first aspect, the encoder and the decoder of the sparse auto-encoder have the same structure, both of which are a linear layer.

[0020] The second aspect, the present application provides a text processing system based on sparse auto-encoder, which comprises:

[0021] The model acquisition module acquires the text processing model trained by the frequency adaptive feature selection method according to any one of the schemes of the first aspect;

[0022]

[0023] ​The result acquisition module is used to obtain prompt words containing the target text and task instructions, and input these prompt words into the trained text processing model. The prompt words are processed through an embedding layer and a preprocessor layer. After the first Transformer block processes the data, the second... The activation vectors extracted from each Transformer block are input into the text sparse autoencoder to activate some latent features in the text sparse autoencoder's feature dictionary. Different activation values ​​are assigned to the activated latent features, while the remaining latent features are assigned an activation value of 0. The activation values ​​are then used as weights to perform a weighted summation of all activated latent features to obtain a sparse linear approximation of the activation vector, which is then used as the output of the text sparse autoencoder. This sparse linear approximation is then processed... After processing by the first Transformer block, the second... The semantic vectors output by each Transformer block are passed through a linear layer for linear projection to obtain Logits vectors. The Softmax function then transforms these Logits vectors into the probability distribution of the next token and outputs it, thus achieving the processing of target text under a specified task. This indicates the number of Transformer blocks in the text processing model.

[0024] As a preferred embodiment of the second aspect above, when the task instruction type is summary generation, the output of the text processing model is a summary generated based on the text to be processed;

[0025] Alternatively, when the task instruction type is translation, the output of the text processing model is the result of translating the target text;

[0026] Alternatively, when the task instruction type is question-and-answer, the output of the text processing model is the answer generated based on the target text;

[0027] Alternatively, when the task instruction type is keyword extraction, the output of the text processing model is the keywords or key phrases in the target text;

[0028] Alternatively, when the task instruction type is continuation writing, the output of the text processing model is new content generated based on the target text;

[0029] Alternatively, when the task instruction type is classification, the output of the text processing model is the classification category of the target text.

[0030] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the frequency-adaptive feature selection method based on a sparse autoencoder as described in any of the solutions of the first aspect above.

[0031] In a fourth aspect, the present application provides a computer electronic device, comprising a memory and a processor;

[0032] The memory is configured to store a computer program.

[0033] The processor is configured to implement the method for frequency adaptive feature selection based on sparse autoencoder when executing the computer program.

[0034] Compared with the prior art, the technical scheme provided by the present application has the following remarkable beneficial effects:

[0035] First, effectively counteracts feature absorption and promotes the equalization of activation frequencies. The frequency adaptive bias mechanism introduced by the present application can dynamically provide selection advantages for low-frequency features while suppressing the dominance of high-frequency features, thereby breaking the positive feedback cycle that leads to feature absorption. This makes the activation frequency distribution in the feature dictionary more uniform, creating conditions for the model to learn a more fair and diverse feature representation.

[0036] Second, a more complete and rich feature dictionary is obtained. By giving low-frequency features more activation opportunities, the present application can effectively prevent them from being too low in activity due to continuous competition failure during training. This enables the sparse autoencoder to discover and retain a larger number of monosemantic features representing fine-grained or rare concepts, greatly improving the completeness and diversity of the feature dictionary.

[0037] Third, while improving interpretability, it maintains high reconstruction fidelity. The present application innovatively separates the feature selection process from the numerical reconstruction process, with bias only affecting selection decisions, while the activation values actually used for reconstruction remain in their original, unmodified state. This design ensures that intervention in the feature activation distribution does not introduce additional reconstruction errors, thereby achieving an improvement in interpretability without sacrificing model representation capabilities.

[0038] Fourth, it comprehensively enhances the effectiveness of sparse autoencoder as a mechanical interpretability tool. A more balanced and complete feature dictionary means that researchers can obtain a more comprehensive and less biased view of the internal computer mechanisms of the model. This enables subsequent analysis based on SAE, such as circuit discovery and causal intervention, to be built on a more reliable foundation, thereby providing stronger technical support for in-depth and accurate understanding and control of complex artificial intelligence systems. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The present application provides a method for the step flowchart;

[0040] Figure 2A flowchart for training a sparse autoencoder for the method of the present application;

[0041] Figure 3 A conceptual diagram of the core idea of the method of the present application;

[0042] Figure 4 A system block diagram of the method of the present application. DETAILED DESCRIPTION

[0043] In order to make the above objectives, characteristics and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to provide a sufficient understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0044] In the description of the present application, it should be understood that the terms "first", "second" are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features.

[0045] The present application aims to provide a method for optimizing the Top-K feature selection process through frequency adaptive bias in sparse autoencoder (SAE) training to solve the feature absorption problem. The specific implementation steps of the method and the beneficial effects brought about by the method will be described in detail below. As shown in Figure 1 The figure shows the complete flow of the frequency adaptive feature selection method based on sparse autoencoder provided by the embodiments of the present application, which specifically includes the following steps:

[0046] S1, obtain a sparse autoencoder using a Top-K activation mechanism as a text sparse autoencoder, embed the text sparse autoencoder before the attention sublayer in a Transformer block of a pre-trained large language model of the Transformer architecture to form a text processing model.

[0047] S2, input a batch of text data into the text processing model for one iteration training, in the forward propagation stage of training, the text sparse autoencoder takes the activation vector containing the context information of the text data extracted by the Transformer block as input, the feature dictionary learned by the text sparse autoencoder is related to the semantic of the text data, the latent features in the feature dictionary are used to perform sparse linear approximation on the activation vector, after continuously tracking the activation state of each latent feature, the exponential moving average method is used to update the activation frequency estimation value of each latent feature in the latent feature set based on the activated latent feature set.

[0048] It should be noted that in S2 of the present application, the above-mentioned text data can come from an original data set. The original data set here can use existing text data sets such as The Pile, OpenWebText, etc. Other text data sets can also be selected, as long as the overall architecture of the model can adapt to the text sparse autoencoder of the present application, which is not limited in the present application.

[0049] It should be noted that in S2 of the present application, the above-mentioned text data can come from an original data set. The original data set here can use existing text data sets such as The Pile, OpenWebText, etc. Other text data sets can also be selected, as long as the overall architecture of the model can adapt to the text sparse autoencoder of the present application, which is not limited in the present application.

[0050] In the context of SAE, the feature dictionary is the weight matrix of the decoder layer. Each column of this matrix (or the weight vector corresponding to each neuron) can be viewed as a "feature dictionary atom" or "basis vector," and each basis vector represents a latent feature. Here, a latent feature refers to a basic pattern, structure, or concept existing in the data, learned and represented by each neuron in the hidden layer (latent space). In other words, each latent feature corresponds to a neuron in the hidden layer, and the activation value of this neuron represents the degree to which the aforementioned activation vector contains that latent feature. The activation frequency estimate is an estimate of the actual average activation probability of the neuron across the entire training set (or a batch of training data used in a training iteration), used to characterize the recent activation trend of each latent feature. During training, the activation frequency estimate is dynamically updated with each batch of training data processed (e.g., using an exponential moving average method).

[0051] In a preferred embodiment, the training data type is text data, and the process of tracking the activation frequency estimates is implemented using the Exponential Moving Average (EMA) method. Specifically, after processing each batch of training data, it is determined which latent features were activated (i.e., entered the Top-K list) in that batch of training data. For the... One latent feature, its activation frequency estimate The update formula is:

[0052]

[0053] in, It is a small decay factor; It is the first The estimated activation frequency of each latent feature in the previous iteration; It is an indicator variable, if the first If any latent feature is activated in this batch of training data, then It is 1 if it is not 0 otherwise, thus smoothly reflecting the recent activation trend of each potential feature.

[0054] S3. For each latent feature, a frequency-adaptive bias value is calculated based on its corresponding activation frequency estimate. The calculated bias value forms a bias vector with the same dimension as the feature dictionary. The encoder of the text sparse autoencoder performs a linear transformation on the activation vector to generate a high-dimensional original pre-activation vector. Each element in the original pre-activation vector represents the original activation value of the corresponding latent feature before it is processed by the ReLU activation function.

[0055] It should be noted that in the S3 of the present application, for each potential feature, a frequency adaptive bias value is calculated according to its corresponding activation frequency estimation value through a preset mathematical relationship, and a bias vector with the same dimension as the feature dictionary is formed by the calculated bias value; wherein the size of the bias value is inversely proportional to the activation frequency estimation value of the potential feature, aiming to apply a positive selection incentive to the low-frequency potential feature which is in a long-term competitive disadvantage, and at the same time apply a negative selection inhibition to the high-frequency potential feature which has a dominant position. Subsequently, the encoder of the text sparse automatic encoder performs linear transformation processing on the obtained activation vector to generate a high-dimensional original pre-activation vector, and each element in the original pre-activation vector represents the original activation value of the corresponding potential feature before being processed by the nonlinear ReLU activation function.

[0056] It should be noted that in the S3 of the present application, the bias value of the first potential feature is obtained by multiplying the activation frequency difference value and the preset bias intensity hyperparameter, and the activation frequency difference value is the difference between the preset target activation frequency and the activation frequency estimation value of the potential feature; wherein the bias intensity hyperparameter is a positive value.

[0057] Further, the bias value of the first potential feature is calculated according to the following formula:

[0058]

[0059] Wherein, β is the preset bias intensity hyperparameter; and is the preset target activation frequency.

[0060] In a specific embodiment, based on the activation frequency estimation value of each potential feature tracked in real time in step S2, a bias vector B is calculated and generated , and the dimension of the vector is the same as the number of potential features of the text sparse automatic encoder (i.e. the same as the dimension of the above-mentioned feature dictionary). The first element of the bias vector is the bias value of the first potential feature, and its calculation formula is as shown in the above formula, which will not be described again. In the above formula, the bias intensity hyperparameter β serves as a bias intensity coefficient for controlling the amplitude of bias adjustment, and the value can be selected and set by those skilled in the art according to actual needs, and in the present embodiment, the above-mentioned bias intensity hyperparameter takes 0.1; and the preset target activation frequency

[0061] ​​​​​​, which represents the average activation level that each potential feature should have in an ideal state. In a preferred embodiment, the target activation frequency is set as the ratio of the number of potential features in the potential feature set to the total number of potential features: , is the number of Top-K selection, is the total number of potential features.

[0062] According to this formula, when the activation frequency estimate value of a potential feature is lower than the target activation frequency, it will obtain a positive bias value, and vice versa, thereby achieving a dynamic adjustment inversely proportional to the activation frequency estimate value.

[0063] It should be noted that in S3 of the present application, the encoder of the text sparse autoencoder is a linear layer.

[0064] S4, the bias vector generated in S3 is added to the original pre-activation vector element by element to obtain a bias-adjusted pre-activation vector as the adjusted pre-activation vector, and each element in the adjusted pre-activation vector represents the adjusted activation value of the corresponding potential feature.

[0065] It should be noted that in S4 of the present application, the bias vector generated in S3 step contains the dynamic adjustment of the selection probability of all potential features is linearly added element by element to the original pre-activation vector containing the original activation value, thereby generating a bias-adjusted pre-activation vector , that is, the adjusted pre-activation vector, the value distribution of which has been modulated by the activation frequency estimate value. The only purpose of the adjusted pre-activation vector is to be used in the subsequent competitive feature selection step, which is the core operation of the present application to intervene in the feature selection process.

[0066] S5, performing Top-K algorithm on the adjusted pre-activation vector to find the K elements with the largest adjusted activation values as winning elements, and taking the potential features corresponding to the winning elements as winning features, obtaining the index of each winning feature in the feature dictionary to form an index set of the winning features.

[0067] It should be noted that step S5 of the present application aims to apply the standard Top-K algorithm to the bias-adjusted pre-activation vector obtained in step S4 to competitively screen out and determine a preset number (K) of potential features (i.e. winning features) with the largest adjusted activation values. These winning features represent the K potential features that ultimately win after considering the original activation value and the bias value in the forward propagation, so the final output is the index set of these winning features in the feature dictionary .

[0068] S6, extracting all original activation values corresponding to the index set from the original pre-activation vector, taking the extracted original activation values as to-be-processed activation values, and obtaining a sparse activation vector by applying a ReLU activation function to all to-be-processed activation values, the sparse activation vector being used for decoder reconstruction of the text sparse autoencoder; wherein the sparse activation vector only has non-negative values at all positions pointed to by the index set, and the values at the remaining positions are zero.

[0069] It should be noted that S6 of the present application aims to perform the decoupling construction step of feature selection and reconstructed numerical value. The present application does not use the adjusted activation values in the pre-activation vector but uses the index set obtained in step S5 , returns to the original pre-activation vector obtained in step S3 without any bias adjustment , and accurately extracts the original activation values at the K positions pointed to by by index operation. Then, these extracted original activation values are taken as to-be-processed activation values and passed through a nonlinear activation function (such as ReLU) to ensure that the values at these positions are non-negative and all the remaining positions are zero, thereby constructing the final sparse activation vector , which is used for decoder reconstruction of the original input of the SAE. In this way, the present application successfully separates the bias affecting feature "selection" from the activation value determining the "numerical value" of reconstruction, achieving optimization of the feature activation distribution without introducing interference to the reconstruction ability of the model.

[0070] S7, inputting the sparse activation vector into the decoder of the text sparse autoencoder to complete the reconstruction calculation of the activation vector, taking the mean square error between the reconstruction result and the activation vector as the reconstruction loss, updating the learnable parameters of the text sparse autoencoder based on minimizing the reconstruction loss through the back propagation algorithm, and fixing the parameters of the large language model, thereby completing one complete iteration training.

[0071] It should be noted that in S7 of the present application, the decoder of the text sparse autoencoder is a linear layer. After the above steps S2~S7, the text sparse autoencoder completes one iteration training.

[0072] S8, constantly iterating S2~S7, and obtaining a trained text processing model when the number of iterations reaches the preset maximum number of iterations.

[0073] ​It should be noted that the core innovation of the method of the present application is that a dynamic bias value negatively correlated with the activation frequency estimation value is introduced to adjust the probability of feature selection, and the original activation value without bias is still used in the reconstruction calculation. This separation mechanism of selection and reconstruction is the key to effectively solving the feature absorption problem without sacrificing the model reconstruction fidelity.

[0074] As shown in Figure 2 , the figure shows the core processing flow of the text sparse automatic encoder during training iteration. As shown in Figure 3 , it is a conceptual diagram of the core idea of the present application, Figure 3 which intuitively shows how the present application solves the feature competition problem by separating the feature selection path and the numerical reconstruction path. Take an original pre-activation vector containing a high-frequency latent feature A (original activation value ) and a low-frequency latent feature B (original activation value ) as an example, where . In the illustrated path 1 (selection path), a frequency-adaptive negative bias value is calculated for the high-frequency latent feature A according to the activation frequency estimation value, and a frequency-adaptive positive bias value is calculated for the low-frequency latent feature B. After adding these bias values to the original activation values, the adjusted pre-activation vector for Top-K selection is obtained. In this vector, the adjusted activation value may be less than , so that the index of the low-frequency latent feature B, which was originally at a disadvantage, wins in the Top-K selection. At the same time, in the illustrated path 2 (reconstruction path), the original pre-activation vector without any modification is completely retained. Finally, in the step of constructing the sparse activation vector, the winning feature index (the index of the low-frequency latent feature B) selected by path 1 is used to extract the unmodified original activation value of the low-frequency latent feature B from the original pre-activation vector retained by path 2, and is used for subsequent decoder reconstruction. This mechanism clearly shows the core innovation of the present application, which separates feature "selection" from reconstruction "numerical value".

[0075] In order to better show the specific implementation and technical effects of the present application, the frequency-adaptive feature selection method based on sparse automatic encoder shown in steps S1-S8 in the above preferred implementation mode will be applied to a specific example.

[0076] Embodiment

[0077] The frequency-adaptive feature selection method based on sparse automatic encoder used in this embodiment is described above and will not be repeated here.

[0078] To verify the actual effect of the method provided by the present application, the method of the present application is compared with the benchmark, unmodified standard Top-K sparse autoencoder method in a set of controlled experimental environment.

[0079] The experiments of the present embodiment are carried out on a subset of the OpenWebText supported by the National Science Foundation (NSF), which contains 100 million tokens and can represent the complex distribution of natural language in the real world. The experiments use two core indicators to measure the balance of feature activation. The first is the feature activation frequency variance (FAFV), which is used to measure the dispersion of all feature activation frequencies. The smaller the variance, the more uniform the activation frequency. The second is the Gini coefficient (GINI), which is used to quantify the concentration of feature activation frequency. The lower the Gini coefficient, the more evenly distributed the activation frequency.

[0080] After sufficient training, the experimental data clearly shows that the method proposed by the present application has achieved significant results in improving the uniformity of feature activation. The specific experimental results are shown in Table 1:

[0081] Table 1: Comparison of feature activation balance on OpenWebText dataset

[0082]

[0083] As can be seen from Table 1, regarding the feature activation frequency variance (FAFV), the stable value of this indicator is reduced from 0.000010569 of the benchmark method to 0.000006371 of the method of the present application, achieving a significant reduction of 39.7%. More importantly, the peak value at the beginning of training is reduced from 0.000068 to 0.0000094, with a peak value suppressed by 86.2%. These data show that the method can intervene and break the positive feedback loop that leads to the continuous dominance of high-frequency features from the early stages of training. Regarding the Gini coefficient (GINI), the stable value of this indicator is optimized from 72.181% to 71.117%, and the peak value at the beginning of training is significantly reduced from 95.25% to 88.46%, further confirming that the overall balance of feature activation distribution has been improved.

[0084] The quantitative experimental results of this embodiment strongly demonstrate that the present invention can effectively suppress the "feature absorption" phenomenon, promote balanced and sparse activation of features, and thus learn a more complete and diverse feature dictionary. The present invention cleverly intervenes in the Top-K feature selection process in sparse autoencoders by introducing a novel bias mechanism dynamically related to feature activation frequency. Its core contribution lies in effectively solving the feature absorption problem commonly encountered during training without sacrificing model reconstruction fidelity, and significantly improving the balance, diversity, and completeness of the learned feature dictionary.

[0085] It should also be noted that the frequency-adaptive feature selection method based on sparse autoencoders in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a text processing system based on sparse autoencoders, corresponding to the frequency-adaptive feature selection method based on sparse autoencoders provided in the above embodiments, such as... Figure 4 As shown, it includes:

[0086] The model acquisition module acquires a text processing model trained according to the frequency adaptive feature selection method provided in the above embodiments;

[0087] The result acquisition module is used to obtain prompt words containing the target text and task instructions, and input these prompt words into the trained text processing model. The prompt words are processed through an embedding layer and a preprocessor layer. After the first Transformer block processes the data, the second... The activation vectors extracted from each Transformer block are input into the text sparse autoencoder to activate some latent features in the text sparse autoencoder's feature dictionary. Different activation values ​​are assigned to the activated latent features, while the remaining latent features are assigned an activation value of 0. The activation values ​​are then used as weights to perform a weighted summation of all activated latent features to obtain a sparse linear approximation of the activation vector, which is then used as the output of the text sparse autoencoder. This sparse linear approximation is then processed... After processing by the first Transformer block, the second... The semantic vectors output by each Transformer block are passed through a linear layer for linear projection to obtain Logits vectors. The Softmax function then transforms these Logits vectors into the probability distribution of the next token and outputs it, thus achieving the processing of target text under a specified task. This indicates the number of Transformer blocks in the text processing model.

[0088] It should be noted that in the above text processing system, the output result of the text processing model is related to the above task type. Specifically, when the task instruction type is summary generation, the output result of the text processing model is a summary generated based on the text to be processed; or when the task instruction type is translation, the output result of the text processing model is the result of the translation of the target text; or when the task instruction type is question and answer, the output result of the text processing model is an answer generated based on the target text; or when the task instruction type is keyword extraction, the output result of the text processing model is a keyword or key phrase in the target text; or when the task instruction type is continuation, the output result of the text processing model is new content generated based on the target text; or when the task instruction type is classification, the output result of the text processing model is the classification category of the target text.

[0089] In this embodiment, taking the news text "Humanitarian staff coordinated their efforts" as the target text, without the aforementioned text sparse autoencoder, the output of the large language model itself is "Humanitarian staff coordinated their efforts to create safe spaces where mothers could receive prenatal care and essential supplies for their newborns," which translates to "Humanitarian staff coordinated their efforts to create safe spaces where mothers could receive prenatal care and essential supplies for their newborns." With the aforementioned text sparse autoencoder, feature number 39351 in the feature dictionary is related to women. After setting it to 0, the output of the text processing model is "Humanitarian staff coordinated their efforts to distribute food, cleanwater, and medical supplies to all residents of the earthquake-strickentown," which translates to "Humanitarian staff coordinated their efforts to distribute food, cleanwater, and medical supplies to all residents of the earthquake-strickentown." As can be seen, without a text sparse autoencoder, the output is general, neutral, but relatively vague, describing a very common and safe scenario in humanitarian work—caring for mothers and infants. This scenario can occur in any humanitarian crisis (long-term refugees, impoverished areas, post-war reconstruction), lacking specificity for particular emergencies. However, after introducing a text sparse autoencoder using the method of this invention, the output is specific, targeted, and highly scenario-based. It clearly indicates the disaster type (earthquake), the target group (all residents of the town), and the specific action (distributing life-sustaining supplies), significantly improving information density and practicality.

[0090] This example demonstrates that the introduction of a sparse text autoencoder is equivalent to adding a regulator to the "thought process" of a large language model. By intervening in specific latent features activated by the sparse text autoencoder, it is possible to guide the model to generate text in the direction desired by the user. Although there may be no active intervention in this example, the very existence of the sparse text autoencoder alters the distribution of feature activations, making the output no longer a "random" association by the model, but a more constrained and targeted result. This is crucial for building reliable and controllable AI applications.

[0091] In summary, the text processing model containing the text sparse autoencoder effectively suppresses the inherent generalization tendency and "hallucination" risk of the original large language model through its sparse and feature decoupling characteristics, generating more specific, more relevant, higher information density and more practical text. This proves that the text sparse autoencoder of the present application is not only an interpretability tool, but also a potential performance enhancement tool that can make the output of the model more suitable for the needs of specific tasks or contexts.

[0092] It can be understood that the frequency adaptive feature selection method based on sparse autoencoder described in S1-S8 above can be realized by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer program product corresponding to the frequency adaptive feature selection method based on sparse autoencoder provided by the above-mentioned embodiments, which includes computer programs / instructions that can implement the frequency adaptive feature selection method based on sparse autoencoder as described in the above-mentioned embodiments when executed by a processor.

[0093] Similarly, based on the same inventive concept, another preferred embodiment of the present application also provides a computer electronic device corresponding to the frequency adaptive feature selection method based on sparse autoencoder provided by the above-mentioned embodiments, which includes a memory and a processor.

[0094] The memory is used to store computer programs.

[0095] The processor is used to implement the frequency adaptive feature selection method based on sparse autoencoder in the above-mentioned embodiments when executing the computer programs.

[0096] In addition, the logical instructions in the above-mentioned memory can be realized in the form of software functional units and sold or used as independent products when used, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products, which are stored in a storage medium and include instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.

[0097] Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer readable storage medium corresponding to the frequency adaptive feature selection method based on the sparse auto-encoder provided by the above-mentioned embodiment, and the storage medium stores a computer program. When the computer program is executed by a processor, the frequency adaptive feature selection method based on the sparse auto-encoder in the above-mentioned embodiment can be implemented.

[0098] It can be understood that the storage medium described above can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0099] It can be understood that the processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0100] In addition, it should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In each embodiment provided in the present application, the division of steps or modules in the system and method is only a logical functional division, and actual implementation can have another division mode, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0101] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A frequency-adaptive feature selection method based on a sparse autoencoder, characterized in that, Includes the following steps: S1. Obtain a sparse autoencoder that adopts the Top-K activation mechanism as the text sparse autoencoder. Embed the text sparse autoencoder before the attention sublayer in a Transformer block of a pre-trained large language model with a Transformer architecture to form a text processing model. S2. Input a batch of text data into the text processing model for one iteration of training. During the forward propagation phase of training, the text sparse autoencoder takes the activation vector containing the context information of the text data extracted by the Transformer block as input. The feature dictionary learned internally is related to the semantics of the text data. The latent features in the feature dictionary are used to perform sparse linear approximation on the activation vector. After continuously tracking the activation state of each latent feature, the exponential moving average method is used to dynamically update the activation frequency estimate corresponding to each latent feature in the latent feature set in real time based on the set of activated latent features. S3. For each latent feature, a frequency-adaptive bias value is calculated based on its corresponding activation frequency estimate. The calculated bias value forms a bias vector with the same dimension as the feature dictionary. The encoder of the text sparse autoencoder performs a linear transformation on the activation vector to generate a high-dimensional original pre-activation vector. Each element in the original pre-activation vector represents the original activation value of the corresponding latent feature before it is processed by the ReLU activation function. S4. After adding the bias vector to the original pre-activation vector element by element, we get a pre-activation vector that has been adjusted by the bias, which is the adjusted pre-activation vector. Each element in the adjusted pre-activation vector represents the adjusted activation value of the corresponding latent feature. S5. Perform the Top-K algorithm on the adjusted pre-activation vector, find the K elements with the largest adjusted activation values ​​as the winning elements, take the latent features corresponding to the winning elements as the winning features, obtain the index of each winning feature in the feature dictionary, and form the index set of winning features. S6. Extract all original activation values ​​corresponding to the index set from the original pre-activation vector, and use the extracted original activation values ​​as activation values ​​to be processed. After processing all activation values ​​to be processed by the ReLU activation function, a sparse activation vector is obtained, which is used for decoder reconstruction of the text sparse autoencoder. S7. Input the sparse activation vector into the decoder of the text sparse autoencoder to complete the reconstruction calculation of the activation vector, and use the mean square error between the reconstruction result and the activation vector as the reconstruction loss. Based on minimizing the reconstruction loss, update the learnable parameters of the text sparse autoencoder through the backpropagation algorithm. The parameters of the large language model are fixed, thus completing a complete iterative training. S8. Iterate through S2 to S7 until the preset maximum number of iterations is reached, and then obtain the trained text processing model.

2. The frequency-adaptive feature selection method based on a sparse autoencoder as described in claim 1, characterized in that, In S3, the magnitude of the bias value of each latent feature is inversely proportional to the estimated activation frequency of that latent feature.

3. The frequency-adaptive feature selection method based on a sparse autoencoder as described in claim 2, characterized in that, In S3, the first The bias value of a latent feature is obtained by multiplying the activation frequency difference by a preset bias intensity hyperparameter. The activation frequency difference is the difference between the preset target activation frequency and the estimated activation frequency of the latent feature; where the bias intensity hyperparameter is a positive value.

4. The frequency-adaptive feature selection method based on a sparse autoencoder as described in claim 3, characterized in that, The target activation frequency is set as the ratio of the number of latent features in the latent feature set to the total number of latent features.

5. The frequency-adaptive feature selection method based on a sparse autoencoder as described in claim 1, characterized in that, In S6, the sparse activation vector has non-negative values ​​only at all positions pointed to by the index set, and the values ​​at the remaining positions are zero.

6. The frequency-adaptive feature selection method based on a sparse autoencoder as described in claim 1, characterized in that, The encoder and decoder structures of a sparse autoencoder are identical, both consisting of a single linear layer.

7. A text processing system based on a sparse autoencoder, characterized in that, include: The model acquisition module acquires a text processing model trained according to the frequency adaptive feature selection method described in any one of claims 1 to 6; The result acquisition module is used to obtain prompt words containing the target text and task instructions, and input these prompt words into the trained text processing model. The prompt words are processed through an embedding layer and a preprocessor layer. After processing by the first Transformer block, the second... The activation vectors extracted from each Transformer block are input into the text sparse autoencoder to activate some latent features in the text sparse autoencoder's feature dictionary. Different activation values ​​are assigned to the activated latent features, while the remaining latent features are assigned an activation value of 0. The activation values ​​are then used as weights to perform a weighted summation of all activated latent features to obtain a sparse linear approximation of the activation vector, which is then used as the output of the text sparse autoencoder. This sparse linear approximation is then processed... After processing by the first Transformer block, the second... The semantic vectors output by each Transformer block are passed through a linear layer for linear projection to obtain Logits vectors. The Softmax function then transforms these Logits vectors into the probability distribution of the next token and outputs it, thus achieving the processing of target text under a specified task. This indicates the number of Transformer blocks in the text processing model.

8. The text processing system based on a sparse autoencoder as described in claim 7, characterized in that, When the task instruction type is summary generation, the output of the text processing model is a summary generated based on the text to be processed; Alternatively, when the task instruction type is translation, the output of the text processing model is the result of translating the target text; Alternatively, when the task instruction type is question-and-answer, the output of the text processing model is the answer generated based on the target text; Alternatively, when the task instruction type is keyword extraction, the output of the text processing model is the keywords or key phrases in the target text; Alternatively, when the task instruction type is continuation writing, the output of the text processing model is new content generated based on the target text; Alternatively, when the task instruction type is classification, the output of the text processing model is the classification category of the target text.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the frequency adaptive feature selection method based on a sparse autoencoder as described in any one of claims 1 to 6.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the frequency-adaptive feature selection method based on a sparse autoencoder as described in any one of claims 1 to 6 when executing the computer program.

Images