Text classification method and system

Abstract

The application provides a text classification method and system, and belongs to the field of text classification processing. The method comprises the following steps: preprocessing input text, and generating clean text and perturbed text by applying perturbation to the preprocessed text; constructing a dual-view text classification model training method comprising an online network and a target network, and training the text classification model by using the clean text and the perturbed text, wherein a dynamic low-rank attention mechanism and a dual-view consistency loss function are introduced into the text classification model; the clean text and the perturbed text are used as the dual-view input of the text classification model; and in the process of training the classification model, the dual-branch architecture is used to process any input text in parallel. The application solves the problem of instability of the text classification method in a high-noise environment.

Description

Technical Field

[0001] This invention belongs to the field of text classification and processing, and particularly relates to a text classification method and system. Background Technology

[0002] Text classification is one of the fundamental tasks in natural language processing, widely used in areas such as public opinion analysis, spam detection, medical report screening, news topic identification, and content moderation. Its goal is to automatically determine the category of input text, playing a crucial role in downstream information retrieval, content recommendation, and knowledge mining.

[0003] However, real-world text data often contains various noise disturbances, such as spelling errors, OCR recognition errors, homonymous character substitutions, missing punctuation marks, or sentence truncation. This noise can lead to inconsistent input distributions during training and inference, significantly degrading the performance of text classification models. Improving model robustness while maintaining computational efficiency in noisy environments has become a key challenge for current text classification systems.

[0004] Transformer-based pre-trained language models have achieved significant success in text classification tasks in recent years. Their self-attention mechanism can capture long-distance dependencies, thereby effectively improving semantic representation capabilities. However, the computational complexity of self-attention in standard Transformers is... (in n The computational cost is high when processing long texts (where the text length is specified). To reduce computational complexity, researchers have proposed various efficient attention improvement methods, such as the Linformer, Performer, BigBird, and Nyströmformer models. These models balance performance and efficiency to some extent through low-rank approximation or kernel function mapping. However, most of these efficient Transformer methods use a fixed low-rank structure, and their design for the "efficiency-performance" trade-off makes it difficult to consider the semantic complexity of different input samples. A fixed attention rank value limits the model's ability to express complex texts, especially when input information is missing or noise is amplified, which can easily lead to a sharp drop in performance and insufficient robustness. In addition, research on improving model robustness has also attracted attention in the field of natural language processing, such as through adversarial training, data augmentation, or consistency regularization to enhance the model's resistance to disturbances. However, existing robust training strategies are usually independent of improvements to the model structure, and their integration with efficient attention mechanisms is still insufficient. This means that the performance degradation problem of efficient Transformer models in noisy environments has not been effectively solved, and there is an urgent need for a text classification method that simultaneously considers computational efficiency and noise robustness. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, this invention provides a text classification method and system that solves the problem of instability in text classification methods under high-noise environments.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a text classification method, comprising the following steps:

[0007] S1. Preprocess the input text and apply perturbation to the preprocessed text to generate clean text and perturbed text;

[0008] S2. Construct a text classification model that includes an online network and a target network, and train the model using clean and perturbed text. The text classification model incorporates a dynamic low-rank attention mechanism and dual-viewpointing. Figure 1 Consistency loss function; clean text and perturbed text are used as dual-view inputs to the text classification model;

[0009] S3. During the training of the text classification model, a dual-branch architecture is used to process any input text in parallel and output the text classification results.

[0010] Furthermore, both the online network and the target network include an encoder and a projector, and the online network also includes a predictor and a classifier.

[0011] The encoder is used to encode two views using a dynamic low-rank attention mechanism to obtain the encoded result. and In this process, the same multi-layer stacked Transformer structure is used in both the online network and the target network encoders in the dual-branch architecture. The dynamic low-rank attention mechanism is embedded in the self-attention calculation module of each layer of the encoder, and semantic information density is introduced into the dynamic low-rank attention mechanism. The effective rank value can be adaptively determined based on the semantic information density measurement result.

[0012] Projector, used for encoding results Using its corresponding projector, the projection result is obtained. and based on the encoding results Using its corresponding projector, the projection result is obtained. ;

[0013] Predictor, used for the projection results Make a prediction to obtain the prediction vector. ;

[0014] A classifier is used to analyze the encoded results. Perform discrimination to obtain the predicted category probability distribution. This completes the classification process for the text.

[0015] Furthermore, the dual-view encoding is performed using a dynamic low-rank attention mechanism, including the following steps:

[0016] The dual-view data is embedded and represented, where semantic information density is used to reflect the number and differences of semantic features activated by the input dual-view text.

[0017] Based on the embedding representation results, for each attention head, the query matrix Q, key matrix K, and value matrix V of the current layer input are projected into a low rank using the effective rank value determined by the dynamic low rank attention mechanism.

[0018] Based on the low-rank projection results, the outputs of each attention head are spliced ​​and linearly transformed;

[0019] After splicing and linear transformation, a position-by-position nonlinear transformation is performed using a feedforward neural network;

[0020] The results of the nonlinear transformation are aggregated to obtain the encoded result. and .

[0021] Furthermore, the expression for the rank is as follows:

[0022] ;

[0023] ;

[0024] ;

[0025] in, Represents the rank value. and These represent the preset minimum rank and maximum rank, respectively. Represents the normalized result , This represents the variance of the hidden representation. and Minimum variance and maximum variance, respectively. Represents the smoothing constant. This represents the variance of all hidden vectors. Indicates batch size, Indicates the sequence length. Indicates the hidden layer dimension. Indicates the first b The first sample i The first token of the token j Vygote indicates that, This represents the global mean. This represents the hidden layer representation tensor.

[0026] Furthermore, attention is calculated as follows:

[0027] ;

[0028] ;

[0029] in, This represents the final attention weight matrix. express Activation function Represents the query matrix. Represents a low-rank projection matrix. This represents the matrix transpose operation. Indicates the scaling factor. Represents the key matrix. Represents a value matrix, Represents a low-rank projection matrix. This represents the result of the attention calculation.

[0030] Furthermore, the training process of the text classification model includes the following steps:

[0031] Based on clean text and perturbation text, construct a clean view and a perturbation view; the clean view is input into an online network, and the perturbation view is input into a target network;

[0032] Encoders using online networks Encode the clean view to obtain the encoding result. and encoders utilizing the target network Encode the disturbed view to obtain the encoding result. ;

[0033] Based on encoding results Using its corresponding projector, the projection result is obtained. and based on the encoding results Using its corresponding projector, the projection result is obtained. ;

[0034] Projection results The input is given to the predictor to obtain the prediction vector. ;

[0035] Encode the result The data is input into the classifier, which calculates the predicted class probability distribution. ;

[0036] Based on projection results Predicting the probability distribution of categories and prediction vector Define a two-view algorithm that includes classification loss and consistency loss. Figure 1 Consistency loss function;

[0037] Minimize dual views using gradient descent Figure 1 The text classification model is trained by using a consistency loss function to update the online network parameters and by using an exponential moving average to update the target network parameters. The online network parameters include the encoder, projector, predictor, and classifier. After training, only the encoder and classifier in the online network are retained.

[0038] Furthermore, the dual-view Figure 1 The consistency loss function is expressed as follows:

[0039] ;

[0040] ;

[0041] ;

[0042] in, Indicates binocular vision Figure 1 Consistency loss function Represents classification loss. Represents the balance parameters. Indicates consistency loss. Indicates batch size, Indicates the true label, Indicates the online network's response to the sample. Predicted labels, This represents the prediction vector obtained from the perturbation view. Represents cosine similarity. The clean view of the target network encoder represents the projection result obtained by projection.

[0043] The present invention also provides a text classification system, comprising:

[0044] The first processing module is used to preprocess the input text and apply perturbation to the preprocessed text to generate clean text and perturbed text.

[0045] The second processing module is used to construct a text classification model that includes an online network and a target network, and trains the text classification model using clean text and perturbed text. The text classification model incorporates a dynamic low-rank attention mechanism and dual-viewpoint processing. Figure 1 Consistency loss function; clean text and perturbed text are used as dual-view inputs to the text classification model;

[0046] The third processing module is used to process any input text in parallel using a dual-branch architecture during the training of the text classification model, and output the text classification results.

[0047] The beneficial effects of this invention are:

[0048] To address the problems existing in the background technology, this invention proposes a dynamic low-rank attention and dual-view approach for text classification. Figure 1 A text classification method and system combining consistency constraints. Unlike previous efficient Transformers with fixed low-rank structures, the Dynamic Low-Rank Attention (DLA) mechanism proposed in this invention can adaptively adjust the attention rank based on the variance information of the input features: for semantically complex samples with high information density, a higher rank is assigned to enhance expressive power; for simple samples, the rank is reduced to maintain near-linear computational complexity, thereby achieving a balance between representational flexibility and efficiency without significantly increasing computational overhead. Simultaneously, this invention introduces dual-view... Figure 1 The consistency constraint mechanism constructs two views of the same text—clean and perturbed—and constrains their semantic representations to be consistent, suppressing representation shifts caused by noise and further improving the model's robustness. Combining these two approaches will enhance the robustness of the dual-view model. Figure 1 The consistency mechanism is deeply integrated into the dynamic low-rank attention structure, which enables consistency optimization to not only constrain the consistency of semantic representation, but also guide the adaptive adjustment of attention rank, thereby achieving a balance between robustness and efficiency at the structural design level. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention.

[0050] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0051] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0052] Example 1

[0053] like Figure 1 As shown, this invention provides a text classification method, the implementation of which is as follows:

[0054] S1. Preprocess the input text and apply perturbation to the preprocessed text to generate clean text and perturbed text.

[0055] In this embodiment, the input text is preprocessed, for example, by segmenting the text using a word segmenter for the corresponding language and truncating or padding the sequence to a fixed length. Then, a perturbation function (noise injection function) is introduced to perturb the text, thereby generating a text version contaminated with noise. The purpose of this perturbation function is to simulate various real-world noises without altering the overall semantics of the text, thereby improving the robustness of the text classification model. The perturbation function can disrupt the text to varying degrees according to a set perturbation strength parameter, including but not limited to the following types:

[0056] Character-level perturbation: Randomly scramble or replace characters in the text, such as replacing characters with other letters adjacent to the keyboard, replacing them with characters that look similar (e.g., replacing the letter O with the number 0), simulating character errors caused by OCR recognition errors, or randomly shuffling the order of some characters.

[0057] Word / text-level perturbation: Introducing noise at the word level of the text, such as randomly swapping word positions, randomly deleting certain words, or truncating text fragments to simulate incomplete input.

[0058] Symbol and formatting perturbations: Adjust the formatting symbols of the text, such as removing all punctuation marks, uniformly converting letters to lowercase, inserting extra spaces or special symbols and emojis, etc.

[0059] The above-described perturbation function produces a text version that has the same semantic content as the original text but includes noise perturbations. The perturbation intensity of the perturbation function can be predefined at different levels to control the range and magnitude of the perturbation (e.g., from minor spelling errors to severe text loss). This step yields two text input formats: clean text (the preprocessed original text) and perturbed text (the variant text processed by the perturbation function).

[0060] S2. Construct a text classification model that includes an online network and a target network, and train the text classification model using clean text and perturbed text. The text classification model incorporates a dynamic low-rank attention mechanism and dual-viewpointing. Figure 1 Consistency loss function; clean text and perturbed text are used as dual-view inputs to the text classification model;

[0061] In this embodiment, both the online network and the target network include an encoder and a projector. The online network also includes a predictor and a classifier.

[0062] The encoder is used to encode two views using a dynamic low-rank attention mechanism to obtain the encoded result. and In this process, the same multi-layer stacked Transformer structure is used in both the online network and the target network encoders in the dual-branch architecture. The dynamic low-rank attention mechanism is embedded in the self-attention calculation module of each layer of the encoder, and semantic information density is introduced into the dynamic low-rank attention mechanism. The effective rank value can be adaptively determined based on the semantic information density measurement result.

[0063] Projector, used for encoding results Using its corresponding projector, the projection result is obtained. and based on the encoding results Using its corresponding projector, the projection result is obtained. ;

[0064] Predictor, used for the projection results Make a prediction to obtain the prediction vector. ;

[0065] A classifier is used to analyze the encoded results. Perform discrimination to obtain the predicted category probability distribution. This completes the classification and processing of the text.

[0066] In this embodiment, a dynamic low-rank attention mechanism is used to encode the dual views, including the following steps:

[0067] The dual-view data is embedded and represented, where semantic information density reflects the number and differences of semantic features activated by the input dual-view text. Based on the embedding representation, for each attention head, a low-rank projection is performed on the query matrix Q, key matrix K, and value matrix V of the current layer input using the effective rank value determined by a dynamic low-rank attention mechanism. Based on the low-rank projection results, the outputs of each attention head are concatenated and linearly transformed. After concatenation and linear transformation, a position-by-position nonlinear transformation is performed using a feedforward neural network. The nonlinear transformation results are aggregated to obtain the encoded result. and .

[0068] In this embodiment, the training process of the text classification model includes the following steps:

[0069] Based on clean and perturbed text, clean and perturbed views are constructed; the clean view is input into an online network, and the perturbed view is input into a target network; the encoder of the online network is utilized. Encode the clean view to obtain the encoding result. and encoders utilizing the target network Encode the disturbed view to obtain the encoding result. Based on encoding results Using its corresponding projector, the projection result is obtained. and based on the encoding results Using its corresponding projector, the projection result is obtained. Projection results The input is given to the predictor to obtain the prediction vector. ; Encode the result The data is input into the classifier, which calculates the predicted class probability distribution. Based on projection results Predicting the probability distribution of categories and prediction vector Define a two-view algorithm that includes classification loss and consistency loss. Figure 1 Consistency loss function; minimize dual-view loss using gradient descent. Figure 1 The text classification model is trained by using a consistency loss function to update the online network parameters and by using an exponential moving average to update the target network parameters. The online network parameters include the encoder, projector, predictor, and classifier. After training, only the encoder and classifier in the online network are retained.

[0070] S3. During the training of the text classification model, a dual-branch architecture is used to process any input text in parallel and output the text classification results.

[0071] In this embodiment, clean text and perturbed text serve as two input views for the text classification model, constructing a dual-view input. Clean text is considered the clean view, representing normal input without noise contamination; perturbed text is considered the perturbed view, representing the corresponding input containing noise. These two views semantically correspond to the same text data, but differ in their surface form. The two-branch structure is structurally consistent, encoding the clean and perturbed views in parallel. By constructing clean and noisy dual views for the same input and feeding them into the text classification model, the dual-view input will be applied subsequently. Figure 1 Consistency is ensured to guarantee that the text classification model produces semantically consistent internal representations for both input formats, thereby enhancing the text classification model's ability to adapt to noise.

[0072] In this embodiment, a Dynamic Low-Rank Attention (DLA) mechanism is introduced into the encoder of the text classification model to achieve a low-rank approximation that adaptively adjusts according to the input complexity during attention computation. Specifically, the encoder adopts a Transformer-based multi-head self-attention structure, but performs low-rank approximation on the self-attention matrix of each input sample to reduce computational complexity. Placement: The Dynamic Low-Rank Attention mechanism is embedded in the self-attention computation module of each layer of the encoder, replacing the full-rank self-attention computation in the standard Transformer. Adjustment Method: To accommodate the semantic complexity of different text samples, this invention dynamically determines the rank value r of the attention approximation for each input. The selection of this rank value r is based on a measure of the semantic information density of the input sentence, adaptively adjusting the expressive power and efficiency of the attention. The semantic density is measured using statistical information of the input features (variance of word embedding vectors). One implementation of this invention is to calculate the average variance of the word vectors of the input sequence in the hidden dimension as a semantic complexity index v, and then linearly map this semantic complexity index to a preset rank range. Within this range, we determine the attention rank r used for this sample. Let... This ensures that the rank *r* changes monotonically with semantic complexity. Through this adaptive adjustment, higher rank values ​​are assigned to text samples with complex semantic content and high information density to increase the expressive power of the attention mechanism and capture more fine-grained dependencies; for samples with simple semantics and low information content, lower rank values ​​are used to reduce computational overhead and maintain near-linear computational complexity. The dynamic low-rank attention mechanism achieves on-demand allocation of computational resources: it dynamically improves the representational power of complex samples without significantly increasing the overall computational cost, thereby achieving a dynamic balance between model efficiency and representational performance.

[0073] In this embodiment, the encoder of the text classification model adopts a multi-layer stacked Transformer structure, with each layer incorporating the aforementioned dynamic low-rank attention mechanism to achieve efficient encoding. Each encoder layer includes a multi-head dynamic low-rank self-attention layer and a feedforward neural network, supplemented by residual connections and layer normalization operations for stable training. The encoding process is as follows: First, the input text sequence (whether clean or perturbed) is embedded, such as mapping words to word vector representations and adding positional encoding, and then fed into the first-layer encoder. Multi-head attention computation is performed in parallel in each layer: For each attention head, a dynamically determined dynamic rank r is used to perform low-rank projection or approximation computation on the query matrix Q, key matrix K, and value matrix V input to the current layer, such as reducing the dimensionality of a sequence representation of length n (where n represents the sequence length) through a projection matrix of size r, thereby reducing the original dimensionality. The attention matrix is ​​approximately as follows and The matrix multiplication improves computational efficiency. The outputs of each attention head are concatenated and linearly transformed before being passed to the feedforward neural network for position-by-position nonlinear transformation. Multi-layer stacking means that this attention computation is performed L times (L being the number of encoding layers), gradually aggregating higher-level semantic features. Dynamic low-rank attention plays a role in each layer, allowing the text classification model to adjust the attention representation dimension according to the actual complexity of the input at that layer. It should be noted that this invention uses the same dynamic low-rank attention Transformer structure (with consistent initial parameters) in both the online network and the encoder of the target network in the dual-branch architecture, thus ensuring compatibility between the two branches in terms of structure and representation space. After computation by the L-layer encoder, the input text is converted into a high-dimensional representation vector sequence containing contextual semantics. This typically uses a representation vector with a special classification label (e.g., the [CLS] label at the beginning of the sequence) or an aggregated representation of the entire sequence as the overall semantic representation of the text for use by subsequent modules.

[0074] In this embodiment, after encoding the input text, the online network branch uses a classifier to discriminate the semantic representation of the text and outputs the corresponding predicted category probability distribution. The specific implementation involves: retrieving the semantic representation of the text generated by the encoder in the online network (the encoded result). ), input into the classifier In Chinese, classifiers typically consist of one or more fully connected neural networks (e.g., a linear transformation plus softmax activation) used to map semantic features to a predefined class space. The output vector is calculated by the classifier. And convert them into probability values ​​for each category using the Softmax function. In this way, the text classification model generates a probability assessment of the input text belonging to each candidate category, where the category with the highest probability is the classification result predicted by the text classification model. During the training phase, the output vector is adjusted using the true labels. y The classification loss is calculated; however, during the inference phase, the label with the highest probability is directly selected as the final classification output.

[0075] In this embodiment, to improve the stability of the text classification model in noisy environments, the present invention introduces dual-view training. Figure 1 A consistency loss function is used to constrain the text classification model to maintain consistency in the internal representations of clean and perturbed view outputs. This dual-view... Figure 1 The consistency loss is constructed based on cosine similarity, imposing a bidirectional constraint on the representations of the two views. Specifically, after encoding by the online network and the target network respectively, representation vectors of the clean view and the perturbed view are obtained. Let the clean view representation output by the encoder of the online network be transformed by projection and predictor to obtain a vector. The perturbation view representation of the encoder output of the target network is projected to obtain a vector. Similarly, for the opposite direction, the perturbation view representation of the encoder in the online network is passed through the predictor to obtain a vector. A clean view of the encoder of the target network is projected to obtain the projection result. The consistency loss is defined as the negative of the cosine similarity between the two sets of corresponding vectors, as shown in the following formula:

[0076] ;

[0077] Here, cos(a,b) represents the cosine similarity between vectors a and b. The above formula constrains the consistency of representations in both the "clean → perturbation" and "perturbation → clean" directions, taking the average to ensure symmetry. If the representations of the two views are completely identical, the cosine similarity is 1, and the corresponding loss term is 0; conversely, if there are differences, a positive loss value is generated. By minimizing this consistency loss, the similarity between the clean and perturbation view representations can be maximized, prompting the text classification model to learn robust feature representations that are insensitive to noise. In other words, dual-view... Figure 1 Consistency loss forces text classification models to ignore irrelevant changes introduced by perturbations, retaining only key information relevant to the task, thus suppressing noise interference on the discrimination boundary of the text classification model at the representation level. As a regularization constraint, this loss effectively prevents text classification models from outputting unstable or even collapsed representations due to noise. During training, it works in conjunction with classification loss to improve the model's generalization performance and robustness.

[0078] In this embodiment, the text classification model is trained using a joint optimization classification task and dual-view... Figure 1 The training process employs consistency constraints and updates the target network parameters using an exponential moving average (EMA) mechanism to ensure stability. The training procedure is as follows:

[0079] (a) Dual-view input preparation: For each training sample, generate a corresponding clean view. x and perturbation view (Referring to the generation process described above), the input pairs that constitute the current batch of the text classification model Clean view x Enter the online network branch, perturbation view Input the target network branch. The encoders of both branches have the same initial parameters and process the input in parallel. x Indicates a clean view. This indicates a perturbed view; here is a clean view. x With clean view X For the same technical features， Disturbance view With perturbation view They share the same technical features.

[0080] (b) Forward Propagation and Loss Calculation: Encoders for Online Networks Encode a clean view to obtain the encoding result. , Represents a clean view, target network encoder Encode perturbation view to obtain encoding result , The perturbation view is represented by two separate inputs into their respective projectors to obtain the projection result. and , This represents a support vector machine, which is then input into the predictor of an online network to obtain predicted vectors. , This indicates a fully connected layer, and the projection result will be... Input classifier Calculate the probability distribution of the predicted categories .

[0081] By defining two loss functions, we obtain dual-view... Figure 1 Consistency loss function:

[0082] Classification loss: Consistency loss (bidirectional cosine similarity): Binocular vision Figure 1 The consistency loss function is: ;

[0083] in, Indicates binocular vision Figure 1 Consistency loss function Represents classification loss, Represents the balance parameters. Indicates consistency loss. This indicates the batch size, which is the number of samples input at one time. Indicates the true label, Indicates the online network's response to the sample. Predicted labels, This represents the prediction vector obtained from the perturbation view. This represents the cosine similarity.

[0084] (c) Parameter optimization and synchronous update: Minimize the dual-look using gradient descent algorithms (such as AdamW). Figure 1 Consistency loss function Update online network parameters This includes the encoder, projector, predictor, and classifier. Target network parameters. Updated using an exponential moving average: ,in, This represents the momentum coefficient (e.g., 0.99), used to smooth target network updates and stabilize text classification model training.

[0085] (d) Inference and Deployment: After training, only the encoder in the online network is retained. With classifier During inference, any input text is calculated through the following process: Output predicted class probability distribution The perturbation view or target network is no longer used in the inference phase. The consistency mechanism only applies to the training process, but gives the model robustness to input perturbations.

[0086] The following section discusses dynamic low-rank attention mechanisms and dual vision. Figure 1 The consistency should be explained in detail.

[0087] The text classification model framework proposed in this invention comprises two core components: first, a dynamic low-rank attention mechanism that achieves a balance between computational efficiency and expressive power through adaptive rank selection; and second, a dual-view... Figure 1 Consistency-based robust training methods enhance the robustness of the model by establishing representation consistency constraints between clean and contaminated inputs.

[0088] In the standard Transformer, the self-attention calculation process is as follows:

[0089] ;

[0090] in, This indicates attention calculation. These represent the query matrix, key matrix, and value matrix, respectively. n Indicates the sequence length. This represents the scaling factor, which requires... Its complexity makes it difficult to apply efficiently in long text tasks. d Indicates the hidden layer dimension. This represents the matrix transpose operation.

[0091] Introducing a low-rank projection matrix Mapping keys and values ​​to a lower-dimensional space yields approximate computations:

[0092] ;

[0093] in, , The attention weight matrix of Linformer (linear transformation attention) is obtained by combining the query matrix Q and the compressed key matrix K. and Representing a low-rank projection matrix, although it effectively reduces complexity, the fixed rank value r makes it difficult to account for the complexity of different input samples, thus limiting the expressive power of the model.

[0094] To overcome the limitations of fixed-rank mechanisms, this invention proposes Dynamic Low-Rank Attention (DLA). The core idea is to introduce a "semantic information density" metric into attention computation and adaptively determine the effective rank value based on this metric. Semantic information density reflects the number and diversity of semantic features activated by the input text. Generally, the more complex the semantic structure, the more dispersed its representation is in the feature space. Existing research shows that complex texts typically activate broader subspace dimensions of language models, and the distribution of latent representations exhibits stronger discreteness and diversity. Current techniques indicate that the hidden representations of each Transformer layer are often embedded in low-dimensional semantic manifolds, while complex inputs lead to an expansion of the activation dimension. Furthermore, in contrastive learning and representation collapse research, the variance of representations is often used as a key indicator to measure representation diversity; high variance usually corresponds to richer semantic activation, while low variance may indicate insufficient feature dimension or limited information.

[0095] In this embodiment, the present invention introduces a semantic information density-guided dynamic low-rank attention mechanism into the attention process, and combines it with a linear projection compression structure to propose an efficient scaling dot product attention module. The overall process of this module is as follows: First, input a clean view. The semantic variance index is calculated using a semantic information density metric function. This index is used to estimate the semantic information richness of the current input text and to generate a dynamic rank value accordingly. Then, a clean view. The query matrix is ​​mapped through three linear layers respectively. Key matrix Sum matrix Among them, the key matrix Sum matrix It will be further implemented through dynamic rank. The linear projection matrix is ​​compressed in dimension to reduce redundant computation. Next, the compressed query matrix... Key matrix Perform a new scaled dot product attention calculation, divide by a scaling factor, obtain the attention weights using the softmax function, and then combine them with the compressed value matrix. Multiplication yields the final attention output representation. The outputs of multiple attention heads are then concatenated along the channel dimension and subjected to a linear transformation to form the module's final output. This mechanism improves computational efficiency while effectively maintaining task-related semantic expressiveness through dynamic semantic density awareness, and enhances the model's adaptability and robustness when handling texts of varying complexity.

[0096] Based on the above observations, this invention uses the variance of the hidden representation as a metric for semantic information density. For the hidden representation of batch input... ,in, B This represents the batch size, i.e., the number of samples input at one time. First, its variance is calculated:

[0097] ;

[0098] To avoid the influence of numerical scale differences, further normalization is performed on the batch dimension:

[0099] ;

[0100] Ultimately, the rank value is determined as follows:

[0101] ;

[0102] in, Represents the rank value. and These represent the preset minimum rank and maximum rank, respectively. Represents the normalized result , This represents the variance of the hidden representation. and Minimum variance and maximum variance, respectively. Represents the smoothing constant. It represents the variance of all hidden vectors, i.e., the overall dispersion of the features, and is used to measure the semantic information density. Indicates batch size, This represents the sequence length, that is, the number of tokens in each text sample. This represents the hidden layer dimension, that is, the feature dimension of each token. Indicates the first b The first sample i The first token of the token j Vygote indicates that, This represents the global mean, which is the global average across all dimensions of all samples and all tokens. It is used for centralization. The hidden layer represents a tensor, which is the hidden feature of all samples in a batch. A token is the smallest semantic unit after dividing natural language text, which can typically be a word, subword, or character, depending on the segmentation or encoding method used. In this invention, the input text is first segmented into a series of tokens and further converted into vector representations for input into the neural network for subsequent processing.

[0103] When the semantic information density is high, the variance increases, resulting in a higher rank value; when the input is relatively simple, the variance is low, thus assigning a smaller rank. After obtaining the dynamic rank value r, the attention calculation formula is updated as follows:

[0104] ;

[0105] in, This represents a dynamically selected projection matrix. n Indicates the sequence length. r Indicates the arithmetic value. This represents the final attention weight matrix. express Activation function Represents the query matrix. This represents a low-rank projection matrix (rank r) acting on the bond matrix. This indicates that it is used for the key matrix. Apply rank constraints. This represents the matrix transpose operation. Indicates the scaling factor. Represents the key matrix. Represents a value matrix, Let represent a low-rank projection matrix (rank r) applied to the value matrix V to impose a rank constraint on V. This represents the result of attention calculation. This mechanism adaptively balances computational complexity and expressive power while ensuring computational efficiency.

[0106] In practical applications, text data is often affected by various noises, such as spelling errors, OCR recognition errors, homograph substitutions, missing punctuation, and truncation. These contaminants often lead to inconsistent input distributions for text classification models during the training and inference phases, resulting in a significant performance degradation.

[0107] To improve the robustness of text classification models in complex environments, this invention designs a dual-view algorithm based on a dynamic low-rank attention encoder. Figure 1 Dual-view Consistency Constraint (DCC) is a robust training method based on the consistency constraint. The core idea of ​​this method is to construct two types of samples—clean and perturbed—for the same input text, and then train the sample using both views. Figure 1 Consistency constraints ensure the proximity of the two views in the representation space, thereby improving the text classification model's ability to adapt to noisy inputs.

[0108] This invention constructs a learning framework based on a dual-branch structure, where the online network and the target network differ in composition and function. The online network consists of an encoder, projector, predictor, and classifier, undertaking not only the classification task but also participating in the optimization of consistency constraints. The target network consists only of an encoder and a projector, whose parameters are synchronized from the online network using an exponential moving average (EMA) method to provide a relatively stable target representation. Both networks' encoders employ an improved structure based on BERT (Bidirectional Encoder-Representation Transformer) with an introduced dynamic low-rank attention mechanism (DLA) to balance semantic representation capability and computational efficiency. This encoder is represented as... .

[0109] Let the original input be a clean view X, and then pass through the interference function. Generate perturbation views of different severity levels:

[0110] ;

[0111] in, s This indicates the noise level. During training, the model simultaneously receives... As a dual-view input This indicates a perturbation view.

[0112] Let the encoder of the online network be... Its parameters Updated directly via backpropagation; the target network encoder is... Its parameters Synchronize from an online network using an exponential moving average:

[0113] ;

[0114] in, This represents the momentum coefficient, used to control the update rate of the target network.

[0115] In the consistency branch, the online network and the target network encode the clean view and the perturbed view respectively, obtaining semantic representations, i.e., the encoding results. and encoding results :

[0116] ;

[0117] Subsequently, the representation is mapped to a projection vector, i.e., the projection result, using a multilayer perceptron (MLP). and projection results :

[0118] ;

[0119] In online networks, the projection vector of the network itself is also transformed using a feedforward neural network (FFN) to obtain the prediction vector. :

[0120] ;

[0121] The predicted vector in the above "clean → polluted" direction Based on this, the prediction in the opposite direction (i.e., the "contaminated → clean" direction) is defined as:

[0122] ;

[0123] in, The representation is obtained by projecting the encoded contaminated sample onto a Multilayer Perceptron (MLP). The target network encoder encodes the clean sample, and the projection result is obtained by projecting the encoded sample onto a Multilayer Perceptron (MLP). .

[0124] Binocular vision Figure 1 Consistency loss is used to constrain the proximity of clean and perturbed views at the representation layer. This invention uses negative cosine similarity as a metric and employs a symmetrical design, simultaneously considering both the "clean → contaminated" and "contaminated → clean" directions.

[0125] ;

[0126] This loss term encourages the model to maintain consistency between the two types of inputs in the representation space, thereby improving robustness.

[0127] Classifier in the classification branch Directly obtain the semantic representation after online network encoding and output the predicted class probability distribution. :

[0128] ;

[0129] in, This refers to the encoder of an online network.

[0130] To prevent text classification model collapse and improve stability, the training objective consists of two parts: classification loss and consistency loss. The classification loss uses standard cross-entropy to ensure the model's discriminative performance under clean input.

[0131] ;

[0132] The final training objective function is:

[0133] ;

[0134] in, Indicates binocular vision Figure 1 Consistency loss function Represents classification loss, Represents the balance parameters. Indicates consistency loss. This indicates the batch size, which is the number of samples input at one time. Indicates the true label, Indicates the online network's response to the sample. Predicted labels, This represents the prediction vector obtained from the perturbation view. Represents cosine similarity. Used to adjust the relative weights of classification objectives and consistency constraints.

[0135] In the specific training process, clean and perturbed views are first generated for the input samples and fed into the online and target networks, respectively. Then, classification loss and consistency loss are calculated, and the parameters of the online network are jointly optimized. Finally, the parameters of the target network are updated using an exponential moving average. This process iterates continuously, enabling the text classification model to gradually learn to establish stable representation consistency between clean and contaminated data, thereby significantly improving generalization performance under noisy conditions. During inference, only the "online encoder → classifier" path is used to generate predictions; the target branch and consistency loss are not included in the calculation.

[0136] Based on the aforementioned consistency constraints, this invention implicitly introduces an optimization process aligned with the Information Bottleneck (IB) concept. The Information Bottleneck (IB) principle emphasizes learning representations... hour( (Representation) should retain as much useful information as possible that is relevant to the task (maximizing) , Representation With task tags (mutual information between them), while suppressing redundant components in the input that are unrelated to noise and disturbances (minimizing) , Representation With disturbance input (noisy input) (Mutual information between them). In the dual-view training framework, since the clean view and the perturbed view differ only in noise components, the consistency loss forces their representations to remain similar. This is equivalent to encouraging the text classification model to ignore noise features that only exist in the perturbed view during optimization, reducing the sensitivity of the representation to noise components, and thus implicitly compressing the representation. With disturbance input The mutual information between them. At the same time, the classification objective prompts the text classification model to retain as much structural semantic information as possible that is useful for class discrimination, so that the representation can stably capture features related to the task label. This process is consistent with the goal of "compressing input noise and retaining task-related information" in the Information Bottleneck (IB) framework, so that the consistency constraint not only plays a regularization role, but also theoretically provides motivational support for the robustness of the model under noise interference.

[0137] In summary, binocular vision Figure 1 Consistency constraints form a task-related feature focusing mechanism within the text classification model: by reducing the disturbance of noise to the representation space, dynamic low-rank attention can more effectively adjust the rank value on the basis of stable representation, thereby further improving the model's generalization ability and classification stability under complex noise conditions.

[0138] Example 2

[0139] like Figure 2 As shown, the present invention provides a text classification system for performing the text classification method described in the embodiments, including:

[0140] The first processing module is used to preprocess the input text and apply perturbation to the preprocessed text to generate clean text and perturbed text.

[0141] The second processing module is used to construct a text classification model that includes an online network and a target network, and trains the text classification model using clean text and perturbed text. The text classification model incorporates a dynamic low-rank attention mechanism and dual-viewpoint processing. Figure 1 Consistency loss function; clean text and perturbed text are used as dual-view inputs to the text classification model;

[0142] The third processing module is used to process any input text in parallel using a dual-branch architecture during the training of the text classification model, and output the text classification results.

[0143] In this embodiment, the functional units can be divided according to a text classification method. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this invention is illustrative and represents only a logical division; in actual implementation, other division methods may be used.

[0144] This invention conducted systematic experiments on multiple sets of Chinese and English text classification datasets, including tasks such as sentiment analysis, topic recognition, and fine-grained sentiment classification. Results show that the proposed method outperforms fixed low-rank baseline models in terms of accuracy, macro-F1 score, and Matthews correlation coefficient (MCC), and exhibits a more moderate performance degradation trend under various noise types and intensities. Ablation experiments verified the complementarity of dynamic rank allocation and consistency constraints. Compared with the standard self-attention model, this invention significantly improves classification stability under noisy conditions while maintaining near-linear complexity. The main contributions of this invention are as follows:

[0145] This study reveals and quantifies the insufficient robustness of fixed low-rank efficient Transformers in noisy text classification from a task perspective; it proposes a dynamic low-rank attention mechanism based on input variance information, achieving an adaptive balance between representational power and computational complexity; and it combines dual-view... Figure 1 By constraining consistency, an efficient training framework for robust text classification is constructed, and its effectiveness is verified through experiments in multilingual and multi-type noise scenarios, providing a new approach for the robustness research of efficient Transformers.

[0146] In this embodiment, the text classification system, in order to realize the principle and beneficial effects of the text classification method, includes hardware structures and / or software modules corresponding to the execution of various functions. Those skilled in the art should readily recognize that, in conjunction with the illustrative units and algorithm steps described in the embodiments disclosed herein, the present invention can be implemented in hardware and / or a combination of hardware and computer software. Whether a function is executed by hardware or computer software depends on the specific application and design constraints of the technical solution. Different methods can be used to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A text classification method, characterized in that, Includes the following steps: S1. Preprocess the input text and apply perturbation to the preprocessed text to generate clean text and perturbed text; S2. Construct a text classification method based on a dynamic low-rank attention mechanism, and a text classification model training method including online network and target network. The text classification model is trained using clean text and perturbed text. The text classification model introduces a dynamic low-rank attention mechanism and a dual-view consistency loss function. Clean text and perturbed text are used as dual-view inputs of the text classification model. Encoding two views using a dynamic low-rank attention mechanism includes the following steps: The dual-view data is embedded and represented, where semantic information density is used to reflect the number and differences of semantic features activated by the input dual-view text. Based on the embedding representation results, for each attention head, the query matrix Q, key matrix K, and value matrix V of the current layer input are projected into a low rank using the effective rank value determined by the dynamic low rank attention mechanism. Based on the low-rank projection results, the outputs of each attention head are spliced ​​and linearly transformed; After splicing and linear transformation, a position-by-position nonlinear transformation is performed using a feedforward neural network; The results of the nonlinear transformation are aggregated to obtain the encoded result. and ; The expression for the dual-view consistency loss function is as follows: ； ； ； in, This represents the two-view consistency loss function. Represents classification loss. Represents the balance parameters. Indicates consistency loss. Indicates batch size, Indicates the true label, Indicates the online network's response to the sample. Predicted labels, This represents the prediction vector obtained from the perturbation view. Represents cosine similarity. A clean view of the target network encoder represents the projection result obtained by projection; S3. During the training of the text classification model, a dual-branch architecture is used to process any input text in parallel and output the text classification results.

2. The text classification method according to claim 1, characterized in that, Both the online network and the target network include an encoder and a projector, and the online network also includes a predictor and a classifier. The encoder is used to encode two views using a dynamic low-rank attention mechanism to obtain the encoded result. and In this process, the same multi-layer stacked Transformer structure is used in both the online network and the target network encoders in the dual-branch architecture. The dynamic low-rank attention mechanism is embedded in the self-attention calculation module of each layer of the encoder, and semantic information density is introduced into the dynamic low-rank attention mechanism. The effective rank value can be adaptively determined based on the semantic information density measurement result. Projector, used for encoding results Using its corresponding projector, the projection result is obtained. and based on the encoding results Using its corresponding projector, the projection result is obtained. ; Predictor, used for the projection results Make a prediction to obtain the prediction vector. ; A classifier is used to analyze the encoded results. Perform discrimination to obtain the predicted category probability distribution. This completes the classification process for the text.

3. The text classification method according to claim 2, characterized in that, The expression for the rank value is as follows: ； ； ； in, Represents the rank value. This represents the preset minimum rank. Indicates the preset maximum rank. Represents the normalized result , This represents the variance of the hidden representation. and These represent the minimum variance and the maximum variance, respectively. Represents the smoothing constant. This represents the variance of all hidden vectors. Indicates batch size, Indicates the sequence length. Indicates the hidden layer dimension. Indicates the first b The first sample i The first token of the token j Vygote indicates that, This represents the global mean. This represents the hidden layer representation tensor.

4. The text classification method according to claim 2, characterized in that, Attention is calculated as follows: ； ； in, This represents the final attention weight matrix. express Activation function Represents the query matrix. Represents a low-rank projection matrix. This represents the matrix transpose operation. Indicates the scaling factor. Represents the key matrix. Represents a value matrix, Represents a low-rank projection matrix. This represents the result of the attention calculation.

5. The text classification method according to claim 1, characterized in that, The training process of the text classification model includes the following steps: Based on clean text and perturbation text, construct a clean view and a perturbation view; the clean view is input into an online network, and the perturbation view is input into a target network; Encoders using online networks Encode the clean view to obtain the encoding result. and encoders utilizing the target network Encode the disturbed view to obtain the encoding result. ; Based on encoding results Using its corresponding projector, the projection result is obtained. and based on the encoding results Using its corresponding projector, the projection result is obtained. ; Projection results The input is given to the predictor to obtain the prediction vector. ; Encode the result The data is input into the classifier, which calculates the predicted class probability distribution. ; Based on projection results Predicting the probability distribution of categories and prediction vector Define a dual-view consistency loss function that includes classification loss and consistency loss; The text classification model is trained by minimizing the two-view consistency loss function using gradient descent and updating the online network parameters, and by updating the target network parameters using exponential moving average. The online network parameters include the encoder, projector, predictor, and classifier. After training, only the encoder and classifier in the online network are retained.

6. A text classification system for performing the text classification method according to any one of claims 1-5, characterized in that, include: The first processing module is used to preprocess the input text and apply perturbation to the preprocessed text to generate clean text and perturbed text. The second processing module is used to construct a text classification model that includes an online network and a target network, and to train the text classification model using clean text and perturbed text. The text classification model incorporates a dynamic low-rank attention mechanism and a dual-view consistency loss function; the clean text and perturbed text serve as the dual-view inputs of the text classification model. Encoding two views using a dynamic low-rank attention mechanism includes the following steps: The dual-view data is embedded and represented, where semantic information density is used to reflect the number and differences of semantic features activated by the input dual-view text. Based on the embedding representation results, for each attention head, the query matrix Q, key matrix K, and value matrix V of the current layer input are projected into a low rank using the effective rank value determined by the dynamic low rank attention mechanism. Based on the low-rank projection results, the outputs of each attention head are spliced ​​and linearly transformed; After splicing and linear transformation, a position-by-position nonlinear transformation is performed using a feedforward neural network; The results of the nonlinear transformation are aggregated to obtain the encoded result. and ; The expression for the dual-view consistency loss function is as follows: ； ； ； in, This represents the two-view consistency loss function. Represents classification loss. Represents the balance parameters. Indicates consistency loss. Indicates batch size, Indicates the true label, Indicates the online network's response to the sample. Predicted labels, This represents the prediction vector obtained from the perturbation view. Represents cosine similarity. A clean view of the target network encoder represents the projection result obtained by projection; The third processing module is used to process any input text in parallel using a dual-branch architecture during the training of the text classification model, and output the text classification results.

Images