Text processing method and training method based on hierarchical vector approximation

Abstract

The disclosure provides a text processing method and a training method based on hierarchical vector approximation, belonging to the technical field of neural networks and context adaptation. The method comprises: obtaining an output embedding vector of an i-th layer of a neural network based on a transformer architecture; inputting the output embedding vector of the i-th layer into a trained low-rank proxy module for low-rank mapping to output a residual approximation; wherein the trained low-rank proxy module is obtained by self-distillation training and is configured to output an approximation value of a residual generated by the i-th layer of the neural network based on the transformer architecture according to the output embedding vector of the i-th layer; and generating an approximate output embedding vector of an i+1-th layer according to the output embedding vector of the i-th layer and the residual approximation.

Description

Technical Field

[0001] This disclosure relates to the fields of neural networks and context-adaptive technologies, and more specifically, to a text processing method and training method based on hierarchical vector approximation. Background Technology

[0002] With the rapid development of deep learning models, especially large language models (LLMs), the number of model parameters and computational complexity have increased exponentially, resulting in high latency and energy consumption during the inference stage, which severely restricts their deployment efficiency in edge devices or real-time application scenarios. To improve inference performance, the industry has proposed a variety of model compression and acceleration techniques, which can be mainly divided into two categories: static compression methods and dynamic inference methods.

[0003] Static compression methods are typically implemented after model training, accelerating computation by reducing the number of model parameters or decreasing numerical precision. Weight quantization, which maps floating-point weights and activation values ​​to low-bit integer representations, significantly reduces storage and computational costs, but inevitably introduces quantization noise, leading to decreased model accuracy. Model pruning, on the other hand, compresses the model structure by identifying and removing redundant or unimportant neurons or connections; however, its effectiveness is highly dependent on the rationality of the pruning criteria and it is difficult to recover performance losses caused by structural damage. Although some research has attempted to combine quantization and pruning to improve compression ratios, these static methods generally employ fixed computational paths and resource allocation strategies, failing to adaptively adjust computational costs based on the actual semantic complexity of the input samples, thus making it difficult to achieve optimal efficiency in diverse inference scenarios.

[0004] Dynamic inference methods attempt to overcome the limitations of static methods by allocating computational resources on demand through conditional execution mechanisms. Typical solutions include layer skipping and early termination. Layer skipping strategies dynamically determine whether to skip certain layers of the neural network based on the transformer architecture based on intermediate features, performing full computation only on key layers. Early termination mechanisms embed multiple intermediate exits in the network, terminating inference prematurely when the confidence of a layer's output reaches a preset threshold. For example, models such as DeeBERT and FastBERT have accelerated BERT through self-distillation and adaptive inference time control. However, these methods, by interrupting the standard forward information flow, are prone to insufficient context modeling, especially in long sequences or complex semantic tasks, often facing a dilemma of balancing accuracy and efficiency, limiting their applicability in high-performance LLM inference.

[0005] In recent years, low-rank approximation has emerged as a new research direction for LLM optimization due to its potential to significantly reduce computational complexity while preserving key semantic information. Existing techniques mainly focus on two types of applications: First, low-rank decomposition of weight tensors, which achieves efficient parameter fine-tuning by superimposing low-rank increment matrices on the original weights. While this effectively reduces the GPU memory and computational requirements during training, the inference process still requires a complete forward propagation, resulting in huge computational resource consumption without a substantial improvement in inference efficiency. Second, low-dimensional projection of intermediate tensors (such as attention matrices) in the activation path, although accelerating specific sub-modules (such as attention computation), still executes all neural network blocks based on the transformer architecture layer by layer, failing to achieve global optimization across layers. Third, existing low-rank approximation strategies mostly operate on local modules, lacking the ability to model the overall residual transformation between layers, making it difficult to maintain the consistency of the deep semantic evolution of the original model. As the inference depth increases, there is a possibility of rapidly deviating from the correct semantic trajectory due to the accumulation of information distortion, ultimately outputting unusable or misleading results. Summary of the Invention

[0006] In view of this, this disclosure provides a text processing method and training method based on hierarchical vector approximation, which can at least partially solve the above-mentioned technical problems.

[0007] One aspect of this disclosure provides a text processing method based on hierarchical vector approximation, comprising: obtaining the output embedding vector of the i-th layer of a transformer-based neural network, wherein the output embedding vector of the i-th layer represents the semantic context features of the words in the input text sequence at the i-th layer; inputting the output embedding vector of the i-th layer into a trained low-rank proxy module to perform low-rank mapping and output residual approximation, wherein the residual approximation represents the transformation increment generated by the semantic context features of the i-th layer between the i-th layer and the (i+1)-th layer; wherein the trained low-rank proxy module is obtained by self-distillation training and is configured to output an approximate value of the residual generated by the i-th layer in the transformer-based neural network based on the output embedding vector of the i-th layer; and generating an approximate output embedding vector of the (i+1)-th layer based on the output embedding vector of the i-th layer and the residual approximation, wherein the approximate output embedding vector of the (i+1)-th layer represents the semantic context features of the words in the input text sequence at the (i+1)-th layer.

[0008] According to an embodiment of this disclosure, the output embedding vector of the i-th layer is input to a trained low-rank proxy module to perform low-rank mapping output residual approximation, including: determining the low-rank dimension of the low-rank mapping based on the information retention threshold and computational resource budget value of the trained low-rank proxy module; and determining the residual approximation based on the low-rank dimension of the low-rank mapping and the output embedding vector of the i-th layer.

[0009] According to embodiments of this disclosure, the low-rank mapping is an incremental transformation of the output embedding vector of the i-th layer by sequentially performing bilinear projections of dimensionality reduction and dimensionality increase.

[0010] According to embodiments of this disclosure, the minimum low-rank dimension of the low-rank mapping is the smallest integer that satisfies the information retention ratio threshold and does not exceed the computational resource budget value.

[0011] According to an embodiment of this disclosure, generating an approximate output embedding vector for the (i+1)th layer based on the output embedding vector of the i-th layer and the residual approximation includes: connecting the output embedding vector of the i-th layer to the output of a trained low-rank proxy module via a skip connection to output the output embedding vector of the i-th layer; and approximating the output embedding vector of the i-th layer and the residual to generate an approximate output embedding vector for the (i+1)th layer.

[0012] This disclosure also provides a training method for a low-rank proxy module, comprising: constructing a teacher model and a student model to be trained; wherein, the teacher model is the i-th layer to be approximated in the original transformer-based neural network, the parameters of the original transformer-based neural network are kept frozen during the training process, and is used to receive the output embedding vector of the i-th layer and output the true output embedding vector of the (i+1)-th layer; the student model to be trained is used to determine the residual approximation by performing a low-rank mapping based on the input output embedding vector of the i-th layer, and output the approximate output embedding vector of the (i+1)-th layer based on the residual approximation; a loss function is determined based on the true output embedding vector of the (i+1)-th layer and the approximate output embedding vector of the (i+1)-th layer, and the parameters of the low-rank mapping are optimized based on the loss function to complete the self-distillation training and obtain the trained student model.

[0013] According to embodiments of this disclosure, a student model to be trained is used to determine a residual approximation by performing a low-rank mapping on the input output embedding vector of the i-th layer, and to output an approximate output embedding vector of the (i+1)-th layer based on the residual approximation. The student model to be trained includes a low-rank surrogate module, a skip connection, and an addition module. The low-rank surrogate module receives the input output embedding vector of the i-th layer and performs a low-rank mapping on the input output embedding vector of the i-th layer to output a residual approximation. The skip connection passes the input output embedding vector of the i-th layer to the addition module. The addition module adds the input output embedding vector of the i-th layer to the residual approximation and outputs an approximate output embedding vector of the (i+1)-th layer.

[0014] According to embodiments of this disclosure, a loss function is determined based on the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector, and the parameters of the low-rank mapping are optimized based on the loss function to complete self-distillation training and obtain a trained student model. This includes optimizing the parameters of the low-rank mapping by minimizing the mean square error between the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector to complete self-distillation training and obtain a trained student model.

[0015] According to embodiments of this disclosure, the method further includes: performing error characteristic analysis on the loss function, dividing the total error between the true output embedding vector of the (i+1)th layer and the approximate output embedding vector of the (i+1)th layer into approximation error and estimation error; wherein, the approximation error is characterized by the ability of the low-rank hypothesis space to approximate the true residual, and the estimation error is characterized by the convergence degree of the parameters of the optimal low-rank mapping in the self-distillation training process; the low-rank hypothesis space is the function class constraint of the low-rank surrogate module; the true residual is the difference between the output and input of the i-th layer in the original transformer-based neural network; and adjusting the approximation error according to the low-rank dimension of the low-rank mapping based on the characteristic analysis results.

[0016] According to embodiments of this disclosure, adjusting the approximation error based on the low-rank dimension of the low-rank mapping includes: reducing the approximation error by increasing the low-rank dimension of the low-rank mapping.

[0017] The text processing method based on hierarchical vector approximation according to embodiments of this disclosure has at least the following beneficial effects:

[0018] By introducing a pre-trained low-rank proxy module to perform a high-fidelity approximation of the residual transformation between layers i and i+1, computational acceleration is achieved while fully preserving the original information flow path and model hierarchical structure, thus significantly improving the stability and robustness of the inference process. Simultaneously, the generated text exhibits high consistency with the original large language model in terms of semantic coherence, factual accuracy, logical consistency, and style preservation, effectively avoiding problems such as semantic breaks, referential errors, or content illusions caused by layer skipping or coarse-grained pruning. It demonstrates particularly strong reliability in long text generation, multi-turn dialogue, and complex inference tasks.

[0019] Meanwhile, the low-rank proxy module of this embodiment requires only a few parameters to approximate the original residual with high accuracy; and the low-rank proxy module learns the residual mapping from the frozen original model through self-distillation, without the need for additional labeled data, and only needs to perform a lightweight low-rank operation once during the inference stage, which significantly reduces the number of floating-point operations and memory accesses, making it particularly suitable for resource-constrained deployment scenarios. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 A flowchart illustrating a text processing method based on hierarchical vector approximation according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 2 The diagram illustrates the performance curves of the text processing method based on hierarchical vector approximation according to embodiments of the present disclosure on MaWPS under different low-rank approximation dimensions.

[0023] Figure 3a A graph illustrating the singularity analysis results of a GSM8K dataset according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 3b A graph illustrating the singularity analysis results of the BoolQ dataset according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] In existing technologies, static model compression methods use a fixed computational budget for all inputs, which cannot be adjusted according to the specific complexity of the input samples, resulting in reduced efficiency. While dynamic inference methods can adaptively adjust the computational load, they may cause information flow disruptions, affecting model performance. Low-rank approximation methods for various levels of large language models, although reducing parameter space or storage costs, maintain the forward computation unchanged and do not fully utilize the low-rank structure to optimize inference efficiency. Most are only used in the fine-tuning stage or for fixed-structure acceleration; the complete forward process is still executed during inference, failing to achieve "computation skipping" or "dynamic approximation."

[0030] Based on this, this disclosure finds that the residual connections between consecutive transformer-based neural network layers in large language models have low-rank characteristics, which means that the changes in inter-layer transformations usually occur in a finite dimension.

[0031] That is, the complete computation process of the i-th layer of the neural network based on the transformer architecture is first expressed in a formula. This layer contains three core components: Layer Normalization (LN), Multi-Head Attention (MHA), and Feed-Forward Network (FFN), and retains the original information flow through residual connections.

[0032] set up E represents the input embedding vector of the i-th layer, where n is the length of the input sequence and d is the model's hidden dimension; i+1 Its output will be used as the input of the (i+1)th layer. Therefore, the forward computation of the i-th layer can be expressed as:

[0033]

[0034] This process preserves input information through residual connections, models global dependencies within sequences using MHA, and enhances local nonlinear semantic expression capabilities through FFN.

[0035] Furthermore, the output of the i-th layer is rewritten as the sum of the input and the inter-layer variation:

[0036]

[0037] Where ΔE represents the residual transformation from the i-th layer to the (i+1)-th layer, combining formulas (1) and (2), we can obtain:

[0038]

[0039] Numerous empirical studies have shown that, despite the complex structure of the original transformer-based neural network layers, the resulting residuals ΔE often exhibit significant low-rank characteristics in practice—that is, their main energy is concentrated in a few principal component directions. This property provides a solid theoretical foundation for efficient approximation.

[0040] Based on this, this disclosure proposes using a lightweight function. To approximate ΔE, that is:

[0041]

[0042] And further utilize low-rank structure to Modeling is performed. Specifically, it is concisely represented using a dual projection method as follows:

[0043]

[0044] in, For the dimension-reduced projection matrix, The matrix is ​​reconstructed by increasing its dimension, where r is a preset low-rank dimension, and This structure can convert the original The complexity of residual transformation is compressed to While significantly reducing computational overhead, it can still capture key semantic information in residuals with high fidelity.

[0045] Based on this, this disclosure proposes, in its embodiments, to use a low-rank proxy module to replace the neural network layer based on the transformer architecture. This allows for a high-fidelity approximation of the residual transformation between the i-th layer and the (i+1)-th layer, achieving computational acceleration while fully preserving the original information flow path and model hierarchy.

[0046] Figure 1 A flowchart illustrating a text processing method based on hierarchical vector approximation according to an embodiment of the present disclosure is shown schematically.

[0047] like Figure 1 As shown, the text processing method based on hierarchical vector approximation in this embodiment includes operations S110 to S130:

[0048] In operation S110, the output embedding vector of the i-th layer based on the Transformer architecture is obtained, where the output embedding vector of the i-th layer represents the semantic context features of the lexical units in the input text sequence at the i-th layer.

[0049] In operation S120, the output embedding vector of layer i is input to the trained low-rank proxy module to perform low-rank mapping and output residual approximation. The residual approximation represents the transformation increment generated by the semantic context features of layer i between layer i and layer i+1. The trained low-rank proxy module is obtained through self-distillation training and is configured to output an approximation of the residual generated by layer i in the transformer-based neural network based on the output embedding vector of layer i.

[0050] In operation S130, based on the output embedding vector of the i-th layer and the residual approximation, an approximate output embedding vector of the (i+1)-th layer is generated. The approximate output embedding vector of the (i+1)-th layer represents the semantic context features of the words in the input text sequence at the (i+1)-th layer.

[0051] In the embodiments of this disclosure, after the text sequence is processed by the first i layers of a neural network based on a transformer architecture, the output representation (i.e., the embedding vector) of the i-th layer is obtained, denoted as... , where n is the sequence length and d is the model's hidden dimension. This embedding vector can be directly obtained from the standard forward propagation path of the original large language model. In the application scenario disclosed herein, the first i layers still execute according to the original structure, ensuring that the context information is completely passed to the current layer. This provides accurate input states for subsequent low-rank proxy modules and is the basis for residual approximation. The i-th layer in the transformer-based neural network is pre-trained, and its parameters are in a frozen state.

[0052] E i Input a pre-trained low-rank proxy module with a lightweight structure (such as two layers of small matrices). This approximates the residual transformation ΔE of the i-th layer of the original transformer-based neural network.

[0053] The self-distillation training mechanism is as follows: freeze the i-th layer of the original model as the "teacher" and use its output... To achieve this, optimize the proxy module parameters. Minimize the mean squared error loss:

[0054]

[0055] Using the standard residual connection mechanism, the original input E is... i Approximate to the residual output by the proxy module Adding them together, we obtain an approximate representation that replaces the original (i+1)th layer output:

[0056]

[0057] According to the embodiments of this disclosure, the information flow interruption problem caused by layer skipping or premature termination in traditional dynamic inference methods is effectively avoided. Unlike coarse-grained acceleration strategies that directly omit network layers, this disclosure does not skip any neural network layers based on the transformer architecture. Instead, it uses a low-rank proxy module to perform a refined approximation of the inter-layer residual transformation and combines it with standard residual connections to generate the next layer's output. This completely preserves the information transmission path and hierarchical semantic structure of the original model, significantly improving the stability and robustness of accelerated inference. Furthermore, thanks to the inherent low-rank characteristics of the real residual transformation, a rank of [missing information] is used. The dual matrix decomposition structure achieves residual approximation, and the number of floating-point operations (FLOPs) per inference is reduced from that of the original layer. The magnitude is reduced to O(dr), while the residual approximation error is kept under control through self-distillation training.

[0058] The original input text can be a natural language string provided by the user.

[0059] A text sequence can be an ordered list of discrete symbol units generated after the original input text has been processed by a pre-trained word segmenter, and is usually further converted into a sequence of integer IDs.

[0060] For example, the system receives raw natural language text input from the user (e.g., "Explain how transformers work"); then, a pre-trained tokenizer segments and encodes the raw text to generate a corresponding text sequence, i.e., a list of integers composed of token identifiers (e.g., [12345, 67890, 205, ...]); next, this text sequence is input to the embedding layer, where it is converted into an initial embedding tensor through a lookup table operation, with a dimension of ... , where n is the sequence length and d is the model hidden dimension.

[0061] The initial embedding tensor is then passed through the first i layers of a transformer-based neural network for standard forward propagation, yielding the output embedding vector of the i-th layer. This vector serves as the intermediate activation state of the model at the current depth, fully preserving the contextual information of the input sequence. Next, the output embedding vector of the i-th layer is input into a low-rank surrogate module pre-trained through self-distillation. This module efficiently approximates the residual transformation generated by the i-th layer of the original model using a lightweight dual-matrix structure, outputting an approximate residual value.

[0062] Subsequently, following the standard residual connection mechanism, the output embedding vector of layer i is approximately added to the residual to generate an approximate output representation of layer i+1. This approximate representation is directly used as input to subsequent network layers, thus maintaining the complete information transmission path of the original model without skipping any network layers. Finally, after processing by the remaining layers, the system outputs natural language text (such as answers, summaries, or code), whose semantic quality is highly consistent with the original large model, but the inference process is significantly accelerated.

[0063] Thanks to the integrity of the information transmission path, this scheme reduces the computational complexity of a single layer from [previous level] to [new level] while maintaining almost no loss in generation quality. Reduce to This significantly reduces the number of floating-point operations, memory access overhead, and inference latency, making it particularly suitable for resource-constrained edge device deployment scenarios. Based on the above embodiments, the output embedding vector of the i-th layer is input to the trained low-rank agent module to perform low-rank mapping and output residual approximation, including: determining the low-rank dimension of the low-rank mapping based on the information retention threshold and computational resource budget of the trained low-rank agent module; and determining the residual approximation based on the low-rank dimension of the low-rank mapping and the output embedding vector of the i-th layer.

[0064] Furthermore, the low-rank mapping is an incremental transformation of the output embedding vector of the i-th layer by performing a bilinear projection of dimensionality reduction and dimensionality increase sequentially.

[0065] Furthermore, the minimum low-rank dimension of the low-rank mapping is the smallest integer that satisfies the information retention ratio threshold and does not exceed the computational resource budget value.

[0066] In embodiments of this disclosure, when embedding the i-th layer output vector E... i Before being input into the low-rank proxy module, the rank r of the low-rank mapping is dynamically determined based on two preset constraints. The minimum feasible low-rank dimension r is defined as: satisfying ≥ information retention ratio And it is ≤ the smallest positive integer r of the computational cost B. This mechanism ensures that the approximate accuracy is maximized under resource constraints, avoiding performance waste or insufficient accuracy caused by blindly setting r.

[0067] After determining r, the low-rank proxy module Ei performs bilinear projection: Step 1 (dimensionality reduction): through matrix E i Projecting to a lower-dimensional latent space; second step (dimensionality increase): through a matrix Reconstruct the low-dimensional representation back to the original dimension; the final output residual is approximately: .

[0068] According to embodiments of this disclosure, a low-rank dimension adaptive determination mechanism based on an information retention ratio threshold and computational resource budget is further introduced, significantly improving the scientific rigor and practicality of model compression. Specifically, when the output embedding vector of the i-th layer is input into the trained low-rank proxy module, the minimum low-rank dimension r that satisfies both accuracy and efficiency constraints is first determined based on a preset information retention ratio threshold τ (e.g., 95%) and computational resource budget B (e.g., maximum number of parameters or FLOPs limit). Subsequently, based on the value of r and the corresponding low-rank parameters, a dual linear projection of dimension reduction and dimension increase is performed on the input embedding vector to output a high-fidelity residual approximation.

[0069] Based on the above embodiments, an approximate output embedding vector for the (i+1)th layer is generated according to the output embedding vector of the i-th layer and the residual approximation. This includes: connecting the output embedding vector of the i-th layer to the output of the trained low-rank proxy module via a skip connection to output the output embedding vector of the i-th layer; and approximately adding the output embedding vector of the i-th layer and the residual to generate the approximate output embedding vector of the (i+1)th layer.

[0070] In embodiments of this disclosure, the true output of the i-th layer is embedded into the vector E. i The result is directly passed (i.e., "via a skip connection") to the output of the low-rank surrogate module; simultaneously, the low-rank surrogate module outputs an approximation of the residual transform. Finally, add the two together: This yields an approximate output embedding vector for the (i+1)th layer. This disclosure ensures a high degree of structural and semantic consistency between the approximate inference process and the original model by retaining and reusing the skip connection mechanism in the original transformer-based neural network architecture. Furthermore, by organically combining skip connections with low-rank residual approximation, this disclosure achieves a substantial improvement in computational efficiency without sacrificing model expressiveness or disrupting the network hierarchy, providing reliable technical support for the efficient, stable, and low-cost deployment of large-scale language models.

[0071] This disclosure also provides a training method for a low-rank proxy module to approximate the residual transformation of layers in a transformer-based neural network. The method includes constructing a teacher model and a student model to be trained. The teacher model is the i-th layer to be approximated in the original transformer-based neural network. The parameters of the original transformer-based neural network are frozen during training. It receives the output embedding vector of the i-th layer and outputs the true output embedding vector of the (i+1)-th layer. The student model to be trained is used to determine the residual approximation by performing a low-rank mapping based on the input output embedding vector of the i-th layer, and outputs the approximate output embedding vector of the (i+1)-th layer based on the residual approximation. A loss function is determined based on the true output embedding vector of the (i+1)-th layer and the approximate output embedding vector of the (i+1)-th layer. The parameters of the low-rank mapping are optimized based on the loss function to complete self-distillation training, resulting in a trained student model.

[0072] In embodiments of this disclosure, the teacher model is the i-th layer (containing LN, MHA, FFN, and residual connections) in the original, complete, and pre-trained transformer-based neural network model. Its parameters remain frozen throughout the training process and are used only for forward inference. Input: The output embedding vector E of the i-th layer. i Output: The actual output embedding vector of the (i+1)th layer. .

[0073] Student model: The low-rank proxy module to be trained, whose structure is a bilinear projection. Used to predict residual approximations And output an approximate representation of the next layer: .

[0074] Output from teacher model As the supervision objective, the student model output is used. For the predicted value, the loss function is defined as the mean squared error between the two values, and the expression is as follows:

[0075]

[0076] The expression is as described above. The specific engineering implementation of the formula.

[0077] Only student model parameters are updated via backpropagation. The teacher model was frozen throughout the process. Among them, training sample E... i It is derived from the intermediate activation values ​​of the original model on a large-scale corpus (i.e., "self-generated" data), without the need for manual annotation, thus achieving completely unsupervised distillation training.

[0078] This method constructs a teacher-student training framework, using the i-th layer to be approximated in the original transformer-based neural network as a frozen teacher model, and utilizing its output's true (i+1)-th layer embedding vector as a supervision signal. Simultaneously, a lightweight low-rank mapping structure is used as the student model, learning the residual transform through bilinear projection and combining it with skip connections to generate an approximate output. By minimizing the reconstruction error between the student model output and the teacher model output, end-to-end optimization of the low-rank parameters is performed, ultimately obtaining a high-fidelity residual approximator.

[0079] Based on the above embodiments, the student model to be trained is used to determine the residual approximation by performing a low-rank mapping on the input output embedding vector of the i-th layer, and to output an approximate output embedding vector of the (i+1)-th layer based on the residual approximation. This includes: the student model to be trained includes a low-rank surrogate module, a skip connection, and an addition module. The low-rank surrogate module receives the input output embedding vector of the i-th layer and performs a low-rank mapping on it to output the residual approximation. The skip connection passes the input output embedding vector of the i-th layer to the addition module. The addition module adds the input output embedding vector of the i-th layer to the residual approximation, outputting an approximate output embedding vector of the (i+1)-th layer.

[0080] Based on the above embodiments, a loss function is determined according to the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector, and the parameters of the low-rank mapping are optimized according to the loss function to complete self-distillation training and obtain a trained student model. This includes: optimizing the parameters of the low-rank mapping by minimizing the mean square error between the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector to complete self-distillation training and obtain a trained student model.

[0081] Building upon the above embodiments, the method further includes: performing error characteristic analysis on the loss function, dividing the total error between the true output embedding vector of layer (i+1) and the approximate output embedding vector of layer (i+1) into approximation error and estimation error. The approximation error characterizes the ability of the low-rank hypothesis space to approximate the true residual, while the estimation error characterizes the convergence degree of the self-distillation training process to the parameters of the optimal low-rank mapping. The low-rank hypothesis space represents the function class constraints of the low-rank surrogate module; the true residual is the difference between the output and input of layer i in the original transformer-based neural network. Based on the characteristic analysis results, the approximation error is adjusted according to the low-rank dimension of the low-rank mapping.

[0082] Furthermore, the approximation error can be controlled based on the low-rank dimension of the low-rank mapping, including reducing the approximation error by increasing the low-rank dimension of the low-rank mapping.

[0083] In the embodiments of this disclosure, step 1: First, a theoretical analysis is performed on the defined loss function, which is then expressed as an optimization objective function. That is, the true residual The expected value of the squared L2 norm of the difference between the output of the proxy module and the output of the proxy module. :

[0084]

[0085] The objective function can be further expanded into an inner product form to facilitate subsequent differentiation and analysis:

[0086]

[0087] Step 2: Apply the variational method to derive the necessary conditions for the optimal solution, under the optimal parameters. At, objective function The derivative must be zero. To do this, consider the hypothesis space of rank r. Inside, along any direction For the optimal solution Applying a small perturbation, its derivative satisfies:

[0088]

[0089] in, It is the amplitude of the disturbance.

[0090] By calculating the derivative, we can obtain:

[0091]

[0092] Step 3: Determine the orthogonality of the error vector. Based on the necessary condition derived in Step 2 above, due to the direction of the disturbance... In the hypothesis space The interior is arbitrary.

[0093] Therefore, we can conclude that: at the optimal parameters Below, error vector with hypothesis space any function within All are orthogonal. This orthogonality property can be expressed by the following formula:

[0094]

[0095] in This represents the low-rank hypothesis space Any function in the space is used to express the orthogonality between the error vector and the space.

[0096] Step 4: Using the orthogonality properties described above, verify the characteristic analysis of the total error by expanding the square norm of the total error:

[0097]

[0098] Based on the orthogonality conclusion above, the expectation of the cross term in the above equation is zero. Therefore, the expectation of the total error can be precisely decomposed into the sum of the expectations of the approximation error and the estimation error. This verification provides mathematical support for the assertion in step 3 of the first aspect that "the approximation error is dominant and there is no bias-variance inversion," and confirms the rationality of the rank selection strategy based on information preservation.

[0099] Figure 2 The diagram illustrates the performance curves of the text processing method based on hierarchical vector approximation according to embodiments of the present disclosure on MaWPS under different low-rank approximation dimensions.

[0100] In some possible embodiments, embodiments of this disclosure explore the relationship between the r-value and approximation error through bias-variance decomposition, pointing out that a larger r-value increases the likelihood of estimation error. Performance tests were performed on the MaWPS dataset with r-values ​​ranging from 32 to 1024 to discuss the impact of different r-values ​​on model performance.

[0101] Specifically, this embodiment includes the following steps:

[0102] Step 4: Using the hierarchical vector approximation-based text processing method provided in this disclosure, performance validation is performed on the MaWPS dataset with a limited training sample size. The rank parameter of the low-rank proxy module is traversed through a set of discrete values ​​{32, 64, 128, 256, 512, 1024}. For each rank value, the model is fine-tuned independently, and finally a set of model performance evaluation results corresponding to different ranks is obtained.

[0103] Step 5: Plot the rank-performance curve and perform trend analysis.

[0104] like Figure 2 As shown, the results indicate that the model performance increases monotonically with the increase of the rank parameter and tends to saturate, without any performance degradation due to overfitting.

[0105] Figure 3a A graph illustrating the singularity analysis results of a GSM8K dataset according to an embodiment of the present disclosure is shown schematically.

[0106] Figure 3b A graph illustrating the singularity analysis results of the BoolQ dataset according to an embodiment of the present disclosure is shown schematically.

[0107] To further verify the prevalence of the low-rank characteristic of inter-layer residuals in neural networks across different types of tasks, this example selects a mathematical reasoning task (GSM8K dataset) and a question-answering task (BoolQ dataset). Singular value decomposition (SVD) is performed on the inter-layer residuals to analyze their rank structure and information retention. This example includes the following steps:

[0108] Step 6: Cross-task residual data extraction and singular value decomposition. In two representative task scenarios, mathematical reasoning (GSM8K) and question answering (BoolQ), the inter-layer residual matrices generated during model fine-tuning are collected. Singular value decomposition (SVD) is performed on each collected set of residual matrices to obtain their respective singular value spectra.

[0109] Step 7: Analysis and verification of low-rank structure characteristics.

[0110] like Figure 3a and Figure 3b The figures show the singular value analysis results on the GSM8K dataset and the BoolQ dataset, respectively. Analysis of the singular value spectra obtained after decomposition shows that the singular values ​​all exhibit a rapid decay trend, verifying the prevalent low-rank structure of inter-layer residuals. Furthermore, quantitative analysis of information retention shows that when the rank is 256, the information retention of the vast majority of inter-layer residuals falls within the range of 90%–95%. These results provide strong empirical support for the use of a low-rank approximation method to replace inter-layer residuals in this disclosure.

[0111] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A text processing method based on hierarchical vector approximation, characterized in that, include: Obtain the output embedding vector of the i-th layer of the neural network based on the transformer architecture, where the output embedding vector of the i-th layer represents the semantic context features of the words in the input text sequence at the i-th layer; The output embedding vector of layer i is input into a trained low-rank proxy module to perform low-rank mapping and output residual approximation. This residual approximation represents the transformation increment generated by the semantic context features of layer i between layer i and layer i+1. The trained low-rank proxy module is obtained through self-distillation training and is configured to output an approximation of the residual generated by layer i in a transformer-based neural network based on the output embedding vector of layer i. A lightweight function f(E) is used. i The residual ΔE is approximated by the following expression: We then model f(Ei) using a double projection approach, and the expression for f(Ei) is as follows: in, For the dimension-reduced projection matrix, The matrix is ​​reconstructed by increasing its dimension, where r is a preset low-rank dimension, and E i Let be the input embedding vector of the i-th layer, and d be the hidden dimension of the neural network based on the transformer architecture, LN( ) is for layer normalization, MHA( ) is a multi-head self-attention mechanism, FFN( () is a feedforward neural network. This represents the residual transformation from layer i to layer i+1; Based on the output embedding vector of the i-th layer and the residual approximation, an approximate output embedding vector of the (i+1)-th layer is generated. The approximate output embedding vector of the (i+1)-th layer represents the semantic context features of the lexical units in the input text sequence at the (i+1)-th layer.

2. The method according to claim 1, characterized in that, The step of inputting the output embedding vector of the i-th layer into the trained low-rank proxy module for low-rank mapping and output residual approximation includes: The low-rank dimension of the low-rank mapping is determined based on the information retention threshold and computational resource budget value of the trained low-rank agent module. The residual approximation is determined based on the low-rank dimension of the low-rank mapping and the output embedding vector of the i-th layer.

3. The method according to claim 2, characterized in that, The low-rank mapping is an incremental transformation that sequentially performs dimensionality reduction and dimensionality increase on the output embedding vector of the i-th layer through a bilinear projection.

4. The method according to claim 2, characterized in that, The minimum low-rank dimension of the low-rank mapping is the smallest integer that satisfies the information retention ratio threshold and does not exceed the computing resource budget value.

5. The method according to claim 1, characterized in that, The step of generating an approximate output embedding vector for the (i+1)th layer based on the output embedding vector of the i-th layer and the residual approximation includes: The output embedding vector of the i-th layer is connected to the output of the trained low-rank proxy module by a skip connection to output the output embedding vector of the i-th layer. The output embedding vector of the i-th layer and the residual are approximately added together to generate an approximate output embedding vector of the (i+1)-th layer.

6. A training method for a low-rank agent module, characterized in that, include: Construct teacher models and student models to be trained; The teacher model is the i-th layer to be approximated in the original transformer-based neural network. The parameters of the original transformer-based neural network are kept frozen during the training process and are used to receive the output embedding vector of the i-th layer and output the true output embedding vector of the (i+1)-th layer. The student model to be trained is used to determine the residual approximation by performing a low-rank mapping on the input output embedding vector of the i-th layer, and outputting an approximate output embedding vector of the (i+1)-th layer based on the residual approximation; using a lightweight function f(E) i The residual ΔE is approximated by the following expression: We then model f(Ei) using a double projection approach, and the expression for f(Ei) is as follows: in, For the dimension-reduced projection matrix, The matrix is ​​reconstructed by increasing its dimension, where r is a preset low-rank dimension, and E i Let be the input embedding vector of the i-th layer, and d be the hidden dimension of the neural network based on the transformer architecture, LN( ) is for layer normalization, MHA( ) is a multi-head self-attention mechanism, FFN( () is a feedforward neural network. This represents the residual transformation from layer i to layer i+1; The loss function is determined based on the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector, and the parameters of the low-rank mapping are optimized based on the loss function to complete the self-distillation training and obtain the trained student model.

7. The method according to claim 6, characterized in that, The student model to be trained is used to determine the residual approximation by performing a low-rank mapping on the input output embedding vector of the i-th layer, and to output an approximate output embedding vector of the (i+1)-th layer based on the residual approximation, including: The student model to be trained includes a low-rank proxy module, a skip connection, and an addition module; The low-rank proxy module is used to receive the output embedding vector of the i-th layer of the input and to perform low-rank mapping based on the output embedding vector of the i-th layer of the input to output the residual approximation; The skip connection is used to pass the output embedding vector of the i-th layer of the input to the addition module; The addition module is used to approximately add the input layer i-th layer's output embedding vector to the residual, and output an approximately layer i+1's output embedding vector.

8. The method according to claim 6, characterized in that, The step of determining the loss function based on the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector, and optimizing the parameters of the low-rank mapping based on the loss function to complete self-distillation training and obtain a trained student model includes: By minimizing the mean square error between the true (i+1)th layer output embedding vector and the approximate (i+1)th layer output embedding vector, the parameters of the low-rank mapping are optimized to complete self-distillation training, resulting in a well-trained student model.

9. The method according to claim 6, characterized in that, Also includes: An error characteristic analysis is performed on the loss function, and the total error between the true output embedding vector of the (i+1)th layer and the approximate output embedding vector of the (i+1)th layer is divided into approximation error and estimation error. The approximation error represents the ability of the low-rank hypothesis space to approximate the true residual, while the estimation error represents the convergence degree of the parameters of the optimal low-rank mapping in the self-distillation training process. The low-rank hypothesis space is the function class constraint of the low-rank surrogate module. The true residual is the difference between the output and input of the i-th layer in the original transformer-based neural network. Based on the characteristic analysis results, the approximation error is adjusted according to the low-rank dimension of the low-rank mapping.

10. The method according to claim 9, characterized in that, The step of adjusting the approximation error based on the low-rank dimension of the low-rank mapping includes: The approximation error is reduced by increasing the low-rank dimension of the low-rank mapping.

Images