Low rank adaptation method and system based on singular value decomposition implicit routing

Abstract

The application discloses a low-rank adaptation method and system based on singular value decomposition implicit routing, and belongs to the technical field of low-rank adaptation. The method comprises the following steps: performing singular value decomposition on a pre-training weight matrix of a target layer, constructing an implicit expert with right singular vectors thereof and generating a projection matrix, training parameters of the projection matrix throughout the whole process, and only retaining an output matrix to participate in fine tuning; in a forward propagation stage, input features are projected to a low-dimensional hidden space through the projection matrix to obtain hidden features, and a sparse routing mechanism is introduced on the hidden features; and finally, the target layer output is obtained through activated expert output mapping and weighted fusion after the output matrix. The application does not need an independent gating network, and routing decisions are directly determined by the absolute value of the hidden features after the input features are projected through the projection matrix, thereby eliminating the problems of explicit routing load imbalance and additional overhead, reducing the number of trainable parameters, realizing the integration and cooperation of low-rank adaptation and routing decisions, and being applicable to efficient fine tuning of large language models in resource-constrained scenarios.

Description

Technical Field

[0001] This invention belongs to the field of low-rank adaptation technology, specifically relating to a low-rank adaptation method and system based on singular value decomposition implicit routing. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the exponential growth of parameters in large language models, the Hybrid Expert (MoE) architecture has become a core technological approach to overcome the computational bottleneck of dense Transformers and achieve efficient training and inference of trillion-parameter models. By decomposing the feedforward network (FFN) into multiple independent sparse activation experts, the MoE architecture successfully decouples the number of model parameters from the computational cost, achieving significantly better performance under the same computational budget. Meanwhile, Parameter Efficient Fine-Tuning (PEFT) techniques, represented by Low-Rank Adaptation (LoRA), have become a core approach to overcome the computational bottleneck of dense Transformers and achieve efficient fine-tuning of large language models, thanks to their advantage of approximating the effect of full-parameter fine-tuning by training only a small number of low-rank matrices.

[0004] However, the inventors discovered that existing MoE-LoRA schemes still have fundamental flaws: First, existing MoE models rely on independent gating networks for hard assignment of terms to experts, resulting in unbalanced expert load, sensitivity to noise in routing decisions, and the introduction of additional computational and communication overhead; second, traditional MoE-LoRA methods treat low-rank updates and routing as independent processes, which cannot achieve end-to-end collaborative optimization, limiting the upper limit of the model's task adaptability; third, training the LoRA matrix independently for each expert leads to a linear increase in the number of parameters with the number of experts, while sharing the LoRA matrix will erase the functional specificity of experts; in addition, standard LoRA uses random Gaussian initialization of the low-rank matrix, ignoring the inherent low-rank structure in the pre-trained weights, resulting in slow convergence and easy introduction of noise in small sample tasks, while existing improved methods based on weight singular value decomposition can only extract general semantic features, and the projection direction does not match the data distribution of downstream tasks, making it difficult to overcome the accuracy bottleneck. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a low-rank adaptation method and system based on singular value decomposition implicit routing.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the technical solution of the present invention provides a low-rank adaptation method based on singular value decomposition implicit routing, including: Singular value decomposition is performed on the pre-trained weight matrix of the target layer to extract the right singular vector and its corresponding singular value. The number of singular value components that are as many as the rank that are adapted to the low rank are selected as implicit experts and combined to generate the projection matrix. The parameters of the projection matrix are frozen throughout the training process, and only the output matrix is ​​retained for fine-tuning. During the forward propagation phase, the input features are projected onto the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features for forward computation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated. The final output of the target layer is obtained by weighted fusion of the outputs of the activated experts after mapping through the output matrix; In this approach, implicit experts do not require a separate gating network for route allocation; the route decision is directly determined by the absolute value of the latent features after the input features are projected through the projection matrix.

[0007] In at least one embodiment, the number of experts activated in the Top-K sparse routing mechanism is less than the total number of implicit experts, and a single token only activates a portion of the expert components, thereby achieving dynamic sparse forward propagation.

[0008] In at least one embodiment, the projection matrix is ​​generated by combining selected singular value components, and the row vectors of the projection matrix are the transpose of the first preset number of columns of the right singular vectors, with all row vectors maintaining the unit norm; wherein, the preset number is equal to the rank of the low-rank adaptation; the projection matrix does not participate in gradient updates throughout the training process.

[0009] In at least one embodiment, the initialization of the projection matrix is ​​constructed based on the input activation feature distribution of the downstream task training data, and a lightweight streaming accumulation scheme is designed to adapt to the memory constraints of large model fine-tuning, including: A small amount of calibration data is sampled from the downstream training set, and the input activation features of the target linear layer are captured through the network forward hook. The cumulative token count, activation feature summation vector and activation outer product matrix are counted by three sets of GPU-side lightweight accumulators configured for each layer. After the calibration data traversal is completed, the centered input covariance matrix is ​​calculated based on the accumulated statistics, and low-rank singular value decomposition is performed on the covariance matrix to obtain the right singular vector and singular value matrix adapted to the downstream task. A predetermined number of column vectors before the right singular vector are selected as the row vectors of the projection matrix. All row vectors maintain the unit norm, and the projection direction is constructed entirely based on the downstream data distribution to eliminate the bias of the general prior of pre-training. At the same time, a fault tolerance mechanism is designed so that if there is no task SVD cache, it will fall back to the weight SVD initialization scheme.

[0010] In at least one embodiment, a pre-trained weight subspace is introduced as a credibility reference, a layer-by-layer adaptive alignment gating screening mechanism is designed, a high-quality initialization projection matrix is ​​constructed using a hybrid approach, and prior knowledge and task adaptability are balanced. Specifically, this includes: Extract the first preset number of right singular vectors from the singular value decomposition of the pre-trained weights, construct a pre-trained knowledge subspace, and calculate the projection length of the adaptive singular value decomposition feature vector of each task in the pre-trained knowledge subspace as the feature alignment degree. For each network layer, an adaptive alignment threshold is designed that is adapted to the current network layer. It is a preset multiple of the statistical expectation of the projection length of the random vector in the pre-trained knowledge subspace. Each network layer is calculated independently. Task singular value components with feature alignment not less than the adaptive alignment threshold are retained as row vectors of the projection matrix to realize the transfer and reuse of pre-trained general knowledge; task singular value components with feature alignment less than the adaptive threshold are discarded and the gaps are filled with unit norm Gaussian random vectors.

[0011] In at least one embodiment, a learnable bias term is configured for each implicit expert dimension, and the routing order is based on the combination value of the absolute value of the latent feature and the corresponding bias term. During training, the activation frequency of each implicit expert is counted in real time, the bias term is dynamically updated, the bias weight of over-activated implicit experts is reduced, the bias weight of long-term idle implicit experts is increased, and the utilization rate of all experts is forced to be balanced.

[0012] In at least one embodiment, the forward output of the final target layer is determined based on the projection matrix, the latent features sparsified by the sparse routing mechanism, and the output matrix. Specifically, the projection matrix maps the input features to a low-dimensional latent space, the Top-K sparse routing mechanism sparsifies the latent features, and the output matrix maps the sparsified latent features back to the target output space.

[0013] In at least one embodiment, the output of the target layer is represented as:

[0014] In the formula, This is the pre-trained weight matrix for the target layer. For input features, For the output matrix, This is a Top-K sparse routing mechanism. Let be the projection matrix. The rank that is adapted to the lower rank. This refers to the scaling hyperparameter.

[0015] In at least one embodiment, the set of singular value components selected for each task is tracked, an index of the degree of overlap of singular value components between tasks is calculated, and shared components and unique components are identified; when learning new tasks sequentially, the unique components of the previous task are protected by setting the gradient to zero in the current task training, while the shared components across tasks are kept plastic to achieve positive knowledge transfer.

[0016] Secondly, the technical solution of the present invention also provides a low-rank adaptation system based on singular value decomposition implicit routing, comprising: The projection matrix construction module is configured to: perform singular value decomposition on the pre-trained weight matrix of the target layer, extract the right singular vector and the corresponding singular value, select the number of singular value components that are equal to the rank that are adapted to the low rank as implicit experts, combine them to generate the projection matrix, freeze the parameters of the projection matrix throughout the training process, and only retain the output matrix to participate in fine-tuning. The implicit routing module is configured as follows: during the forward propagation stage, the input features are projected to the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features to participate in the forward calculation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated. The fusion output module is configured to: weight and fuse the outputs of the activated experts after mapping through the output matrix to obtain the final output of the target layer; In this approach, implicit experts do not require a separate gating network for route allocation; the route decision is directly determined by the absolute value of the latent features after the input features are projected through the projection matrix.

[0017] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) This invention directly uses the right singular vector of the SVD of the pre-trained weight matrix as the implicit expert, and replaces the independent gated network in the traditional MoE with the magnitude of the SVD projection activation value as the routing signal. This fundamentally eliminates the inherent problems such as unbalanced expert load, sensitivity of routing decisions to input noise, and additional communication overhead. By introducing a load balancing routing bias mechanism to dynamically adjust the bias terms of each expert in real time, it effectively prevents expert collapse and ensures that all implicit experts are fully trained.

[0018] 2) By freezing the projection matrix in step one and retaining only the output matrix for fine-tuning, the present invention can train about half the number of parameters as the standard LoRA same-rank configuration, which significantly reduces the amount of parameter updates and memory overhead during training.

[0019] 3) This invention creatively embeds the routing function into the SVD low-rank projection process, so that the routing decision is directly determined by the intrinsic correlation between the input features and the directions of the pre-trained weight features, rather than being controlled by an independent gating network. The low-rank update affects the routing decision end-to-end through the fixed direction of the projection matrix and the differentiated update of the output matrix, realizing the integrated and collaborative optimization of the two, rather than treating them as independent processes in traditional schemes.

[0020] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0022] Figure 1 This is a schematic diagram of the low-rank adaptation method based on singular value decomposition implicit routing disclosed in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the network structure of the low-rank adaptation method based on singular value decomposition implicit routing disclosed in Embodiment 1 of the present invention. Figure 3 This is a flowchart of the training process for the network structure of the low-rank adaptation method based on singular value decomposition implicit routing disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0026] Terminology Explanation: Implicit routing: Implicit routing is an expert allocation mechanism in hybrid expert models (MoE) that does not have independent routers and whose routing rules are implicit in the model structure / parameters. The core is that it does not train a separate explicit gated network (Router), and the routing decision is directly determined by the intrinsic relationship between the input features and the expert parameters.

[0027] Low-rank adaptation for large language models: Low-rank adaptation (LoRA) is a mainstream technique for efficient parameter fine-tuning (PEFT) of large language models. Its core is to freeze all the original parameters of the pre-trained base model and only add a small number of low-rank matrices to participate in training, so as to achieve downstream task adaptation with a very low number of parameters, replacing the high-cost full parameter fine-tuning.

[0028] Example 1 As described in the background section, existing MoE-LoRA schemes still have fundamental flaws. First, there is an inherent bottleneck in explicit gating routing mechanisms. Existing MoE models generally rely on independent gating networks to achieve hard assignment of terms to experts. This explicit routing strategy has three main problems: First, it is prone to and difficult to eradicate the phenomenon of unbalanced expert load, with a large number of terms flowing to a few popular experts, resulting in wasted computing resources and idle model capacity. Second, routing decisions are sensitive to input noise and gating parameter perturbations. Small changes in the gating network can sometimes cause drastic fluctuations in expert assignment results, affecting model stability. Third, the additional gating computation and communication overhead significantly increases inference latency when deployed on edge devices, partially offsetting the computing power advantage of the MoE architecture.

[0029] Secondly, the traditional MoE-LoRA method treats low-rank update and expert routing as two independent processes, inserting a LoRA branch only within the expert network, while routing decisions remain controlled by the original gating network. This decoupled design makes it difficult for the two to achieve end-to-end collaborative optimization: low-rank update cannot effectively guide routing decisions to adjust in a direction more beneficial to downstream tasks, and routing results are also difficult to provide targeted feedback for the learning of the low-rank matrix, ultimately limiting the upper limit of the model's task adaptability.

[0030] Furthermore, the existing MoE-LoRA scheme is caught in a dilemma: if a LoRA matrix is ​​trained independently for each expert, the number of trainable parameters will increase linearly with the number of experts. In scenarios with a large number of experts, the number of parameters will quickly exceed the acceptable range of conventional LoRA, thus losing the advantage of parameter efficiency; if all experts share a LoRA matrix, the functional specificity of different experts will be eliminated, causing the model to fail to fully utilize the multimodal representation capabilities of the hybrid expert architecture.

[0031] Furthermore, the standard LoRA method initializes the low-rank matrix using a random Gaussian distribution, completely ignoring structured prior knowledge such as the singular vector directions (i.e., the main directions of change in the weight matrix) and singular value magnitudes (i.e., the importance ranking of each direction) that can be obtained from singular value decomposition of pre-trained weights. This leads to slow model convergence, instability in the early stages of training, and the introduction of noise irrelevant to the target task in small-sample downstream tasks, potentially destroying the general knowledge accumulated by the pre-trained model. Existing improved LoRA methods based on weight singular value decomposition can only extract general semantic features from pre-training. Their projection directions do not match the data distribution of differentiated downstream tasks such as mathematical reasoning, code generation, and professional question answering, making it difficult to specifically adapt to the characteristics of downstream tasks and overcome the accuracy bottleneck of traditional fine-tuning schemes.

[0032] To overcome the shortcomings of the prior art, in a typical embodiment of the present invention, such as... Figures 1 to 3 As shown, this embodiment discloses a low-rank adaptation method based on singular value decomposition (SVD) implicit routing. It utilizes SVD to extract structured feature components of the pre-trained weight matrix as implicit experts, and uses the absolute value of the activation values ​​after low-rank projection of the input features as the routing signal. This replaces the independent gated network in traditional MoE, achieving an integrated design of routing decision and low-rank adaptation. The specific steps of this low-rank adaptation method based on SVD implicit routing are as follows: S1. Perform singular value decomposition on the pre-trained weight matrix of the target layer, extract the right singular vector and the corresponding singular value, select the number of singular value components that are as close as the rank to the low rank as implicit experts, combine them to generate the projection matrix, freeze the parameters of the projection matrix throughout the training process, and only retain the output matrix to participate in fine-tuning. S2. In the forward propagation stage, the input features are projected to the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features to participate in the forward calculation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated. S3. The outputs of the activated experts are weighted and fused after being mapped by the output matrix to obtain the final output of the target layer; In this approach, implicit experts do not require a separate gating network for route allocation; the route decision is directly determined by the absolute value of the latent features after the input features are projected through the projection matrix.

[0033] The following section provides a detailed explanation of the low-rank adaptation method based on singular value decomposition implicit routing, using specific implementation methods.

[0034] S1. Perform singular value decomposition on the pre-trained weight matrix of the target layer, extract the right singular vector and its corresponding singular value, select the number of singular value components that are equal to the rank of the low-rank component as implicit experts, combine them to generate the projection matrix, freeze the parameters of the projection matrix throughout the training process, and only retain the output matrix to participate in fine-tuning.

[0035] In this step, singular value decomposition is first performed on the pre-trained weight matrix of the target layer to extract the spectral prior information contained in the pre-trained weights. Specifically, the pre-trained weight matrix of the target layer is defined as follows: Perform singular value decomposition (SVD) on it, specifically as follows:

[0036] In the formula, It is a left singular vector matrix; It is a diagonal singular value matrix, with the singular values ​​arranged in descending order along the diagonal; It is a right singular vector matrix.

[0037] Based on the SVD decomposition results, the right singular vectors and their corresponding singular values ​​are extracted. The rank of Low-Rank Adaptation (LoRA) is set to... Based on the extracted right singular vector and its corresponding singular value, select Group singular values ​​as An implicit expert, that is, the one who selects the right singular vector. Column and corresponding front Each singular value is used as There are 10 implicit experts, each corresponding to an independent structured feature direction in the pre-trained weights. The magnitude of the singular value reflects the importance of that direction in the pre-trained weights.

[0038] Based on selection Group SVD feature components, combine to generate frozen projection matrix Its construction method is to take the right singular vector matrix. The former Transpose the columns as the projection matrix The row vector, i.e. All row vectors maintain the unit norm. During training, only the output matrix is ​​used. With gradient updates and fine-tuning, and the projection matrix The implicit experts are frozen throughout the entire training process and do not participate in any gradient updates; their parameters remain unchanged. This effectively fixes the direction of the implicit experts, ensuring that each implicit expert always corresponds to the same structured feature direction of the pre-trained weights during training, thus maintaining the stability and consistency of feature interpretation.

[0039] As a further implementation, this step proposes a task-adaptive SVD initialization strategy, which constructs a projection matrix based on the input activation feature distribution of the downstream task training data. The initialization of the weights is performed, and a lightweight streaming accumulation scheme is designed to adapt to the memory constraints of large model fine-tuning, so as to solve the problem of the disconnect between the traditional pre-trained weight SVD initialization and the downstream task distribution.

[0040] Specifically, a small amount of calibration data is sampled from the downstream training set, and the input activation features of the target linear layer are captured through the network's forward hook. Using three sets of GPU-based lightweight accumulators configured for each layer, the accumulated token count, the activation feature summation vector, and the activation outer product matrix are calculated, respectively. This process only requires interaction with the input features. The memory overhead is proportional to the square of the dimension, eliminating the need to store all original activation data, and computation can be completed on consumer-grade GPUs. After the calibration data traversal is completed, the centered input covariance matrix is ​​calculated based on the accumulated statistics, specifically represented as follows:

[0041] In the formula, For the cumulative number of tokens, To activate the feature mean vector, Let be the activation matrix, where each row corresponds to the input activation vector of a token. For the covariance matrix... Perform low-rank SVD decomposition to obtain the right singular vector adapted to the downstream task. With singular value matrix .

[0042] Select the right singular vector forward Column vectors as projection matrices The row vector, i.e. All row vectors maintain the unit norm, and the projection direction is constructed entirely based on the downstream data distribution, eliminating the bias of the general prior during pre-training. A fault-tolerance mechanism is also designed: if there is no task SVD cache, it falls back to the weight SVD initialization scheme, expressed as: This is to ensure the stability of model training.

[0043] As a further implementation method, to balance the general knowledge reuse of the pre-trained model with the ability to explore features specific to downstream tasks, this step introduces a pre-trained weight subspace as a credibility reference on the basis of task-adaptive SVD initialization, designs a layer-by-layer adaptive alignment gating screening mechanism, and hybridly constructs a high-quality initialization projection matrix. Balancing prior knowledge with task suitability.

[0044] Specifically, extract the first part of the singular value decomposition of the pre-trained weights. Column-right singular vectors are used to construct a pre-trained knowledge subspace. Calculate the adaptive SVD decomposition feature vector for each task. The projection length in this pre-trained knowledge subspace is used as the feature alignment degree, specifically expressed as follows:

[0045] In the formula, For feature alignment. Alignment A higher value indicates a higher degree of matching between the current task feature direction and the general knowledge accumulated during pre-training, and a stronger feature credibility; alignment degree If the value approaches 0, it indicates that the feature direction is a blind spot in the model's pre-training and lacks effective prior knowledge support.

[0046] This step abandons the fixed judgment threshold and designs an adaptive alignment threshold for each network layer. This adaptive alignment threshold is a preset multiple of the statistical expectation of the projection length of the random vector in the pre-trained knowledge subspace. It is calculated independently for each network layer, specifically expressed as follows:

[0047] In the formula, The expected value of the projection of a random vector onto the pre-trained subspace is given. A gain of 1.5x can accurately filter out effective feature components that are significantly better than randomness, eliminate invalid noise directions, and ensure the accuracy of the filtering.

[0048] Finally, the hybrid components are adaptively assembled based on feature alignment and an adaptive alignment threshold, while preserving feature alignment. Not less than the adaptive alignment threshold High-quality task SVD components as projection matrix The row vectors enable the transfer and reuse of pre-trained general knowledge; feature alignment. Less than the adaptive threshold The SVD components of the task are judged to be low-quality and unreliable, and should be discarded. The missing components should be filled with Gaussian random vectors with unit norm.

[0049] Through the above task-adaptive SVD initialization, the input activation covariance matrix of the task calibration data is decomposed by streaming SVD, which greatly reduces the memory overhead. Furthermore, the task components that are well aligned with the pre-trained weight subspace are automatically selected using an alignment-gated hybrid initialization strategy, and orthogonal components are backed up to unbiased random initialization, which further improves the learning efficiency of new tasks.

[0050] The final constructed projection matrix It combines pre-trained reliable task feature directions with controlled random exploration directions, which not only relies on prior knowledge to accelerate training convergence, but also makes up for the lack of feature representation in the blind spots of model pre-training through random components, thus greatly improving the adaptability to downstream tasks.

[0051] S2. In the forward propagation stage, the input features are projected onto the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features to participate in the forward calculation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated.

[0052] During the forward propagation phase, input features First, the frozen projection matrix Projecting onto a low-dimensional latent space yields latent features, represented as:

[0053] In the formula, hidden features Each dimension corresponds to a projected activation value of an implicit expert, the size of which is... This reflects the component strength of the input features along the main feature directions of the pre-trained weights. Due to the projection matrix... The row vector is the right singular vector of the pre-trained weights, and the projection result naturally carries the structured prior information of the pre-trained weights.

[0054] In this step, the hidden features are... The absolute values ​​of each dimension are directly used as the basis for routing decisions, eliminating the need for a separate gating network. Specifically, in the latent features... A Top-K sparse routing mechanism is introduced to retain only latent features. The K most valid expert components with the largest absolute values ​​participate in the forward computation, while the remaining redundant components are suppressed. The activation values ​​of the suppressed components are set to zero, and their gradients are truncated during backpropagation to prevent invalid parameter updates. Here, K is less than the total number of implicit experts. The routing decision is entirely determined by the input features. The intrinsic relationship with the direction of pre-trained weight features directly determines that if the input features... If the projection intensity is high along a certain right singular vector direction, it indicates that the implicit expert corresponding to that direction is highly correlated with the current input and should be activated; otherwise, it should not participate in the calculation.

[0055] Through the Top-K sparse routing mechanism described above, each token only activates a portion of the expert components, and different semantic tokens adaptively match different expert combinations, breaking through the expression bottleneck of fixed low rank with extremely low computational overhead.

[0056] As a further implementation, to address the collapse of the Top-K sparse routing mechanism and the imbalance in expert utilization mentioned above, this step introduces a load-balancing routing bias mechanism. Specifically, a learnable bias term is configured for each implicit expert dimension. The routing ranking criteria are optimized from the original absolute value of latent features to a combination of the absolute value of latent features and the corresponding bias term. .

[0057] During training, the activation frequency of each implicit expert is counted in real time, and the bias term is dynamically updated. The bias weight is reduced for implicit experts that are overactivated (i.e., their activation frequency exceeds a preset target), and the bias weight is increased for implicit experts that are idle for a long time (i.e., their activation frequency is below a preset target). This forces a balance in the utilization of all experts, ensuring the integrity and effectiveness of the multi-expert system. This bias update only applies to the bias term itself and does not interfere with the main gradient flow.

[0058] Through the aforementioned load balancing routing bias mechanism, the cumulative usage frequency of each implicit expert is dynamically counted in each forward propagation. Overloaded implicit experts are penalized, while underloaded implicit experts are incentivized, effectively preventing expert collapse during the training process and ensuring that all implicit experts are fully trained.

[0059] S3. The outputs of the activated experts are weighted and fused after being mapped by the output matrix to obtain the final output of the target layer.

[0060] The activated expert components, after Top-K sparsification, are represented by the output matrix. This is then mapped back to the target output space. The final forward output of the target layer is based on the projection matrix. Latent features and output matrix after sparsification by sparse routing mechanism Specifically, the projection matrix is ​​determined. The input features are mapped to a low-dimensional latent space, and the Top-K sparse routing mechanism sparsifies the latent features, outputting a matrix. The sparsified latent features are then mapped back to the target output space. Finally, the output of the target layer is represented as:

[0061] In the formula, This is a scaling hyperparameter used to stabilize training gradients. In a typical configuration ( Under this approach, a single token activates only 25% of the expert components, and different semantic tokens adaptively match different expert combinations, breaking through the expression bottleneck of fixed low rank with extremely low computational overhead.

[0062] During backpropagation, the gradient passes only through the output matrix corresponding to the activated component. The gradient of the suppressed component is truncated to zero due to the column propagation of the projection matrix. By freezing the entire process, the number of trainable parameters is significantly reduced, achieving ultimate optimization of parameter efficiency.

[0063] As a further implementation, this step also introduces a spectral modulation mechanism to further optimize the training process. Specifically, a frequency-dependent learning rate scaling factor is applied to each rank dimension through a gradient backhook, allowing rank dimensions with smaller singular values ​​to obtain higher learning rates. In this process, the scaling factor is negatively correlated with the singular values: the smaller the singular value, the larger the scaling factor, ensuring that directions with smaller singular values ​​are updated more fully when adapting to downstream tasks; the larger the singular value, the smaller the scaling factor, to protect the learned general knowledge from being destroyed.

[0064] As a further implementation, this embodiment also introduces an overlap-aware orthogonalization strategy to address the catastrophic forgetting problem in multi-task sequential fine-tuning scenarios. Specifically, it tracks the set of SVD components selected for each task, calculates the Inter-task SVD component overlap index (IoU), and identifies shared and unique components. When sequentially learning a new task, it applies gradient zeroing protection to the unique components of the previous task during the current task training, while maintaining the plasticity of cross-task shared components to achieve positive knowledge transfer. This strategy allows shared components to naturally assume the role of cross-task knowledge transfer without introducing any additional inference parameters, while ensuring that task-specific knowledge is not disturbed by subsequent interference, breaking the dilemma of "protection and transfer being mutually exclusive" in traditional full gradient orthogonalization methods.

[0065] As a further implementation, after training, when performing the inference task, inference calculations are performed according to the forward propagation method described in step S2. Due to the projection matrix... Frozen and output matrix Fine-tuning has been completed; the inference process requires no additional gating network computation, directly using input features... The absolute value of the projected latent features is used as the routing basis, and only the k most relevant latent experts to the current input are activated for computation. Compared with existing MoE schemes, the inference process in this embodiment does not involve the additional computation and communication overhead of independent gating networks, and has lower inference latency, making it particularly suitable for edge device deployment.

[0066] To verify the effectiveness of the low-rank adaptation method based on singular value decomposition implicit routing (hereinafter referred to as "this method") disclosed in this embodiment, this embodiment conducted systematic experimental verification on the LLaMA 3.1-8B model using 4-bit NF4 quantization deployment, and on representative benchmarks such as MMLU multi-task knowledge understanding (57 subjects, 14042 test questions). The experimental configuration was rank r=32, top_k=8, lora_alpha=64, and the adapter was injected only into the attention projection layer (q_proj,k_proj,v_proj,o_proj). The training data was the official training set of the dataset. The experimental results are shown in Table 1.

[0067] Table 1 Experimental verification results

[0068] Experimental data on the MMLU benchmark show that the proposed method achieves an overall accuracy of 56.7% when training only the B matrix (the total number of trainable parameters is about 50% of the standard LoRA same-rank configuration). The effectiveness of the SVD guided routing mechanism is fully verified under a lightweight configuration that injects adapters into only 4 attention projection layers.

[0069] Example 2 In a typical embodiment of the present invention, this embodiment discloses a low-rank adaptation system based on singular value decomposition implicit routing, comprising: The projection matrix construction module is configured to: perform singular value decomposition on the pre-trained weight matrix of the target layer, extract the right singular vector and the corresponding singular value, select the number of singular value components that are equal to the rank that are adapted to the low rank as implicit experts, combine them to generate the projection matrix, freeze the parameters of the projection matrix throughout the training process, and only retain the output matrix to participate in fine-tuning. The implicit routing module is configured as follows: during the forward propagation stage, the input features are projected to the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features to participate in the forward calculation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated. The fusion output module is configured to: weight and fuse the outputs of the activated experts after mapping through the output matrix to obtain the final output of the target layer; In this approach, implicit experts do not require a separate gating network for route allocation; the route decision is directly determined by the absolute value of the latent features after the input features are projected through the projection matrix.

[0070] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A low-rank adaptation method based on singular value decomposition implicit routing, characterized in that, include: Singular value decomposition is performed on the pre-trained weight matrix of the target layer to extract the right singular vector and its corresponding singular value. The number of singular value components that are as many as the rank that are adapted to the low rank are selected as implicit experts and combined to generate the projection matrix. The parameters of the projection matrix are frozen throughout the training process, and only the output matrix is ​​retained for fine-tuning. During the forward propagation phase, the input features are projected onto the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features for forward computation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated. The final output of the target layer is obtained by weighted fusion of the outputs of the activated experts after mapping through the output matrix; In this approach, implicit experts do not require a separate gating network for route allocation; the route decision is directly determined by the absolute value of the latent features after the input features are projected through the projection matrix.

2. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 1, characterized in that, In the Top-K sparse routing mechanism, the number of activated experts is less than the total number of implicit experts, and a single token only activates a portion of the expert components, thus achieving dynamic sparse forward propagation.

3. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 1, characterized in that, The projection matrix is ​​generated by combining selected singular value components. The row vectors of the projection matrix are the transpose of the first preset number of right singular vectors, and all row vectors maintain the unit norm. The preset number is equal to the rank of the low-rank fit. The projection matrix does not participate in gradient updates throughout the training process.

4. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 3, characterized in that, The initialization of the projection matrix is ​​constructed based on the input activation feature distribution of the downstream task training data. Simultaneously, a lightweight streaming accumulation scheme is designed to adapt to the memory constraints of fine-tuning large models, including: A small amount of calibration data is sampled from the downstream training set, and the input activation features of the target linear layer are captured through the network forward hook. The cumulative token count, activation feature summation vector and activation outer product matrix are counted by three sets of GPU-side lightweight accumulators configured for each layer. After the calibration data traversal is completed, the centered input covariance matrix is ​​calculated based on the accumulated statistics, and low-rank singular value decomposition is performed on the covariance matrix to obtain the right singular vector and singular value matrix adapted to the downstream task. A predetermined number of column vectors before the right singular vector are selected as the row vectors of the projection matrix. All row vectors maintain the unit norm, and the projection direction is constructed entirely based on the downstream data distribution to eliminate the bias of the general prior of pre-training. At the same time, a fault tolerance mechanism is designed so that if there is no task SVD cache, it will fall back to the weight SVD initialization scheme.

5. The low-rank adaptation method based on implicit routing using singular value decomposition as described in claim 4, characterized in that, By introducing a pre-trained weight subspace as a credibility reference, a layer-by-layer adaptive alignment gating screening mechanism is designed. A high-quality initialization projection matrix is ​​constructed through hybrid methods to balance prior knowledge and task adaptability. Specifically, this includes: Extract the first preset number of right singular vectors from the singular value decomposition of the pre-trained weights, construct a pre-trained knowledge subspace, and calculate the projection length of the adaptive singular value decomposition feature vector of each task in the pre-trained knowledge subspace as the feature alignment degree. For each network layer, an adaptive alignment threshold is designed that is adapted to the current network layer. It is a preset multiple of the statistical expectation of the projection length of the random vector in the pre-trained knowledge subspace. Each network layer is calculated independently. Task singular value components with feature alignment not less than the adaptive alignment threshold are retained as row vectors of the projection matrix to realize the transfer and reuse of pre-trained general knowledge; task singular value components with feature alignment less than the adaptive threshold are discarded and the gaps are filled with unit norm Gaussian random vectors.

6. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 1, characterized in that, Learnable bias terms are configured for each implicit expert dimension, and the routing ranking is based on the combination of the absolute value of the latent feature and the corresponding bias term. During training, the activation frequency of each implicit expert is counted in real time, the bias terms are dynamically updated, the bias weights of over-activated implicit experts are reduced, and the bias weights of long-term idle implicit experts are increased, thus forcibly balancing the utilization rate of all experts.

7. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 1, characterized in that, The final forward output of the target layer is determined based on the projection matrix, the latent features sparsified by the sparse routing mechanism, and the output matrix. Specifically, the projection matrix maps the input features to a low-dimensional latent space, the Top-K sparse routing mechanism sparsifies the latent features, and the output matrix maps the sparsified latent features back to the target output space.

8. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 7, characterized in that, The output of the target layer is represented as: In the formula, This is the pre-trained weight matrix for the target layer. As input features, For the output matrix, This is a Top-K sparse routing mechanism. For the projection matrix, The rank that is adapted to the lower rank. For scaling hyperparameters.

9. The low-rank adaptation method based on singular value decomposition implicit routing as described in claim 1, characterized in that, The system tracks the set of singular value components selected for each task, calculates the index of the degree of overlap of singular value components between tasks, and identifies shared components and unique components. When learning new tasks sequentially, the unique components of the previous task are protected by setting the gradient to zero during the training of the current task, while the shared components across tasks are kept plastic to achieve positive knowledge transfer.

10. A low-rank adaptation system based on singular value decomposition implicit routing, characterized in that, include: The projection matrix construction module is configured to: perform singular value decomposition on the pre-trained weight matrix of the target layer, extract the right singular vector and the corresponding singular value, select the number of singular value components that are equal to the rank that are adapted to the low rank as implicit experts, combine them to generate the projection matrix, freeze the parameters of the projection matrix throughout the training process, and only retain the output matrix to participate in fine-tuning. The implicit routing module is configured as follows: during the forward propagation stage, the input features are projected to the low-dimensional latent space through the projection matrix to obtain latent features. A Top-K sparse routing mechanism is introduced on the latent features to retain only the k effective expert components with the largest absolute value of the latent features to participate in the forward calculation. The activation values ​​of the suppressed components are set to zero and the gradient is truncated. The fusion output module is configured to: weight and fuse the outputs of the activated experts after mapping through the output matrix to obtain the final output of the target layer; In this approach, implicit experts do not require a separate gating network for route allocation; the route decision is directly determined by the absolute value of the latent features after the input features are projected through the projection matrix.

Images