A Low-Rank Adaptive Training Method and System Based on Residual Memory Sharing in Expert Hybrid Models

By calculating the residual memory budget in the expert hybrid model and instantiating shared low-rank adaptation channels, combined with dynamic task scheduling based on routing skew metrics, the problems of unstable memory utilization and sparse training signals are solved, achieving more efficient memory resource utilization and training efficiency.

CN122635481BActive Publication Date: 2026-11-10BEIJING TREND TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202610810107.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-06-05
Publication Date
2026-11-10
Estimated Expiration
2046-06-05

AI Technical Summary

Technical Problem

In the heterogeneous fine-tuning environment of expert hybrid models, existing technologies suffer from problems such as unstable utilization of residual GPU memory, sparse training signals, severe impact of routing skew, and imperfect heterogeneous path synchronization fusion and gradient splitting mechanisms, resulting in low efficiency of GPU memory resource utilization.

Method used

By calculating the residual memory budget of the graphics processor, instantiating a shared low-rank adaptation channel, and dynamically scheduling tasks based on the routing skew metric, the independent calculation and gradient splitting of the routing adaptation path and the shared low-rank adaptation channel are achieved, ensuring full-terminal training signal coverage and stable memory utilization.

Benefits of technology

It improves the stability of GPU memory utilization and training efficiency, alleviates the waiting problem caused by routing skew, and enhances overall execution efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122635481B_ABST
    Figure CN122635481B_ABST
Patent Text Reader

Abstract

A training method and system based on residual memory shared low-rank adaptation using an expert hybrid model includes: a residual memory budget unit calculating the residual memory budget; a shared low-rank channel configuration unit instantiating a shared low-rank adaptation channel; a training signal coverage statistics unit obtaining a routing skew metric; a skew-aware scheduling unit scheduling the tasks of the routing adaptation path and the execution granularity of the shared low-rank adaptation channel; a shared low-rank adaptation execution unit sending the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel respectively; the shared low-rank adaptation channel, independent of route selection, performing calculations on all input tokens in the current batch to generate a shared low-rank output; a forward fusion unit fusing the output of the routing adaptation path with the shared low-rank output in the forward phase to form a layer output; and a backward gradient branch unit splitting the gradient of the layer output to the routing adaptation path and the shared low-rank adaptation channel respectively in the backward phase for parameter updates.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence large model training and efficient parameter fine-tuning technology, and more specifically, to the utilization of residual video memory, shared low-rank adaptation and routing skew-aware training scheduling of expert hybrid large models in CPU-GPU heterogeneous fine-tuning environments. Background Technology

[0002] In recent years, efficient parameter fine-tuning techniques, such as low-rank adaptation and LoRA, have emerged. By adding trainable low-rank matrices alongside the original linear layers, the fine-tuning process requires updating only a small number of additional parameters. When fine-tuning expert hybrid models, low-rank adaptation is typically added to routing experts, shared experts, or other linear layers. Meanwhile, in heterogeneous fine-tuning environments, the large routing expert weights reside in the CPU's main memory and are executed by the CPU, while the GPU continues to perform attention, shared paths, and some local computations. This partitioning of the physical computational architecture inevitably leaves some residual space in the GPU's memory, and the GPU has a usable computational window while waiting for the CPU's expert computation results.

[0003] However, existing technologies generally suffer from the following technical defects when using the aforementioned heterogeneous environments to efficiently fine-tune the parameters of expert hybrid models.

[0004] The utilization of residual GPU memory is unstable and lacks budget constraints: Existing popular expert caching schemes typically use residual GPU memory to cache a small number of high-frequency routing experts, but their fine-tuning benefits are highly dependent on the popularity distribution of experts and cache hit rate. In vertical domain data fine-tuning, routing popularity often changes dynamically with the training phase and data distribution, causing uncached experts to still have to take the original slow path, making this cache-based GPU memory utilization method extremely unstable. In addition, existing dense adapter and shared bypass schemes usually do not take into account the residual GPU memory budget when determining the size of the adaptation channels. If the configuration scale is too large, it will easily crowd out local paths and temporary workspaces, while if the scale is too small, the training benefits will be limited.

[0005] The training signal is sparse and severely affected by routing skew: the routing mechanism of expert hybrid models dictates that each term will only pass through a few experts. Existing common low-rank adaptation schemes, such as LoRA and QLoRA, and expert-aware low-rank adaptation schemes, such as MixLoRA, inevitably retain the original sparse routing paths. This results in the routing-type low-rank adaptation parameters only receiving gradient updates, i.e., obtaining the training signal, when the term is routed to the corresponding expert. In vertical domain fine-tuning, data distribution skew often leads to a few popular experts receiving a large number of terms, while less popular experts form long-tailed small tasks, making the training signal obtained by the trainable parameters corresponding to the remaining GPU memory extremely sparse, reducing the utilization efficiency of GPU memory resources.

[0006] Increasing the global parameter count is difficult to translate into an equivalent number of effective activations for individual tokens: While simply increasing the low-rank adaptation rank of all routing experts increases the total number of parameters macroscopically, the actual activated adaptation parameters for a single token are still limited by the physical isolation of the routing mechanism. Therefore, increasing the expert rank or caching popular experts makes it difficult to distinguish between the number of parameters and the number of effective activations, and it is impossible to stably translate the memory investment into global fine-tuning benefits.

[0007] The heterogeneous path synchronization fusion and gradient splitting mechanisms are inadequate: When introducing a shared adaptation path that is independent of routing, due to the differences in execution timing and data flow between it and the original routing expert path, the forward output needs to be correctly merged, and the backward gradient needs to be accurately imported into the routing adaptation parameters and shared adaptation parameters respectively. Most existing lightweight trainable module solutions lack a rigorous forward fusion, backward gradient branching, and parameter version update mechanism built around the shared channel of residual GPU memory.

[0008] Therefore, the existing technology has problems and needs further improvement and development. Summary of the Invention

[0009] (I) Purpose of the invention: In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a method and system for low-rank adaptation training based on residual memory sharing of expert hybrid models.

[0010] (II) Technical Solution: To solve the above-mentioned technical problems, this technical solution provides a low-rank adaptation training method based on residual memory sharing of expert hybrid models, including the following steps: Step 1: Calculate the residual memory budget based on the runtime memory overhead of the graphics processor, and instantiate a shared low-rank adapter channel in the graphics processor based on the residual memory budget. Step 2: Statistically analyze the routing distribution of the current batch of input tokens in the expert hybrid model to obtain a routing skew metric, and schedule the tasks of the routing adaptation path and the execution granularity of the shared low-rank adaptation channel according to the routing skew metric. Step 3: Send the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel respectively, and perform calculations in the routing adaptation path according to the scheduled tasks to generate the routing path output, and perform segmentation calculations on all input tokens in the current batch in the shared low-rank adaptation channel according to the scheduled execution granularity, independent of the routing selection, to generate the shared low-rank output; Step 4: In the forward phase, the output of the routing adaptation path is merged with the shared low-rank output to form a layer output; Step 5: In the reverse phase, the gradients output by the layer are split into the routing adaptation path and the shared low-rank adaptation channel, respectively, for parameter updates.

[0011] Specifically, the calculation of the residual video memory budget based on the runtime video memory overhead of the graphics processor includes: After the expert hybrid model is loaded and before the shared low-rank adapter channel is instantiated, the execution flow of the current graphics processor is synchronized, and the current free video memory and releasable cache are read through the runtime video memory query interface; Perform benchmark micro-batch training to test and record the base local path peak, activation peak, operator temporary workspace peak, and communication buffer peak, respectively; Calculate the sum of the current free video memory and the free cache, and subtract the peak usage and the preset video memory protection margin from the sum of the current free video memory and the free cache to calculate the available residual video memory budget.

[0012] Specifically, the instantiation of a shared low-rank adapter channel in the graphics processor based on the residual video memory budget includes: Construct a set of candidate configurations that includes the number of shared channels, shared rank, intermediate dimension, and parameter precision; Calculate the total video memory usage for each candidate configuration and eliminate candidate configurations whose total video memory usage exceeds the remaining video memory budget limit. For candidate configurations that meet the specified ratio, a comprehensive configuration score is calculated based on a preset priority mode of quality, throughput, or user-specified priority. The target configuration with the highest score is selected for parameter and buffer instantiation allocation, and a shared low-rank adaptation channel is generated.

[0013] The shared low-rank adaptation channel is attached to the output merging point of the expert mixing layer in the expert mixing model, or to the bypass position before the residual summation in the expert mixing layer.

[0014] The step of statistically analyzing the routing distribution of the current batch of input terms in the expert hybrid model to obtain a routing skew metric includes: The routing skew metric is calculated by counting the number of times each expert is accessed by the route in the current batch of input terms. Experts with a greater than zero number of accesses are considered active experts. The maximum number of accesses for a single expert is divided by the average number of accesses for all active experts.

[0015] The scheduling of the route adaptation path task specifically includes: The thresholds for popular experts and long-tail experts are calculated based on the average and standard deviation of the number of visits to active expert terms. Experts whose access count is greater than or equal to the popular expert threshold are marked as popular experts, and experts whose access count is less than or equal to the long-tail expert threshold are marked as long-tail experts. The routing adaptation tasks of the popular experts are split into multiple subtasks according to word segments or matrix blocks and submitted in parallel, and the routing adaptation tasks of multiple long-tail experts are merged and submitted.

[0016] Specifically, scheduling the execution granularity of the shared low-rank adaptation channel includes: The number of micro-batch tokens executed in a single submission of the shared low-rank adaptation channel is calculated based on the total number of tokens in the current batch, the preset shared channel temporary buffer capacity, the residual memory budget, and the routing skew metric.

[0017] The step of merging the output of the route adaptation path with the shared low-rank output in the forward phase further includes an alignment step: If the word order of the route adaptation path is disrupted due to expert aggregation calculation, the route path output is restored to the original batch order based on the word index recorded when the route distribution is statistically analyzed, so as to ensure that the route path output has the same word order and micro-batch range as the shared low-rank output.

[0018] The fusion in the forward stage can be achieved using any one of the following methods: direct summation, scaling summation, gated summation, or residual summation.

[0019] Specifically, updating parameters during the reverse phase includes: The routing expert low-rank adaptation parameter group and the shared low-rank adaptation channel parameter group are respectively identified as independent parameter groups; For routing expert low-rank parameters that have not received any tokens in the current training step, skip the corresponding optimizer state update; After completing the gradient update of the parameters of the shared low-rank adaptation channel, refresh the parameter version, scaling factor and cache status of the shared low-rank adaptation channel.

[0020] A low-rank adaptation training system based on residual memory sharing of an expert hybrid model includes: The residual memory budget unit is used to calculate the residual memory budget based on the runtime memory overhead of the graphics processor. A shared low-rank channel configuration unit is used to instantiate a shared low-rank adapter channel in the graphics processor based on the residual video memory budget. The training signal covers the statistical unit, which is used to statistically analyze the routing distribution of the current batch of input words in the expert hybrid model to obtain a routing skew metric. A skew-aware scheduling unit is used to schedule the tasks of the route adaptation path and the execution granularity of the shared low-rank adaptation channel according to the route skew metric. The shared low-rank adaptation execution unit is used to send the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel respectively, and to perform calculations in the routing adaptation path according to the scheduled tasks to generate the routing path output, and to perform segmentation calculations on all input tokens in the current batch in the shared low-rank adaptation channel according to the scheduled execution granularity and independent of the routing selection to generate the shared low-rank output; The forward fusion unit is used to fuse the output of the routing adaptation path with the shared low-rank output during the forward phase to form a layer output; The reverse gradient branch unit is used to split the gradient output by the layer into the routing adaptation path and the shared low-rank adaptation channel respectively during the reverse phase for parameter updates.

[0021] (III) Beneficial Effects: This invention provides a low-rank adaptation training method and system based on residual memory sharing in an expert hybrid model, which includes the following effects: To improve the coverage of low-rank adaptation training signals, shared low-rank channel calculation is performed on all words in each batch, so that the shared channel parameters can receive dense gradient backpropagation of all words in each step of the reverse stage, making the parameter updates corresponding to the residual memory more intensive, which is more suitable for fine-tuning tasks with a small amount of domain data. To improve the stability of residual memory utilization, the calculated residual memory is used to build a shared channel independent of route selection, instead of caching a small number of popular route experts. Since the shared channel is executed on all terms, its training gain is determined by the full batch term coverage, and no longer directly depends on the cache hit rate of a single popular expert. Therefore, when the routing popularity of the domain data changes, the shared channel can still obtain a stable training signal, effectively reducing the fluctuation of gains caused by traditional caching schemes. To alleviate execution waiting caused by routing skew, tasks are dynamically scheduled based on statistically obtained routing skew metrics and related thresholds. Popular expert tasks are split into multiple parallelizable word segments or matrix blocks, and long-tail expert tasks are merged and submitted to reduce scheduling overhead. At the same time, shared channels are executed independently in micro-batches, effectively filling the waiting window of the graphics processor, significantly reducing the synchronous waiting of heterogeneous nodes caused by routing skew, and improving overall execution efficiency. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the steps of a low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to the present invention. Figure 2This invention provides a training method for residual memory sharing low-rank adaptation based on an expert hybrid model and an overall architecture diagram of residual memory sharing low-rank adaptation in the system. Figure 3 This invention provides a method for low-rank adaptation training based on shared residual memory using an expert hybrid model and a flowchart of the process for selecting residual memory budget and shared channel size in the system. Figure 4 This invention relates to a low-rank adaptation training method based on residual memory sharing of an expert hybrid model and a skew-aware adaptation task scheduling graph in the system. Figure 5 This invention relates to a low-rank adaptation training method based on residual memory sharing in an expert hybrid model and a forward merging and backward gradient splitting graph in the system. Figure 6 This is a comparison diagram of the low-rank adaptation training loss in the system based on the residual memory sharing of an expert hybrid model according to the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to preferred embodiments. More details are set forth in the following description in order to provide a full understanding of the present invention. However, the present invention can obviously be implemented in many other ways different from those described herein. Those skilled in the art can make similar extensions and derivations based on actual application situations without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0024] The accompanying drawings are schematic diagrams of embodiments of the present invention. It should be noted that these drawings are for illustrative purposes only and are not drawn to scale, and should not be construed as limiting the actual scope of protection of the present invention.

[0025] A method for training with shared low-rank adaptation based on residual memory using an expert hybrid model is applicable to training systems based on shared low-rank adaptation using residual memory using an expert hybrid model. The training system includes a residual memory budget unit, a shared low-rank channel configuration unit, a training signal coverage statistics unit, a skew-aware scheduling unit, a shared low-rank adaptation execution unit, a forward fusion unit, and a backward gradient branch unit.

[0026] The residual memory budget unit includes a residual memory budget module. This module reads the total video memory and currently free video memory through the graphics processor runtime memory query interface. This query interface can be a graphics processor driver management interface, a runtime memory interface, or a memory query function encapsulated by the training framework. Before querying, the current graphics processor execution flow is synchronized to avoid reading distortion caused by asynchronous operators not yet releasing temporary buffers. Subsequently, after initialization of the base model local path, activation checkpoints, communication buffer, and temporary working area, a baseline micro-batch is executed, and the peak values ​​of the base local path, activation, temporary working area, and communication buffer are recorded respectively. Unlike the method of statically deducting only from the total video memory, the residual memory budget module does not subtract a fixed constant from the total video memory all at once. Instead, before the shared channel is instantiated, it calculates the residual video memory available for the shared low-rank adaptation channel based on the currently free video memory, releasable cache, various runtime peak values, and protection margin, avoiding misallocation of temporary peak values ​​in the training steps to the shared channel. The residual video memory budget unit includes a residual video memory budget table, which records the total video memory of the graphics processor, the basic path usage, the activation usage, the working area usage, the protection margin, the available residual video memory, and the timestamp.

[0027] The shared low-rank channel configuration unit includes a shared adaptation access module and a shared low-rank channel configuration module. The shared adaptation access module reads model structure, expert layer positions, number of experts per layer, top-k routes, low-rank adaptation base rank, number of candidate shared channels, candidate shared channel rank, batch size, sequence length, parameter precision, and memory protection margin from model configuration files, model architecture descriptors, efficient parameter fine-tuning configurations, training parameters, and runtime memory probing results. Model structure and expert layer positions are primarily derived from layer type, hidden dimensions, and expert layer indices in the model configuration file. The number of experts per layer and top-k routes are derived from router configurations or model object attributes. The low-rank adaptation base rank, scaling factor, and target linear layer are derived from efficient parameter fine-tuning configurations. The number of candidate shared channels, candidate shared channel rank, and candidate intermediate dimensions are derived from the shared adaptation configuration. Batch size, sequence length, gradient accumulation steps, and micro-batch partitioning are derived from training parameters. Parameter precision is derived from training precision or adapter precision configurations. Memory protection margin is derived from runtime memory probing results or a preset safety ratio. The shared adaptation access module outputs a shared adaptation access configuration record and determines the insertion point of the shared low-rank adaptation channel into the expert mixing layer output merging point, the position before residual summation, or the specified linear projection bypass. The shared low-rank channel configuration module enumerates candidate configurations and calculates the total memory usage of each candidate configuration based on the remaining memory budget, hidden dimension, intermediate dimension, low-rank adaptation rank, parameter precision, number of channels, enabled layer set, and temporary buffer requirements. A candidate configuration consists of the number of channels, shared rank, intermediate dimension, enabled layer set, parameter precision, temporary buffer limit, and output scaling factor. The shared low-rank channel configuration module first eliminates candidates exceeding the remaining memory budget, and then calculates a score based on quality gain, full token coverage gain, execution time overhead, and memory usage. For candidate configurations that meet the budget, the shared low-rank channel configuration module calculates the configuration score according to training quality priority, throughput priority, or user-specified priority, and outputs the number of channels, rank, intermediate dimension, parameter precision, layer position, output scaling factor, and output merging method. The shared low-rank channel configuration unit includes a shared low-rank channel configuration table, which records candidate number, enabled layer set, number of channels, rank, intermediate dimension, parameter precision, layer position, output scaling factor, budget usage ratio, estimated new execution time, occupied video memory, and execution device.

[0028] The training signal coverage statistics unit includes a training signal coverage statistics module. In each expert mixing layer or training step, the training signal coverage statistics module reads the expert ID matrix and routing weight matrix output by the router, accumulates the number of word accesses for each selected expert ID, and forms the number of words received by each expert, the active expert set, the popular expert set, the long-tail expert set, the routing skew metric, and the number of words covered by the shared channel. The training signal coverage statistics unit includes a routing word distribution table, which records the number of words received by each expert in each layer, the weighted number of words, the word index list, the active expert set, the popular expert set, the long-tail expert set, and the skew metric.

[0029] The skew-aware scheduling unit includes a skew-aware adaptation configuration module and a shared channel execution scheduling module. The skew-aware adaptation configuration module calculates the popular expert threshold, long-tail expert threshold, and task segmentation granularity based on the routing term distribution, shared channel coverage, and residual memory budget, and determines the shared channel rank, number of channels, scaling factor, activation level, and execution granularity accordingly. The task segmentation granularity is used for routing adaptation paths, representing the term segment length or matrix tile size contained within a routing expert task; the execution granularity is used for shared low-rank channels, representing the batch, micro-batch, or term segment range submitted to the shared channel for execution at one time. For expert low-rank adaptation calculations, segmentation can be done by term segment or micro-batch dimension; for shared low-rank channels, execution can be done by batch or micro-batch. The shared channel execution scheduling module submits forward and reverse tasks for the shared low-rank channels based on training steps, micro-batch range, memory usage, and execution flow status, and records completion events and output buffers. The skew-aware scheduling unit includes an adaptation task descriptor, which records the task type, layer number, expert number, lexical range, weight block range, execution device, input pointer, output pointer, gradient pointer, and dependent events.

[0030] The shared low-rank adaptation execution unit includes a shared low-rank adaptation execution module. This module creates one or more shared low-rank adaptation channels in the graphics processor's residual memory. These shared low-rank adaptation channels are independent of routing expert numbers, execute on all terms, and are aligned with the routing adaptation path at the forward merge point. The shared low-rank adaptation execution unit includes a shared channel execution state table, which records the parameter version, input micro-batch range, execution flow, completion event, output buffer, and gradient buffer of the shared low-rank channel.

[0031] The forward fusion unit includes a route adaptation path alignment module and a forward fusion module. The adaptation path alignment module records the tensor shape, lexical order, batch range, micro-batch number, scaling factor, and merging sequence among the route expert output, expert low-rank adaptation output, and shared low-rank channel output. The tensor shape refers to the dimension record of the outputs to be merged. For example, the route path output Y_r and shared path output Y_s of the b-th micro-batch in layer l should both be [m_b, d_out], where m_b is the number of lexical terms in the micro-batch, and d_out is the dimension of the target linear layer output. The lexical order refers to the original batch lexical index sequence corresponding to each row of the output matrix, used to prevent the route expert from disrupting the original order after expert aggregation calculation. The batch range refers to the start and end indices of the global batch lexical terms covered by the output. The micro-batch number refers to the micro-batch sequence number under gradient accumulation or memory partitioning. The scaling factor refers to beta_s or the gate weight g used when the shared channel output participates in the merging. The merging sequence refers to the dependency order among the shared channel forward task submission, routing expert result synchronization, two output events readiness, forward fusion execution, and backward gradient branch execution. It occurs before forward fusion and continues into the gradient branch, ensuring that the two paths are merged at the same layer, in the same batch of terms, and on the same hidden dimension. The forward fusion module acquires the routing path output and the shared low-rank channel output, and performs direct summation, scaled summation, gated summation, residual summation, normalized summation, or spliced ​​projection summation according to preset rules to form the current layer output. The forward fusion unit includes a forward merging descriptor, which records the routing path output, shared low-rank channel output, terminology order, micro-batch number, completion event, scaling factor, merging method, output buffer, and subsequent layer input pointer.

[0032] The backward gradient branching unit includes a backward gradient branching module and a shared adaptation parameter update module. The backward gradient branching module receives the current layer's output gradient, calculates the route adaptation path gradient and the shared low-rank channel gradient according to the forward fusion method, writes the gradient corresponding to the route term into the route expert low-rank parameter gradient buffer, and writes the shared path gradients corresponding to all terms into the shared channel gradient buffer. The shared adaptation parameter update module updates the shared low-rank channel parameters after the gradient accumulation, gradient pruning, and optimizer steps, and refreshes the shared channel parameter version, scaling factor, cache status, and backend visibility pointer. For route expert low-rank parameters that have not received any terms, this parameter update step can be skipped. Because the shared low-rank channel parameters cover all terms, gradient updates can be obtained in each effective training batch. The backward gradient branching unit includes a backward gradient split descriptor, which records the output gradient, route path gradient entry, shared channel gradient entry, gradient synchronization event, parameter group number, whether it participates in this update step, and parameter version flag.

[0033] A low-rank adaptation training method based on residual memory sharing in an expert hybrid model, such as... Figure 1 As shown, it includes the following steps.

[0034] Step 1: The residual memory budget unit calculates the residual memory budget based on the runtime memory overhead of the graphics processor, and the shared low-rank channel configuration unit instantiates a shared low-rank adapter channel in the graphics processor based on the residual memory budget.

[0035] Step 2: The training signal coverage statistics unit counts the routing distribution of the current batch of input tokens in the expert hybrid model to obtain a routing skew metric. The skew-aware scheduling unit schedules the tasks of the routing adaptation path and the execution granularity of the shared low-rank adaptation channel according to the routing skew metric.

[0036] Step 3: The shared low-rank adaptation execution unit sends the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel respectively, and performs calculations in the routing adaptation path according to the scheduled tasks to generate the routing path output, and performs segmentation calculations on all input tokens in the current batch in the shared low-rank adaptation channel according to the scheduled execution granularity, independent of the routing selection, to generate the shared low-rank output.

[0037] Step four: In the forward fusion phase, the forward fusion unit fuses the output of the routing adaptation path with the shared low-rank output to form a layer output.

[0038] Step 5: In the reverse phase, the reverse gradient branch unit splits the gradient output by the layer to the routing adaptation path and the shared low-rank adaptation channel respectively for parameter updates.

[0039] like Figure 2 As shown, after the router selects a routing expert, the input tokens enter the routing expert and the expert low-rank adaptation path (i.e., the routing adaptation path), and the other enters the shared low-rank adaptation channel in the graphics processor's residual video memory. The shared low-rank adaptation channel is independent of routing selection and is executed on all tokens. The results of the two are merged to form the layer output.

[0040] Step 1 includes Step 1.1: Obtaining runtime video memory overhead and budget calculation; Step 1.2: Evaluating candidate configurations and selecting scale; Step 1.3: Instantiating and attaching shared channels.

[0041] Step 1.1 specifically includes: after the expert hybrid model is loaded and before the shared low-rank adaptation channel is instantiated, synchronizing the execution flow of the current graphics processor and reading the current free video memory and freeable cache through the graphics processor runtime memory query interface. Benchmark micro-batch training is performed to test and record the peak values ​​of the basic local path, activation, operator temporary workspace, and communication buffer respectively. The residual memory budget unit calculates the sum of the current free video memory and freeable cache, compares it with the total video memory of the graphics processor, and takes the smaller value as the base of available video memory; subtracting the peak values ​​of the basic local path, activation, operator temporary workspace, communication buffer, and the preset video memory protection margin from the sum of the current free video memory and freeable cache, the available residual memory budget is calculated. Before performing memory probing, preliminary operating indicators such as network hidden dimension, basic rank, micro-batch size, and parameter accuracy are first parsed from the expert hybrid model configuration file, parameter efficient fine-tuning configuration, and training parameters. After completing the configuration reading and loading the expert hybrid model, the execution flow of the current graphics processor is synchronized, and the current free video memory and releasable cache are read through the runtime video memory query interface.

[0042] Specifically, the residual memory budget unit first reads the hidden dimensions, number of expert layers, number of experts per layer, top-k routes, and router type from the expert hybrid model configuration file or expert hybrid model object. Top-k refers to the selection of the top k experts from the expert score vector. It reads the low-rank adaptation base rank, scaling factor, target linear layer, and parameter precision from the parameter efficient fine-tuning configuration. It reads the sequence length, batch size, gradient accumulation steps, and micro-batch splits from the training parameters. Finally, it reads the number of candidate shared channels, candidate shared channel rank, candidate intermediate dimensions, and memory protection margin from the shared adaptation configuration. The hidden dimension, number of expert layers, and expert layer index are derived from the layer type, hidden dimension, and expert hybrid layer label in the expert hybrid model structure configuration; the number of experts per layer and the top-k of routes are derived from the number of experts and the number of experts selected per term in the router configuration or expert hybrid model object; the low-rank adaptation base rank, scaling factor, target linear layer, and parameter precision are derived from the PEFT or LoRA configuration; the number of candidate shared channels, candidate shared channel rank, candidate intermediate dimension, and candidate enabled layer set are derived from the shared adaptation configuration of this invention; the sequence length, batch size, gradient accumulation steps, and micro-batch partitioning are derived from the training parameters; and the memory protection margin is derived from the runtime memory probing results or the safety ratio in the shared adaptation configuration. If a field is missing in the user configuration, the remaining memory budget units are filled in according to the order of model configuration priority, training configuration second, and default range fallback, and the field source, default value, value range, and whether it participates in subsequent budget calculations are written into the shared adaptation access configuration record; if the same field conflicts in multiple sources, such as the model hidden dimension and the adapter target layer dimension being inconsistent, the instantiation of the shared channel is stopped and the conflicting field is output. After model loading and before shared channel instantiation, the total video memory M_vram and the current free video memory M_free0 are read through the graphics processor runtime memory query interface. This memory query interface can be a driver management interface, a runtime memory interface, or a memory query function encapsulated by the training framework. Before reading, the current graphics processor execution flow is synchronized, and the framework-allocated, reserved, and releasable caches are recorded to prevent distortion of free values ​​due to asynchronous operators or cache pools. Subsequently, one or more baseline micro-batches with the same sequence length, micro-batch size, and precision are executed, recording the peak occupancy of the base local path, activation, communication buffer, and temporary workspace. The maximum value among these records is used as the budget deduction item. The remaining video memory M_res is obtained by subtracting the aforementioned peak occupancy and protection margin from the available video memory according to the formula M_res = max(0, min(M_vram, M_free0+ M_releasable) - M_base_path^peak - M_activation^peak - M_workspace^peak - M_comm^peak - M_guard).In this context, M_res represents the residual video memory available for sharing the low-rank adapter channel, in bytes; M_vram represents the total video memory of the graphics processor, in bytes; M_free0 represents the currently free video memory found before the shared channel instantiation, in bytes; M_releasable represents the releasable cache or reclaimable temporary block, in bytes; M_base_path^peak represents the peak usage of the graphics processor's local base path, in bytes; M_activation^peak represents the activation peak usage, in bytes; M_workspace^peak represents the peak usage of the operator temporary workspace, in bytes; M_comm^peak represents the peak usage of the communication buffer, in bytes; and M_guard represents the protection margin, in bytes, which can be max(gamma×M_vram, M_min_guard), where gamma ranges from 0.03 to 0.15. This process does not simply subtract a fixed constant from the total video memory, but incorporates the currently free video memory, releasable cache, runtime peak usage, and protection margin into the constraint. If M_res is less than the minimum shared channel configuration's required video memory, the system will disable the shared channel or enable only the lowest rank configuration, and write the reason in the configuration record.

[0043] like Figure 3 As shown, the graphics processor memory capacity is read through the runtime interface, and the basic path usage, activation usage, workspace usage, and protection margin are dynamically deducted to calculate the residual memory budget that can be used for sharing low-rank channels. Based on this, the target number of channels, shared rank, and intermediate dimension are selected.

[0044] Step 1.2 specifically includes the following steps: The shared low-rank channel configuration unit constructs a candidate configuration set containing the number of shared channels, shared rank, intermediate dimension, parameter precision, and enabled layer set. The total video memory usage of each candidate configuration, including parameters, gradients, optimizer state, and temporary buffers, is calculated, and candidate configurations whose total video memory usage exceeds the remaining video memory budget limit are eliminated. For candidate configurations that meet the limit, a comprehensive configuration score is calculated according to a preset quality-first, throughput-first, or user-specified priority mode, and the target configuration with the highest score is selected for parameter and buffer instantiation allocation to generate a shared low-rank adaptation channel.

[0045] Specifically, the shared low-rank channel configuration unit constructs a candidate configuration set Q, where each specific candidate configuration q in the candidate configuration set Q contains an enabled layer set L_q, a shared channel count N_ch, a shared rank r_s, an intermediate dimension d_mid, parameter precision bytes, an output scaling factor beta_s, an output merging method, and an execution granularity m_s. The candidate configuration set can be directly provided by a configuration table or generated by the Cartesian product of the number of channels, rank, intermediate dimension, enabled layers, and precision. After generation, candidate configurations that do not match the input-output dimensions of the target linear layer are first removed. Then, a specific candidate configuration M_cfg(q) is calculated for each one, including parameters, gradients, optimizer state, forward temporary buffer, backward temporary buffer, and output buffer occupancy. Only candidates satisfying M_cfg(q) <= eta × M_res are retained, where eta is the budget usage ratio, ranging from 0.6 to 0.9. If multiple candidate configurations meet the budget, the highest-scoring candidate is selected according to Score(q) = w_q × Q_est(q) + w_c × C_est(q) - w_t × T_est(q) - w_m × M_cfg(q) / M_res. Here, Q_est represents the quality gain estimated by rank and enabling layer, C_est represents the full token coverage gain, T_est represents the estimated additional execution time, and w_q, w_c, w_t, and w_m are non-negative weights. The quality-first mode increases w_q and w_c, the throughput-first mode increases w_t, and the user-specified priority mode directly selects the first candidate configuration that meets the budget based on candidate ranking. If scores are the same, the candidate configuration with lower memory usage, an enabling layer closer to the expert hybrid layer output merging point, and a shorter expected additional execution time is prioritized.

[0046] Where M_cfg(q)=L_q×N_ch×(M_param+M_grad+M_opt+M_tmp)<=eta×M_res.

[0047] M_param=r_s×(d_in+d_mid+d_out)×bytes.

[0048] Where q represents a candidate shared channel configuration; L_q represents the number of layers with shared channels enabled, and is a positive integer; N_ch represents the number of shared channels, and is a positive integer; M_param represents the parameter usage per layer and per channel; M_grad represents the corresponding gradient usage; M_opt represents the optimizer state usage, which for AdamW can be 2 × M_param or calculated according to precision; M_tmp represents the forward and backward temporary buffer usage; r_s represents the shared low-rank adaptation rank, and is a positive integer; d_in, d_mid, and d_out represent the input dimension, intermediate dimension, and output dimension, respectively, in terms of the number of elements; bytes represents the number of bytes per parameter element; and eta represents the budget usage ratio, ranging from 0.6 to 0.9. If a two-matrix LoRA structure without d_mid is used, then M_param can degenerate into r_s × (d_in + d_out) × bytes. AdamW refers to Adam with Decoupled Weight Decay, an adaptive moment estimation optimizer used to update parameters based on the first moment, second moment, and weight decay.

[0049] Step 1.3 specifically includes allocating the parameters, gradients, optimizer states, and intermediate buffers of the shared low-rank adaptation channel in the residual memory of the graphics processor according to the selected target configuration. The parameters of the shared low-rank adaptation channel are registered as an independent trainable parameter group, ensuring that they do not overwrite the original base model parameters. The shared low-rank adaptation channel is aligned and attached to the output merging point of the expert mixing layer in the expert mixing model, or attached to the bypass position before residual summation within the expert mixing layer.

[0050] Specifically, on the GPU side, shared low-rank adaptation channel parameters, gradients, optimizer states, and intermediate buffers are allocated according to the selected configuration, and the shared low-rank adaptation channel is registered as an independent trainable parameter group. Allocation requirements include: the total number of parameters, gradients, and optimizer states must not exceed eta×M_res; buffers are reserved based on the maximum number of terms in the micro-batch and the hidden dimension. Parameter precision must be consistent with training precision or use convertible precision; the output tensor shape must be consistent with the routing path output; the shared low-rank adaptation channel must not cover the original routing expert state, routing weights, or base model parameters. The shared low-rank adaptation channel is preferably attached at the output merging point of the expert hybrid layer, i.e., before the routing expert output and the original shared expert output of the expert hybrid model form the expert hybrid layer output; in models using residual structures, it can also be attached at a bypass position before residual summation. When attaching, check the layer number, target linear layer name, input dimension, and output dimension. The attachment point is only enabled if the output of the shared low-rank adaptation channel and the output of the routing path can be added within the same lexical order and the same micro-batch range. If the check fails, the shared low-rank adaptation channel of the layer is not instantiated and written to the configuration record.

[0051] Step 2 includes Step 2.1: Forward route statistics and distribution table construction; Step 2.2: Calculation of skewness metric and expert classification identification; Step 2.3: Heterogeneous path load and granular scheduling.

[0052] like Figure 4 As shown, the training signal coverage statistics unit performs real-time statistics on the distribution of routing terms in the current batch and calculates the skewness metric, thereby identifying popular experts and long-tail experts. The skewness-aware scheduling unit divides the routing basic weight task into weight blocks or matrix tiles, divides the expert low-rank adaptation task into term segments, and splits or merges them for submission. At the same time, it controls the shared low-rank channel to execute independently in micro-batches.

[0053] Step 2.1 specifically includes feeding the input tokens of the current batch into the expert hybrid model, calculating the routing score of each token for each expert, and obtaining the routing selection result according to a preset Top-k strategy. The training signal coverage statistics unit, based on the routing selection result, counts the number of times each expert is accessed by the routing in the current batch of input tokens, and generates a token distribution table recording the corresponding token indices. The training signal coverage statistics unit marks experts whose received token access counts are greater than zero as active experts.

[0054] Specifically, in each training step, the hidden state H is flattened into X; H is the original multidimensional feature tensor with sequence features, representing the original multidimensional feature tensor with sequence features received from the preceding network layer, such as a multi-head attention layer or a feedforward network layer, i.e., the hidden state; X is the dimensionality-reduced feature matrix of the decoupled sequence boundary, representing the dimensionality-reduced feature matrix of the decoupled sequence boundary generated after performing the flattening operation on the feature tensor H. The flattening operation physically reduces and aggregates the tensor H on the batch dimension B and the sequence dimension L, thereby reconstructing its shape into a two-dimensional matrix. If the shape of H is [B, L, d_model], then the shape of X is [N_token, d_model], where N_token = B × L represents the total number of tokens in the current batch, and d_model is the hidden dimension. The router calculates logits=Router(X) based on the original model's routing weights. The shape of logits is [N_token, N_expert], where N_expert is the number of experts in this expert hybrid layer. Then, it executes softmax, sigmoid, or the normalization method specified by the model according to the original model's routing rules to obtain the expert score P for each term. logits is an unnormalized routing score tensor, representing the original unnormalized routing score matrix or gated feedforward value; Router(·) is the routing mapping function / router, representing the parameterized routing mapping operator or gated subnetwork; N_token is the total number of global terms, representing the total number of global scalars of all input terms or entity units participating in forward propagation within the current computation cycle, numerically satisfying the product relationship between the current batch size and the length of a single sample sequence; N_expert is the total number of experts, representing the static total number of heterogeneous expert subnetworks or feedforward neural network branches configured in parallel in the current expert hybrid layer, determining the upper dimension of the routing output space and defining the potential maximum number of branches for parallel model computation; softmax / sigmoid... `id` is the normalization activation operator, representing the nonlinear probability mapping activation function. The `softmax` operator is used to perform exponential normalization on the expert dimension, forcing the scores of a single token to all experts to be transformed into mutually exclusive probability distributions, i.e., the sum of the scores of each channel is 1. The `sigmoid` operator is used to perform independent gated confidence mapping, compressing each score independently to the (0,1) interval. `P` represents the generated normalized expert activation probability matrix or routing score tensor, whose shape corresponds homomorphically to `logits` as `[N_{token},N_{expert}]`. The scalar element `P[i,j]` in the matrix is ​​strictly limited to the probability interval `[0,1]`, and is used to quantify the forward dependency or contribution weight of the `i`th token to the `j`th expert subnetwork.The top-k expert IDs are selected to form E[i,j], and their corresponding scores are normalized into routing weights W[i,j]. Here, E is the routing index matrix, generated by the routing module, recording the discrete mapping relationship between all input entities in the current batch and the activated heterogeneous expert sub-networks; W is the gated weight coefficient matrix, homomorphically corresponding to the index matrix E, recording the continuous contribution measure of the activated expert sub-network to the final output features of a specific input entity. i represents the lexical ID, j represents the j-th selected expert, and E has the shape [N_token,k], where N_token represents the total number of input entities after flattening and aggregation, and k represents the preset upper limit for the number of expert nodes activated in a single routing decision. Matrix element E[i,j] stores the physical ID or logical identifier of the j-th optimal expert sub-network allocated to the i-th lexical ID in the graphics processor's memory. Matrix E directly serves as the data distribution addressing basis for the subsequent expert aggregation calculation module. The shape of W is [N_token,k]. W[i,j] and E[i,j] form a strict key-value pair correspondence. W[i,j] represents the scalar scaling factor mapped when the i-th token is processed by the selected expert numbered E[i,j] during the forward merging operation. The value of W[i,j] ranges from 0 to 1. The sum of the top-k weights of the same token is 1 or conforms to the routing rules of the original model. If the original model uses non-normalized gating, the original gating value is saved, and the normalized statistical weights are also saved.

[0055] Initialize a counting array n of length N_expert, and execute n[E[i,j]]=n[E[i,j]]+1 for each term i and each selected expert E[i,j]; alternatively, maintain a weighted count n_w[E[i,j]]=n_w[E[i,j]]+W[i,j]. Here, n_e represents the number of times expert e is accessed by the top-k routes within the current batch. If a term is routed to two experts, the count is performed separately for each expert. The active expert set A is defined as the set of experts where n_e>0. The term distribution table records n_e, n_w, original batch index, sequence position index, micro-batch number, term index list, and route weight list for each expert. The term distribution table is used to estimate the low-rank adaptation load of routing experts, compare the number of terms covered by the shared channel N_token with the actual number of terms n_e obtained by a single expert parameter, and provide an index basis for subsequent recovery of the term order after expert aggregation calculation.

[0056] The step 2.2 specifically includes: the training signal coverage statistics unit acquires the maximum value of the number of token accesses received by a single active expert, and calculates the average value and standard deviation of the number of token accesses of all active experts. The maximum value is divided by the average value to calculate the routing skew metric. A hot expert threshold and a long-tail expert threshold are respectively calculated based on the average value and the standard deviation; an expert whose access count is greater than or equal to the hot expert threshold is marked as a hot expert, and an expert whose access count is less than or equal to the long-tail expert threshold is marked as a long-tail expert.

[0057] Specifically, the skew metric S is calculated according to the formula S=max_e(n_e) / mean_e(n_e). Wherein, n_e represents the number of token accesses received by expert e; max_e(n_e) represents the maximum value of the number of token accesses received by a single expert; mean_e(n_e) represents the average number of token accesses of active experts, preferably, the average is calculated only for active experts with n_e>0. The value of S is greater than or equal to 1. A larger S indicates a more skewed routing distribution, and the skew-aware scheduling unit can adopt a finer task segmentation granularity for hot experts. During calculation, the average is calculated only for active experts with n_e>0, so as to prevent inactive experts from lowering the average value; if there are no active experts in the current layer, this layer skips skew scheduling and records an exception. Meanwhile, the average token count mean_A and the standard deviation sigma_A of the active experts are calculated. The hot expert threshold tau_hot can take max(mean_A+lambda_hot×sigma_A, rho_hot×N_token×top_k / N_expert), and the long-tail expert threshold tau_tail can take max(1, rho_tail×mean_A), wherein lambda_hot ranges from 0.5 to 2, rho_hot ranges from 1 to 3, and rho_tail ranges from 0.2 to 0.8. If n_e>=tau_hot, expert e is marked as a hot expert; if 0<n_e<=tau_tail, expert e is marked as a long-tail expert; if n_e=0, expert e is marked as an inactive expert; other active experts are marked as ordinary experts. The above classification result is written into a token distribution table, and is used as the input for task segmentation and queue priority in step 2.3.

[0058] Step 2.3 specifically includes the following steps: For the routing adaptation path, the skew-aware scheduling unit splits the routing adaptation tasks of the popular experts into multiple subtasks according to preset lexical segments or matrix blocks and submits them to the graphics processor for execution in parallel; and merges the routing adaptation tasks of multiple long-tail experts in the video memory and submits them as a single kernel task. For the shared low-rank adaptation channel, the skew-aware scheduling unit dynamically calculates the number of micro-batch lexicals to be executed in a single submission of the shared low-rank adaptation channel, i.e., the execution granularity m_s, based on the total number of lexicals in the current batch, the preset shared channel temporary buffer capacity, the remaining video memory budget, and the calculated routing skew metric. The unit then segments all lexicals according to this granularity and sends them sequentially into the shared low-rank adaptation channel.

[0059] Specifically, the skew-aware scheduling unit estimates the computational cost of low-rank adaptation for each active expert e as Work_lora(e) = 2 × n_e × r_e × (d_in + d_out), where r_e is the expert's low-rank adaptation rank, d_in and d_out are the input and output dimensions of the target linear layer, respectively, and n_e represents the number of lexical coverages of the routing adaptation path for expert e. If the basic weights of the routing experts are also calculated by the central processing unit (CPU) path, the skew-aware scheduling unit separately estimates the computational cost of the basic weights as Work_base(e) = 2 × n_e × d_in × d_out. The skew-aware scheduling unit writes Work_lora(e), Work_base(e), the number of expert weight reads, the lexical index list, the task priority, and the dependent events into the adaptation task descriptor. For popular experts, the task descriptor is split into multiple tasks by word segments or matrix tiles. Segmenting by word segments involves one-dimensional slicing along the input sequence dimension, while matrix tiles involve two-dimensional segmentation along both the sequence and feature dimensions, forming multiple fixed-size two-dimensional computational tiles / matrix tiles. For long-tail experts, multiple smaller tasks can be merged and submitted to reduce thread scheduling overhead. The queue submission order can be executed according to the following rules: shared channel micro-batch tasks are submitted to the graphics processor first; popular expert split tasks are submitted in descending order of load; and long-tail expert merged tasks are submitted in batches after dependencies at the same level are satisfied. When the preceding slices for a popular expert are not yet completed, the scheduler can submit other expert or shared channel tasks first to reduce waiting time.

[0060] The thresholds for popular experts and long-tail experts can be expressed as: tau_hot=max(mean_A+lambda_hot×sigma_A,rho_hot×N_token×top_k / N_expert).

[0061] tau_tail=max(1,rho_tail×mean_A).

[0062] Where mean_A and sigma_A represent the average and standard deviation of the number of visits to active expert terms, respectively; lambda_hot represents the popularity threshold amplification factor, with a value range of 0.5 to 2; rho_hot represents the global average visit amplification factor, with a value range of 1 to 3; and rho_tail represents the long tail threshold factor, with a value range of 0.2 to 0.8.

[0063] Execution granularity refers to the range of terms and task size submitted to the shared low-rank channel for execution at one time, including three types: batch level, micro-batch level, and term segment level. Execution granularity only determines how many terms the shared channel processes each time, without changing the coverage relationship of the shared channel executing all terms in the current batch. The skew-aware scheduling unit selects m_s based on N_token, M_tmp, M_res, S, and the graphics processor execution flow state, where M_tmp is the temporary buffer capacity of the shared channel, representing the threshold of the temporary buffer capacity allocated to the shared low-rank adaptation channel or the intermediate state memory limit. An implementable rule is m_s=clamp(m_min,m_max,ceil(N_token / ceil(S))), where m_min is the minimum number of terms in the micro-batch, m_max is the maximum number of terms not exceeding the temporary buffer capacity of the shared channel, and clamp indicates that the third parameter is limited to the range [m_min,m_max]. If N_token=100, S=2.8, m_min=16, and m_max=64, then ceil(S)=3, ceil(N_token / ceil(S))=34. The skew-aware scheduling unit can divide the shared channel task into three micro-batch segments of 34, 34, and 32. If the expected waiting time for the routing path is long, the skew-aware scheduling unit can use a larger execution granularity m_s to reduce the submission overhead; if the temporary memory buffer is tight or the routing skew is high, the skew-aware scheduling unit can use a smaller execution granularity m_s to allow the shared channel output to enter the merging queue earlier.

[0064] The router output can be represented as: P_i= normalize(Router(x_i)).

[0065] E_i = topk(P_i, k).

[0066] W_i = renorm(P_i[E_i]).

[0067] Where x_i represents the hidden state of the i-th word; Router represents the original model router; P_i represents the expert score vector; normalize represents the softmax, sigmoid, or other normalization rule used by the original model; E_i represents the k expert IDs selected for the i-th word; and W_i represents the routing weights corresponding to the selected experts. Each element in W_i takes a value from 0 to 1 and is normalized according to the routing rules of the original model.

[0068] Step three specifically includes: The shared low-rank adaptation execution unit receives the flattened two-dimensional input feature matrix X and sends the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel in the graphics processor's video memory, forming two computational branch entry points at the bottom layer that are physically address-independent, control flow-separated, but data-from-the-source. The shared low-rank adaptation execution unit distributes the corresponding tokens to the corresponding routing expert nodes in the video memory based on the routing selection index matrix E and the gate weight coefficient matrix W generated in the preceding steps. Each activated expert node computes its received tokens in parallel through the independent stream of the graphics processor, CUDA Stream, outputs a local feature matrix, and multiplies it by the corresponding gating weight W[i,j]. The shared low-rank adaptation channel completely shields the addressing signal of the routing matrix E, and forces the reception of all N_{token} input tokens; Based on the execution granularity m_s calculated in the previous steps, the full vocabulary is divided into multiple micro-batches, which are then sequentially fed into the dimension reduction matrix A and dimension increase matrix B of the shared channel for linear projection calculation, generating a shared low-rank output matrix Y_s with the same dimension as the input features.

[0069] Specifically, for the micro-batch input X_s, the shared low-rank adaptation execution unit generates the shared adaptation output according to the low-rank adaptation formula. The shape of X_s is [m_s, d_in], and its row order follows the original batch word order recorded in step 2.1. A preferred implementation is U = X_s × A_s, Y_s = beta_s × U × B_s, where the shape of A_s is [d_in, r_s], the shape of U is [m_s, r_s], the shape of B_s is [r_s, d_out], the shape of Y_s is [m_s, d_out], and beta_s is the scaling factor. A_s and B_s are the trainable parameters of the shared channel registered in step 1.3. Initialization can use a small random value for A_s, zero initialization for B_s, or an initialization method consistent with the LoRA framework being accessed. beta_s can be alpha_s / r_s or a value given in the configuration table. Another implementation involves first passing through a shared low-rank bottleneck in the intermediate dimension d_mid, i.e., U1 = X_s × A1, U2 = phi(U1 × A2), Y_s = beta_s × U2 × B, where phi is the activation function or identity function. The shared channel is executed for each term in X_s, without querying E[i,j] or relying on expert popularity.

[0070] Before fusing the output of the route adaptation path with the shared low-rank output in the forward phase, an alignment step is also included: If the word order of the route adaptation path is disrupted due to expert aggregation calculation, the route path output will be restored to its original batch order based on the word index recorded when statistically analyzing the route distribution. This ensures that the route path output has the same word order and micro-batch range as the shared low-rank output. like Figure 5 As shown, in the forward phase, the output of the routing path and the output of the shared low-rank channel are merged after word order alignment to form the layer output; in the backward phase, the loss gradient is split from the summing node into the routing adaptation path and the shared low-rank path respectively, so as to perform heterogeneous updates on the two sets of independent parameters.

[0071] Step four specifically includes: The forward fusion unit monitors the execution status of the route adaptation path. If the physical order of the terms in the memory is disrupted due to the expert aggregation calculation, the forward fusion unit extracts the original term indexes cached during the preceding statistics. Using the original token index, the merged output of the routing path is restored to the initial [N_{token},d_{model}] sequence order by executing the discrete aggregation Scatter or index restoration operator, denoted as Y_e.

[0072] The forward fusion unit reads the preset shared scaling factor beta_s from the shared adaptation configuration and multiplies it into the shared low-rank output matrix Y_s.

[0073] In the graphics processor, the tensor summation operator is called to perform element-wise addition of the aligned route output Y_e and the scaled shared output, i.e., Y_{out}=Y_e+beta_s×Y_s.

[0074] The summed two-dimensional tensor Y_{out} is reshaped along the batch and sequence dimensions to restore it to the three-dimensional tensor shape [B,L,d_{model}] consistent with the input of this layer, thus completing the forward pass of the current expert hybrid layer.

[0075] The forward fusion unit waits for the completion events of the routing path output Y_r and the shared channel output Y_s, and reads the tensor shape field, lexical order field, batch range field, micro-batch number field, scaling factor field, fusion timing status, and fusion method from the forward fusion descriptor. If the routing path changes the lexical order due to expert aggregation calculation, it first restores the original batch order according to the lexical index recorded in step two, and then performs fusion after confirming that the two have the same lexical order, the same micro-batch range, and the same output dimension. Direct summation is Y = Y_r + Y_s; scaled summation is Y = Y_r + beta_s × Y_s; gated summation is g = sigmoid(X_s × W_g + b_g), Y = Y_r + g × Y_s, where the shape of g can be [m_s, 1] and broadcast to the output dimension, or [m_s, d_out]; residual summation is Y = X_s + Y_r + beta_s × Y_s, or Y_r + beta_s × Y_s can be used as the output of the expert mixing sublayer and fed to the outer residual layer within the existing residual framework of the model. Alternative methods include summation after normalization, adjusting beta_s according to the training stage, dynamically adjusting the shared channel weights according to the word, or linear projection after concatenation.

[0076] Forward fusion can be represented as: Y=F_merge(Y_r,Y_s,X,beta_s,g).

[0077] Where Y_r represents the output of the routing expert and the expert low-rank adapted path; Y_s represents the output of the shared low-rank channel; X represents the input hidden state; beta_s represents the shared channel scaling factor; and g represents the gating weight. F_merge can be Y_r+Y_s, Y_r+beta_s×Y_s, Y_r+g×Y_s, X+Y_r+beta_s×Y_s, or a normalized sum.

[0078] Step five specifically includes: The back gradient branch unit receives the layer global gradient tensor G_{out}, which is backpropagated from the subsequent network layers to the output of the current expert hybrid layer. Based on the addition nodes of the computation graph formed in the forward computation phase, the backward gradient branch unit clones the gradient tensor G_{out} into two identical gradient branches without loss. For the route adaptation path: the first gradient branch is back-scattered according to the gate weight W and index E during the forward routing, and routed to the active expert nodes that participated in the forward calculation, and the local gradient of the route expert parameters is calculated. For shared low-rank channels: the second gradient branch is directly used as the gradient G_s of the shared low-rank output. According to the chain rule, the gradient matrices Delta A and Delta B of the dimension reduction matrix A and the dimension increase matrix B are independently calculated using X and intermediate activation values ​​cached in the forward stage.

[0079] Invoke the optimizer in the graphics processor, such as AdamW, to perform heterogeneous parameter weight updates for the routing expert low-rank parameters and shared low-rank parameters, which are marked as independent parameter groups.

[0080] Refresh the internal state variables of the shared low-rank adaptation channel and release the corresponding temporary reverse computation working area to prepare for the next training iteration.

[0081] The fusion in the forward phase can be performed using any one of the following methods: direct summation, scaling summation, gated summation, or residual summation.

[0082] The parameter update during the reverse phase specifically includes: The routing expert low-rank adaptation parameter group and the shared low-rank adaptation channel parameter group are respectively identified as independent parameter groups; For routing expert low-rank parameters that have not received any tokens in the current training step, skip the corresponding optimizer state update; After completing the gradient update of the parameters of the shared low-rank adaptation channel, refresh the parameter version, scaling factor and cache status of the shared low-rank adaptation channel.

[0083] Specifically, during backpropagation, the backpropagation gradient branch unit receives the current layer's output gradient G = dL / dY. For direct summation, the routed path receives G_r = G, and the shared path receives G_s = G; for scaled summation, the shared path receives G_s = beta_s × G; for gated summation, the shared path receives G_s = g × G, and simultaneously calculates the gradient of the gate parameters. Taking Y_s = beta_s × (X_s × A_s) × B_s as an example, the backpropagation gradient branch unit calculates dL / dB_s = (X_s × A_s)^T × G_s, dL / dA_s = X_s^T × (G_s × B_s^T), and calculates the gradient of the returned input G_x_shared = G_s × B_s^T × A_s^T. The route adaptation path only calculates the gradient of the corresponding expert's low-rank parameters for the lexical slices routed to expert e: the backward gradient branch unit retrieves X_e and the corresponding output gradient G_e from the lexical distribution table. If route weights are used in the fusion, G_e is first multiplied by the corresponding W[i,j]. For the low-rank matrices A_e and B_e of expert e, they can be calculated as U_e=X_e×A_e, dL / dB_e=U_e^T×G_e, and dL / dA_e=X_e^T×(G_e×B_e^T). Unselected experts do not generate gradients for this path in this step, and the backward gradient branch unit only retains its original gradient buffer state or marks it as having no update in this step.

[0084] The lexical coverage of shared paths and routing paths can be represented as: C_shared=N_token.

[0085] C_routed_param(e)=n_e.

[0086] Where C_shared represents the number of tokens covered by the shared low-rank channel; N_token represents the total number of tokens in the current batch; and C_routed_param(e) represents the number of token visits in the current batch where the routing low-rank parameter of expert e obtains the training signal. The shared low-rank channel produces an adaptation output for each token, while the low-rank parameter of a single routing expert only obtains its gradient from the n_e token visits routed to that expert.

[0087] After gradient accumulation, the backpropagation gradient branch unit performs gradient merging, gradient pruning, weight decay, and optimizer update on the expert low-rank adaptation parameter set and the shared low-rank channel parameter set, respectively. In distributed training, the gradients of the parameters involved in this update can be synchronized or reduced before this step. When using AdamW, the parameter theta is updated based on the first moment m, the second moment v, the learning rate lr, the weight decay lambda, and the stability term epsilon as theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon) - lr × lambda × theta. The backpropagation gradient branch unit does not directly and unconditionally overwrite all parameters after backpropagation. Instead, it first writes the gradients to the corresponding buffers according to the parameter identifiers, and then the optimizer updates the parameters with valid gradients according to the parameter set. The routing expert low-rank parameters are determined by their expert number, layer number, projection matrix, and adapter name; the shared low-rank channel parameters are determined by the shared channel number, layer number, and matrix name. For routing expert low-rank parameters that do not receive any terms, the backpropagation gradient branch unit skips its optimizer state update or simply maintains the state according to the selected optimizer rule; shared low-rank channel parameters, because they cover all terms, are typically updated in each valid training batch. After the parameters are updated, the backpropagation gradient branch unit refreshes the shared channel parameter version, scaling factor, backend visibility pointer, and cache state, and uses the latest parameters in the next forward-looking iteration.

[0088] Repeat the above process for each training batch and each gradient accumulation step. If the peak memory usage exceeds the protection margin during runtime, reduce the shared channel rank, reduce the number of enabled layers, or disable some shared channels. If the routing skew exceeds the threshold for multiple consecutive steps, reduce the granularity of popular expert tasks and prioritize submitting long-tail expert merging tasks. If the shared channel output or gradients show numerical anomalies, revert to the previous stable parameter version and record the abnormal batch in the shared channel execution status table.

[0089] The shared low-rank channel back gradient can be expressed as: dL / dB_s=U^T×G_s.

[0090] dL / dA_s=X_s^T×(G_s×B_s^T).

[0091] Where U = X_s × A_s, X_s represents the current shared channel micro-batch input, G_s represents the output gradient received by the shared path, and A_s and B_s represent the low-rank matrices of the shared low-rank channels, respectively. If a structure with intermediate dimensions is adopted, the gradients of A1, A2, B, and the activation function phi are calculated using the chain rule.

[0092] The base model for the expert hybrid model described here is the Qwen3.5-397B-A17B expert hybrid model. The domain data is Llemr's professional domain data. The primary rank r=8 for the ordinary low-rank adaptation, the maximum training sequence length is 512, the optimizer is AdamW, the learning rate is 0.0001, and bf16 is used for training accuracy. The final train_loss for the ordinary low-rank adaptation is 0.901014, for the shared expert high-rank configuration it is 0.822729, and for the shared low-rank channel configuration it is 0.763047. Looking at the average loss over the last ten steps, the ordinary low-rank adaptation loss is 0.559450, the shared expert high-rank configuration loss is 0.532999, and the shared low-rank channel configuration loss is 0.506407. The shared low-rank channel configuration does not change the primary LoRA parameter configuration compared to the ordinary low-rank adaptation; instead, it enables kt_use_lora_experts, sets the number of shared channels to 1, and the intermediate dimension to 256.

[0093] In one embodiment, a training batch is performed on an expert hybrid layer with 8 routing experts. This batch, after flattening, contains N_token = 100 tokens, and the top-k routing count is 2. After the router outputs the expert IDs, the number of visits for each expert is calculated as n = [70, 45, 30, 20, 15, 10, 7, 3], with a total of 200 visits. The average number of visits for active experts is mean_A = 25, the standard deviation is sigma_A = 21.24, and the maximum number of visits is 70. Therefore, the skewness metric S = 70 / 25 = 2.8. If lambda_hot = 1, rho_hot = 1.5, and rho_tail = 0.5, then tau_hot = max(25 + 21.24, 37.5) = 46.24, and tau_tail = 12.5. Expert 0 is marked as a popular expert, and experts 5, 6, and 7 are marked as long-tail experts. The routing adaptation task for expert 0 is split into multiple tasks based on term segments. The tasks for experts 5, 6, and 7 are merged and submitted, while maintaining the shared low-rank channel to execute all 100 terms. The total GPU memory (M_vram) is 24GB, the peak usage for model loading and basic local paths is 16GB, the peak usage for activation is 3GB, the peak usage for the working area and communication buffer is 2GB, and the protection margin is 1GB, resulting in M_res of 2GB. Assuming eta = 0.8, the budget limit for the shared channel candidate configuration is 1.6GB. Candidate configuration q1 has 1 shared channel, rank 128, intermediate dimension 256, and 20 layers enabled; candidate configuration q2 has 1 shared channel, rank 256, intermediate dimension 256, and 20 layers enabled; candidate configuration q3 has 2 shared channels, rank 256, intermediate dimension 512, and 20 layers enabled. According to the formula M_cfg(q)=L_q×N_ch×(M_param+M_grad+M_opt+M_tmp)<=eta×M_res; M_param=r_s×(d_in+d_mid+d_out)×bytes, the parameters, gradients, optimizer state, and temporary buffer usage are estimated. Candidates exceeding 1.6GB are eliminated, and the final configuration is selected from the remaining candidates based on quality priority or throughput priority. In the forward phase, the current micro-batch input X_s is simultaneously fed into the routing path and the shared low-rank channel. The routing path calculates the routing expert output Y_r based on E_i and W_i; the shared low-rank channel calculates U=X_s×A_s and Y_s=beta_s×U×B_s. After waiting for the completion events of Y_r and Y_s, Y=Y_r+beta_s×Y_s is executed, and Y is handed over to subsequent layers or outer residual connections. During the reverse phase, the output gradient G is fed into both the routing path and the shared path. The routing path only updates the low-rank parameters of the selected expert, while the shared path calculates the gradients of A_s and B_s for all terms. After the optimizer step is completed, the shared channel version is refreshed so that the next training step reads the updated shared low-rank parameters.

[0094] Regarding the shared low-rank channel format, this invention preferably employs a shared, routing-independent low-rank adaptation channel. Alternatively, multiple shared low-rank channels, layer-based shared channels, expert group-based shared channels, dense adapters, gated shared adapters, low-rank expert groups, trainable scaling vectors, or other lightweight trainable structures can be used. As long as this structure resides in the residual GPU memory and produces adapted outputs for all or most tokens, a similar purpose can be achieved.

[0095] Regarding the residual video memory budget, this invention preferably explicitly deducts the base path, activation, working area, and protection margin. Alternatively, the available residual video memory can be determined based on runtime video memory monitoring, historical peak values, static configuration, tiered budgeting, dynamic adjustment by tier, or an externally specified threshold.

[0096] Regarding channel size, this invention preferably selects the number of channels, rank, and intermediate dimension based on the remaining GPU memory. Alternatively, the channel size can be dynamically adjusted based on training loss, gradient norm, routing entropy, expert popularity, task type, target throughput, or target quality.

[0097] Regarding skewness measures, this invention preferably uses the ratio of the maximum number of expert lexical units to the average number of expert lexical units as the skewness measure. Variance, entropy, Gini coefficient, top-k expert percentage, historical moving average, prediction execution time, or hardware counter feedback can also be used as skewness measures.

[0098] Regarding task segmentation, this invention preferably segments based on weight blocks using basic weights, segments based on word segments using expert low-rank adaptation, and executes shared low-rank channels using micro-batch execution. Alternatively, it can employ output channel segmentation, input channel segmentation, matrix tile segmentation, expert group segmentation, hierarchical queue segmentation, or multi-level scheduling methods.

[0099] Regarding forward merging, this invention preferably adds or scales the output of the routing expert and the output of the shared low-rank channel. Alternatively, it can employ gated merging, residual merging, merging after layer normalization, adjusting the scaling factor according to the training stage, or dynamically adjusting the weight of the shared channel according to the word.

[0100] Regarding the reverse engineering and optimizer, the present invention preferably assigns the gradients to the routing adaptation path and the shared low-rank path respectively. Alternatively, a unified gradient buffer, a group optimizer, asynchronous gradient merging, delayed updates, mixed precision updates, or low-rank parameter piecewise updates can be used.

[0101] Regarding the collaborative environment, this invention is preferably used in a resource-layered fine-tuning environment where the routing expert is executed by the central processing unit and the shared path is executed by the graphics processing unit. Alternatively, a dedicated accelerator card, integrated graphics card, neural network processor, or other computing device with access to large-capacity main memory can be used to execute the routing expert path, while the graphics processing unit continues to carry the shared low-rank channel.

[0102] In the context of low-rank fine-tuning of expert hybrid large models, any deployment of route-independent shared adaptation channels based on residual GPU memory, combined with route skew statistics, adaptation task splitting, forward merging, and reverse gradient splitting to improve heterogeneous fine-tuning efficiency and training signal coverage, can be considered as an equivalent implementation or variation of the ideas in this invention.

[0103] A low-rank adaptation training method and system based on residual memory sharing in an expert hybrid model improves the stability of residual memory utilization. The residual memory is used for shared, route-independent low-rank adaptation channels, rather than for caching a small number of popular routing experts. The mechanism is that the shared channels execute on all terms, and the training gain is determined by full batch term coverage and shared parameter updates, not directly dependent on the cache hit rate of a single popular expert. Therefore, even when the routing popularity of domain data changes, the shared channels can still obtain stable training signals. Compared to popular expert caching schemes, this reduces the fluctuation in gains caused by popularity migration. Improve low-rank adaptation training signal coverage: A shared low-rank channel is executed for all tokens in each batch, and the training signal coverage of shared parameters and individual routing expert parameters is distinguished by C_shared=N_token and C_routed_param(e)=n_e. The mechanism is that the shared channel parameters receive the gradient of all tokens at each step, while the low-rank parameters of the routing experts only receive the gradient of the routed tokens. Therefore, the updates of the parameters in the remaining GPU memory are more intensive. Compared to the scheme of only adding low-rank adaptation parameters to the routing experts, this is more suitable for fine-tuning with a small amount of domain data. Alleviate execution wait due to routing skew: The task is split according to S=max_e(n_e) / mean_e(n_e), the popularity threshold, and the long-tail threshold. The mechanism is that the popular experts are split into multiple parallelizable token segments or matrix tiles, the long-tail experts are merged and submitted to reduce scheduling overhead, and the shared channel is executed independently in micro-batches and can fill the GPU waiting window. Therefore, the synchronization wait caused by routing skew is weakened. Compared to the scheme of coarse-grained serial execution by expert, this improves the parallelism of heterogeneous paths. Distinguishing between global parameter quantity and effective adaptation per word: Constrained by the residual memory budget, a portion of the trainable capacity is allocated to the shared channel for all words. The mechanism is that increasing the global parameter quantity does not necessarily improve the actual adaptation parameters activated per word, while the shared channel generates adaptation outputs for each word. Therefore, under similar parameter budgets, the system can increase the effective adaptation capacity per word. Compared to simply increasing the low-rank adaptation rank of the routing expert, this reduces the risk of insufficient training of newly added parameters due to route sparsity. Clear boundaries between forward fusion and backward gradient branches: Explicitly recording word order, micro-batch range, completion event, merging method, and parameter group number. The mechanism is that fusion only occurs in the forward phase when both the routing path output and the shared channel output are ready and have the same shape. In the backward phase, the routing path gradient and the shared path gradient are calculated separately according to the fusion formula. Therefore, the parameter update boundaries for the two paths are clear. Compared to schemes that only describe the adaptation calculation of a single path, this is easier to implement and troubleshoot gradient errors.It features runtime memory budget constraints: M_res is calculated using total memory, currently free memory, releasable cache, peak base path memory, peak activation memory, peak working area memory, peak communication buffer memory, and protection margin. The mechanism involves shared channel configurations being first filtered by the budget and then selected based on the target score. Furthermore, when the peak memory exceeds the protection margin, the rank can be reduced, the number of enabled layers decreased, or some channels can be disabled. Therefore, shared channels are less likely to encroach on the basic training path and temporary working area. Compared to fixed shared module configurations, this reduces the risk of training interruptions and memory overflow. It is compatible with existing low-rank adaptation and optimizer mechanisms: expert low-rank adaptation parameter groups and shared low-rank channel parameter groups are identified separately and updated through a unified optimizer interface or a group optimizer interface. The mechanism involves routing expert parameters determined by layer number, expert number, projection matrix, and adapter name, while shared channel parameters are determined by layer number, shared channel number, and matrix name. This allows existing training processes to continue performing gradient accumulation, gradient pruning, AdamW updates, and parameter version refreshes. Compared to solutions requiring a complete rewrite of the trainer, the integration cost is lower. Figure 6 As shown, under the same vertical domain fine-tuning task, the convergence differences in training loss are observed between the ordinary low-rank adaptation scheme, the shared expert high-rank configuration scheme, and the shared low-rank channel configuration scheme described in this invention. The experimental data in the figure demonstrates that, with similar computational and memory resource budgets, the shared low-rank channel scheme described in this invention can significantly accelerate training convergence and reduce the final loss.

[0104] The above description illustrates preferred embodiments of the present invention and helps those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative and should not be construed as limiting the specific implementation of the present invention to these embodiments. For those skilled in the art, several simple deductions and modifications can be made without departing from the inventive concept, and all such modifications should be considered within the protection scope of the present invention.

Claims

1. A low-rank adaptation training method based on residual memory sharing in an expert hybrid model, characterized in that, Includes the following steps: Step 1: Calculate the residual memory budget based on the runtime memory overhead of the graphics processor, and instantiate a shared low-rank adapter channel in the graphics processor based on the residual memory budget. Step 2: Statistically analyze the routing distribution of the current batch of input tokens in the expert hybrid model to obtain a routing skew metric, and schedule the tasks of the routing adaptation path and the execution granularity of the shared low-rank adaptation channel according to the routing skew metric. Step 3: Send the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel respectively, and perform calculations in the routing adaptation path according to the scheduled tasks to generate the routing path output, and perform segmentation calculations on all input tokens in the current batch in the shared low-rank adaptation channel according to the scheduled execution granularity, independent of the routing selection, to generate the shared low-rank output; Step 4: In the forward phase, the output of the routing adaptation path is merged with the shared low-rank output to form a layer output; Step 5: In the reverse phase, the gradients output by the layer are split into the routing adaptation path and the shared low-rank adaptation channel, respectively, for parameter updates. The instantiation of the shared low-rank adapter channel in the graphics processor based on the residual video memory budget specifically includes: constructing a candidate configuration set containing the number of shared channels, shared rank, intermediate dimension, and parameter precision; calculating the total video memory usage of each candidate configuration and eliminating candidate configurations whose total video memory usage exceeds the limit ratio of the residual video memory budget; for candidate configurations that meet the limit ratio, calculating a comprehensive configuration score according to a preset quality priority, throughput priority, or user-specified priority mode, and selecting the target configuration with the highest score for parameter and buffer instantiation allocation to generate a shared low-rank adapter channel; The execution granularity of the shared low-rank adaptation channel is scheduled, specifically including: calculating the number of micro-batch tokens to be executed in a single submission of the shared low-rank adaptation channel based on the total number of tokens in the current batch, the preset shared channel temporary buffer capacity, the residual memory budget, and the routing skew metric.

2. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 1, characterized in that, The calculation of the residual video memory budget based on the runtime video memory overhead of the graphics processor specifically includes: After the expert hybrid model is loaded and before the shared low-rank adapter channel is instantiated, the execution flow of the current graphics processor is synchronized, and the current free video memory and releasable cache are read through the runtime video memory query interface; Perform benchmark micro-batch training to test and record the base local path peak, activation peak, operator temporary workspace peak, and communication buffer peak, respectively; Calculate the sum of the current free video memory and the free cache, and subtract the peak usage and the preset video memory protection margin from the sum of the current free video memory and the free cache to calculate the available residual video memory budget.

3. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 1, characterized in that, The shared low-rank adaptation channel is attached to the output merging point of the expert mixing layer in the expert mixing model, or to the bypass position before the residual summation in the expert mixing layer.

4. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 1, characterized in that, The method of statistically analyzing the routing distribution of the current batch of input terms in the expert hybrid model to obtain a routing skew metric includes: The routing skew metric is calculated by counting the number of times each expert is accessed in the current batch of input terms, identifying experts with a greater than zero number of accesses as active experts, and dividing the maximum number of accesses for a single expert by the average number of accesses for all active experts.

5. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 4, characterized in that, The scheduling of the route adaptation path task specifically includes: The thresholds for popular experts and long-tail experts are calculated based on the average and standard deviation of the number of visits to active expert terms. Experts whose access count is greater than or equal to the popular expert threshold are marked as popular experts, and experts whose access count is less than or equal to the long-tail expert threshold are marked as long-tail experts. The routing adaptation tasks of the popular experts are split into multiple subtasks according to word segments or matrix blocks and submitted in parallel, and the routing adaptation tasks of multiple long-tail experts are merged and submitted.

6. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 1, characterized in that, Before fusing the output of the route adaptation path with the shared low-rank output in the forward phase, an alignment step is also included: If the word order of the route adaptation path is disrupted due to expert aggregation calculation, the route path output is restored to the original batch order based on the word index recorded when the route distribution is statistically analyzed, so as to ensure that the route path output has the same word order and micro-batch range as the shared low-rank output.

7. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 1, characterized in that, The fusion in the forward phase can be performed using any one of the following methods: direct summation, scaling summation, gated summation, or residual summation.

8. The low-rank adaptation training method based on residual memory sharing of an expert hybrid model according to claim 1, characterized in that, The parameter update during the reverse phase specifically includes: The routing expert low-rank adaptation parameter group and the shared low-rank adaptation channel parameter group are respectively identified as independent parameter groups; For routing expert low-rank parameters in the routing expert low-rank adaptation parameter group that have not received any tokens in the current training step, skip the corresponding optimizer state update. After completing the gradient update of the parameters of the shared low-rank adaptation channel, refresh the parameter version, scaling factor and cache status of the shared low-rank adaptation channel.

9. A low-rank adaptation training system based on residual memory sharing of an expert hybrid model, characterized in that, include: The residual memory budget unit is used to calculate the residual memory budget based on the runtime memory overhead of the graphics processor. A shared low-rank channel configuration unit is used to instantiate a shared low-rank adapter channel in the graphics processor based on the residual video memory budget. The training signal covers the statistical unit, which is used to statistically analyze the routing distribution of the current batch of input words in the expert hybrid model to obtain a routing skew metric. A skew-aware scheduling unit is used to schedule the tasks of the route adaptation path and the execution granularity of the shared low-rank adaptation channel according to the route skew metric. The shared low-rank adaptation execution unit is used to send the input tokens of the current batch into the routing adaptation path and the shared low-rank adaptation channel respectively, and to perform calculations in the routing adaptation path according to the scheduled tasks to generate the routing path output, and to perform segmentation calculations on all input tokens in the current batch in the shared low-rank adaptation channel according to the scheduled execution granularity and independent of the routing selection to generate the shared low-rank output; The forward fusion unit is used to fuse the output of the routing adaptation path with the shared low-rank output during the forward phase to form a layer output; The reverse gradient branch unit is used to split the gradient output by the layer to the routing adaptation path and the shared low-rank adaptation channel respectively during the reverse phase for parameter updates. The instantiation of the shared low-rank adapter channel in the graphics processor based on the residual video memory budget specifically includes: constructing a candidate configuration set containing the number of shared channels, shared rank, intermediate dimension, and parameter precision; calculating the total video memory usage of each candidate configuration and eliminating candidate configurations whose total video memory usage exceeds the limit ratio of the residual video memory budget; for candidate configurations that meet the limit ratio, calculating a comprehensive configuration score according to a preset quality priority, throughput priority, or user-specified priority mode, and selecting the target configuration with the highest score for parameter and buffer instantiation allocation to generate a shared low-rank adapter channel; The execution granularity of the shared low-rank adaptation channel is scheduled, specifically including: calculating the number of micro-batch tokens to be executed in a single submission of the shared low-rank adaptation channel based on the total number of tokens in the current batch, the preset shared channel temporary buffer capacity, the residual memory budget, and the routing skew metric.

Citation Information

Patent Citations

  • Dynamic routing parameter efficient fine tuning method and system based on LoRA-MoE

    CN121480595A

  • Large model fine tuning method and device based on hybrid experts and electronic equipment

    CN122088587A