A residual-guided multi-task model fine-tuning method

Abstract

The application discloses a multi-task model fine-tuning method based on residual guidance, comprising the following steps: a dynamic residual factor extraction step: a multi-task basic model is used to combine the trained residual strategies corresponding to each task to interact, obtain a successful trajectory, and extract a dynamic residual factor; a cross-task normalization step: the successful trajectories of different tasks are normalized based on the distribution of the dynamic residual factor to obtain a normalized residual factor; a curriculum learning weight calculation step: curriculum learning weights are calculated for each sample according to curriculum scheduling parameters and the normalized residual factor; a weighted fine-tuning step: a balanced batch is constructed, the curriculum learning weights are normalized to obtain normalized weights, a weighted loss is calculated based on the normalized weights, and model parameters of the multi-task basic model are updated. The application realizes balanced and efficient performance improvement of the multi-task basic model.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method for fine-tuning a multi-task model based on residual guidance. Background Technology

[0002] Current Vision-Language-Action (VLA) foundational models face significant challenges in the post-training phase after multi-task joint training: Differences in data distribution across tasks (e.g., contact-rich socket tasks versus lightweight placement tasks) lead to conflicting gradient directions and gradient interference. Furthermore, during sequential or hybrid fine-tuning, model learning of new tasks often comes at the cost of overwriting knowledge from older tasks, resulting in catastrophic forgetting. While existing representative works in the VLA field have explored scalable multi-task fine-tuning paradigms, multi-task hybrid training under limited real-world robot data still often results in an average success rate of only 11%. How to effectively utilize single-task interaction experience and mitigate inter-task interference during continuous multi-task updates remains a core open problem in the post-training of VLA models.

[0003] Recent research explores leveraging reinforcement learning (RL) to generate higher-quality training data to improve the performance of general policies. General policy distillation methods based on RL data generation (such as RLDG) use RL-generated data to fine-tune general policies, demonstrating that RL-generated data outperforms pure demonstration data in terms of state coverage and action distribution. "Probe-learn-distillation" framework methods (such as PLD) utilize residual RL to locate failure regions and collect aligned trajectories, then distill them back into the vision-language-action model. However, these methods lack task-level discriminative power when transforming residual information into shared backbone update signals—simply mixing successful trajectories from multiple tasks for supervised fine-tuning (with an average success rate of approximately 23.75%) fails to fully utilize the task-specific corrective structural information inherent in the residual policy.

[0004] The course learning paradigm, which presents samples in ascending order of difficulty to improve model training efficiency, is widely used in fields such as vision, natural language processing, and robot control. However, in the multi-task post-training scenario of vision-language-action models, there is currently no systematic solution for how to automatically extract task difficulty signals from the real-time running data of residual policies and transform them into structured sample weights to guide the weighted flow matching update of the backbone model.

[0005] During the training of residual policies, the interaction between the residual module and the base vision-language-action model naturally generates a structured signal—the dynamic residual factor. This factor comprehensively reflects the value advantage of residual intervention in the current state and whether the policy has fallen into local stagnation, quantifying the degree of residual correction required by the base model at each time step. However, existing methods lack systematic normalization, curriculum scheduling, and balanced training mechanisms when transforming this structured signal into guiding weights for multi-task shared backbone updates.

[0006] Disadvantage 1: The multi-task hybrid supervision fine-tuning lacks a task-level differentiation mechanism, and the gradients of each task interfere with each other, resulting in a significant performance drop in the shared vision-language-action model backbone in real robot multi-task scenarios (the baseline average success rate is only 11.25%).

[0007] Disadvantage 2: When directly mixing and distilling the successful trajectories generated by the residual strategy, the structural information of the residual correction magnitude within the trajectory is ignored, and the samples with high correction requirements and low correction requirements are not effectively distinguished.

[0008] Disadvantage 3: Existing methods lack residual normalization mechanisms for cross-task order-of-magnitude differences. The large heterogeneity of dynamic residual factors between different tasks directly leads to the inability to fairly compare and balance the sample weights across multiple tasks.

[0009] Disadvantage 4: Existing residual knowledge distillation methods lack dynamic course scheduling, making it impossible to achieve progressive learning from low-difficulty basic behaviors to high-difficulty key correction behaviors, which affects the efficiency of policy refinement and the ultimate performance ceiling.

[0010] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0011] The technical problem this application aims to solve is "how to overcome the gradient interference and catastrophic forgetting problems existing in the current technology in the context of continuous fine-tuning of multi-tasks, so as to achieve balanced and efficient performance improvement of the shared model backbone".

[0012] The technical solution adopted in this application to solve the above-mentioned technical problems is as follows.

[0013] This application provides a residual-guided method for fine-tuning a multi-task model, including the following steps: Dynamic residual factor extraction steps: Using the multi-task base model, interact with the trained residual strategies corresponding to each task to obtain multiple successful trajectories, and extract the dynamic residual factors corresponding to each time step from the successful trajectories. The dynamic residual factors are used to quantify the degree of residual correction required by the multi-task base model at the corresponding time step. Cross-task normalization step: For the success trajectories of different tasks, based on the distribution of dynamic residual factors in each task, the dynamic residual factors are normalized respectively to eliminate the difference in the magnitude of dynamic residual factors between different tasks and obtain normalized residual factors. Course learning weight calculation steps: Calculate the course learning weight for each sample in the success trajectory based on the course scheduling parameters and normalized residual factor of the current training step. Weighted fine-tuning steps: Construct a balanced batch containing all samples, normalize the course learning weights within the batch to obtain the normalized weights of each sample, and calculate the weighted loss based on the normalized weights. This is used to supervise and fine-tune the multi-task base model and update its model parameters.

[0014] In some embodiments, during the dynamic residual factor extraction step, the parameters of the residual strategy are frozen, and the dynamic residual factor is obtained by a weighted sum of the greedy value modulation term and the stagnant escape term.

[0015] In some embodiments, the formula for calculating the dynamic residual factor is:

[0016] in, This is the upper bound of the dynamic residual factor. and These are the weighting coefficients for the greedy value modulation term and the stagnation escape term, respectively. For the greedy value modulator, For the stagnation escape term, the range of the greedy value modulation term is [0,1]; the range of the stagnation escape term is [0,+∞).

[0017] In some embodiments, the cross-task normalization step includes quantile-based normalization, which specifically includes: for each task, selecting a first predetermined percentile and a second predetermined percentile of the dynamic residual factor in its successful trajectory set, wherein the second predetermined percentile is greater than the first predetermined percentile; and performing linear scaling and truncation on the dynamic residual factor of each task based on the first predetermined percentile and the second predetermined percentile, mapping it to the [0,1] interval to obtain the normalized residual factor.

[0018] In some embodiments, in the course learning weight calculation step, the course learning weight is calculated using a Gaussian kernel function, and the formula for calculating the Gaussian kernel function is:

[0019] in, As the weight of course learning, Let be the normalized residual factor for the k-th sample of the i-th task. For non-stationary peak values ​​in the course scheduling parameters, Hyperparameters for controlling the width of the Gaussian kernel.

[0020] In some embodiments, after calculating the course learning weights using a Gaussian kernel function, an upper bound truncation is performed on them:

[0021] in The course learning weights are those after being truncated by the upper bound. The upper bound hyperparameter for the course learning weights is denoted as . The peak value of the non-stationary course monotonically increases from the initial value to the final value during the training process.

[0022] In some embodiments, the weighted fine-tuning step includes at least one of weighted flow matching loss, weighted diffusion loss, or weighted regression loss; the form of the weighted loss is as follows: Where B represents the balanced batch, and b is the sample index in the batch. The loss for the b-th sample under the native training paradigm of the multi-task basic model. For normalized weights.

[0023] In some embodiments, the multi-task base model includes a vision-language-action model.

[0024] In some embodiments, a multi-task model training system is also provided, comprising: The successful trajectory collection and dynamic residual factor extraction module is configured to use the multi-task base model to interact with the trained residual strategies corresponding to each task to obtain multiple successful trajectories, and extract the dynamic residual factors corresponding to each time step from the successful trajectories. The dynamic residual factors are used to quantify the degree of residual correction required by the multi-task base model at the corresponding time step. The cross-task residual normalization module is configured to normalize the dynamic residual factors based on the distribution of dynamic residual factors in each task for the success trajectory of different tasks, so as to eliminate the difference in the magnitude of dynamic residual factors between different tasks and obtain normalized residual factors. The residual perception course learning weight calculation module is configured to calculate the course learning weight for each sample in the success trajectory based on the course scheduling parameters and the normalized residual factor. The update module is configured to build a balanced batch containing all samples, normalize the course learning weights within the batch to obtain the normalized weights of each sample, and calculate the weighted loss based on the normalized weights. This is used to supervise and fine-tune the multi-task base model and update its model parameters.

[0025] In some embodiments, an electronic device is also provided, including one or more processors; a memory for storing one or more computer programs; wherein, when the one or more computer programs are executed by one or more processors, the electronic device causes the residual-guided multi-task model fine-tuning method of the present invention to be implemented.

[0026] The present invention has the following beneficial effects: This invention obtains key signals that quantify the degree of residual correction requirements of a multi-task base model by performing a dynamic residual factor extraction step; it eliminates the difference in the magnitude of dynamic residual factors between different tasks by performing a cross-task normalization step, making the sample difficulty comparable across tasks; it transforms the normalized residual factors into course learning weights by performing a course learning weight calculation step, enabling the differentiation of sample training priorities; and it updates the model parameters of the multi-task base model by performing a weighted fine-tuning step, using normalized weights to calculate a weighted loss, thus guiding the update direction of the model parameters towards the samples with the highest need for correction. These steps work together to transform the fine-tuning process of the multi-task base model from a blind, mixed update into a discriminative learning process precisely guided by residual information.

[0027] Clearly, the synergistic combination of the aforementioned technical features systematically solves the technical problem of "how to overcome the gradient interference and catastrophic forgetting problems existing in current technologies in multi-task continuous fine-tuning scenarios, so as to achieve balanced and efficient performance improvement of the shared model backbone." Specifically, the dynamic residual factor provided by the dynamic residual factor extraction step is the foundation for achieving "discrimination"; the normalized residual factor generated by the cross-task normalization step ensures that this discrimination can be carried out fairly across multiple tasks, which is the key to achieving "balance"; the course learning weights generated by the course learning weight calculation step quantify "discrimination" into operable training instructions; finally, the weighted fine-tuning step uses normalized weights and weighted loss to achieve "differentiated" gradient updates at the execution level, thereby effectively overcoming the gradient interference problem caused by "lack of discrimination mechanism" and alleviating catastrophic forgetting by focusing on learning samples with high correction requirements, thus ultimately achieving balanced and efficient performance improvement of the shared model backbone (i.e., the multi-task base model).

[0028] Other beneficial effects of the present invention will be further described below. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 The overall framework diagram of the pre-trained multi-task base model (multi-task vision-language-action model) provided in the embodiments of the present invention is continuously updated with weights. Figure 2 This is an internal flowchart of successful trajectory collection and dynamic residual factor extraction provided in an embodiment of the present invention; Figure 3 This is an internal flowchart of cross-task residual normalization provided in an embodiment of the present invention; Figure 4 This is an internal flowchart for calculating residual-aware course learning weights provided in an embodiment of the present invention. Figure 5 This is an internal flowchart of the batch normalization regularization and weighted flow matching strategy update provided in an embodiment of the present invention; Figure 6 The overall flowchart of multi-task weighted continuous update training provided in the embodiments of the present invention is shown. Detailed Implementation

[0030] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0031] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0032] This invention aims to solve the problems of gradient interference and insufficient utilization of residual information in the continuous updating of vision-language-action models in multiple tasks. It proposes a residual-guided fine-tuning method for multi-task models, which realizes a systematic, balanced and progressive update of the shared vision-language-action model backbone (i.e. a specific implementation of the multi-task basic model) under the condition of limited real robot interaction data.

[0033] Specifically, in response to the problems of multi-task gradient interference and catastrophic forgetting, this invention constructs sample-level weights through residual factors, automatically distinguishes high-value samples from low-value samples under the weighted flow matching objective, and guides the backbone to align with the key behaviors of the task. To address the heterogeneity of dynamic residual factors across tasks, this invention introduces a robust normalization mechanism based on percentiles, which maps the residual factors of each task to the [0,1] interval, thereby achieving comparability of sample weights among multiple tasks. To address the issue of missing course learning, this invention designs a dynamic course scheduling strategy that uses residual factors as a difficulty proxy. Through monotonically increasing scheduling of non-stationary peak values, it guides the model to gradually transition from basic behaviors to complex scenarios with high correction requirements. To address the issue of inconsistent gradient magnitudes across batches, this invention employs batch normalization regularization to ensure gradient stability under different weight distributions, preventing extreme weight samples from dominating the update direction.

[0034] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0035] In some embodiments, as shown in Figure 1, this invention proposes a residual-guided multi-task model fine-tuning method. This method relies on the dynamic residual factor signal generated by the residual policies trained independently for each task. By freezing the residual policy parameters, the residual information is used to guide the weighted supervised update of the shared vision-language-action model backbone. The technical solution of this invention covers the following four sub-modules: Module 1 (Successful Trajectory Collection and Dynamic Residual Factor Extraction Module): As shown in Figure 2, a synthetic control strategy is constructed using a pre-trained shared vision-language-action model backbone and task-independent residual policies. Real-world interactive sampling is performed for tasks 1 to M in a real physical or simulation environment, and successful trajectories are selected and collected to construct a trajectory set. For each time step of each successful trajectory, the dynamic residual factor, pre-calculated during the execution of the residual policy, is extracted using the dynamic residual factor reading module. This factor was calculated synchronously with the optimization of the residual strategy during the independent training of the single-task residual strategy in Phase I. In Phase II of this invention, it is directly read and used as a known structured prior signal, and its parameters are frozen along with the residual strategy and are no longer updated.

[0036] Dynamic residual factor The definition is as follows:

[0037] in, The upper bound of the factor, and These are the weighting coefficients for the two complementary signal terms.

[0038] This is the Greedy Value Modulation term, which ensures that residual intervention is only allowed if it leads to a positive value increase. This term estimates the state-action value of the basic actions using a Critic Network. The state of the action after superimposed residual correction - action value The difference between the two is used to obtain the dominance function. After being gated and mapped (forcing non-positive gains to zero), the output value ranges from [0, 1]. When the gain value is non-positive, The value is zero, thus preventing residual intervention from introducing jitter when it is not beneficial. It should be noted that μ represents the basic action output by the multi-task base model (i.e., the base vision-language-action model), and π represents the residual correction action output by the residual strategy for the current state. The two are superimposed to synthesize the final execution action. = - This value quantitatively characterizes the state-action value gain brought about by residual intervention relative to the base action; the commentator network is a neural network in the field of reinforcement learning used to estimate the state-action value Q function.

[0039] This is the stagnation escape term, which acts as a buffer against excessive conservatism in the commentator network. This provides a supplementary correction trigger signal when the strategy is suppressed and has actually stagnated in performance. This item tracks the actual value gain within a fixed lookback window and compares it with a historical expected gain baseline maintained based on the Exponential Moving Average (EMA). It outputs a positive value only when the actual gain is lower than the historical expected level, with a value range of [0, +∞).

[0040] From a physical perspective, This reflects the degree of residual correction required by the basic vision-language-action model in the current state: a high value indicates that the basic action is insufficient to complete the current operation and requires significant correction; a low value indicates that the basic strategy is close to optimal and residual intervention should be suppressed. This invention utilizes this factor as a sample-level difficulty proxy signal to guide cross-task normalization, course learning weight calculation, and weighted flow matching update in subsequent modules two through four.

[0041] Regarding dynamic residual factor The reliability of the calculation: The factor is calculated synchronously during the single-task residual policy training in stage I. Its reliability is guaranteed by the following technologies: (1) The critic network is preheated and initialized before online training using the objective function based on Calibrated Q-Learning (Cal-QL) to suppress optimistic value estimation of out-of-distribution actions; (2) The strategy of taking the minimum value of the dual critic network is used to suppress Q-value overestimation; (3) Each training batch consists of a mixture of online interactive data and offline expert data to provide continuous high-quality calibration for the critic network; (4) The factor itself is composed of a greedy value modulation term. and stagnation escape item The residual is weighted by two complementary signals. The former is forced to zero when there is no positive value improvement in the residual intervention to prevent unhelpful jitter. The latter provides a supplementary trigger when the commentator is overly conservative by tracking the historical value gain trend, so that the factor has inherent fault tolerance to the single-step estimation bias of the commentator network; (5) The final factor is smoothed by exponential moving average to ensure temporal stability. The detailed implementation of the above technology belongs to the technical scope of single-task residual strategy training in stage I. In stage II of this invention, the residual factor is directly read as a known frozen prior signal and is no longer recalculated or updated.

[0042] Verification of the effectiveness of the dynamic residual factor as a multi-task weighting signal: This invention conducted a systematic comparative experiment on the Franka real robot platform for four typical operation tasks (weighted object insertion, tissue box pulling, balanced placement, and toy hanging). The results are shown in Table 1. Baseline (a multi-task hybrid supervised fine-tuning approach): This method directly mixes the teleoperation teaching trajectories for each task (20 trajectories per task, 80 trajectories in total) and then performs joint supervised fine-tuning on a shared vision-language-action model. This approach does not use any residual policy information and represents the most common multi-task fine-tuning paradigm in the prior art.

[0043] Residual Reinforcement Learning (RL): Building upon the baseline model described above, residual policies are trained independently for each task. During execution, the final control command is generated through the dynamic synthesis of the base action and the residual action. This method requires running both the vision-language-action model and the residual policy network simultaneously during the inference phase, resulting in higher deployment costs. However, it represents the upper bound of performance achievable with residual correction and serves as a reference.

[0044] Residual Distillation (Direct Residual Distillation – Ablation Control): This method uses the exact same successful trajectory dataset as the present invention (successful trajectories collected after the VLA model and residual policy are executed together, 20 per task), but instead of using residual factors as sample weighting signals during the supervised fine-tuning stage, all samples are trained with equal weights. The only difference between this method and the present invention is whether or not a residual factor-guided weighting mechanism is introduced. This method is an ablation experiment used to verify the independent contribution of the weighting mechanism described in the present invention.

[0045] Residual-Guided SFT (Residual Factor Guided Weighted Supervised Fine-Tuning – This Invention): It adopts the same data generation process as Residual Distill, but introduces the complete process of dynamic residual factor extraction, cross-task normalization, course learning weight calculation and weighted loss optimization described in this invention in the supervised fine-tuning stage to perform weighted fine-tuning updates on the shared vision-language-action model.

[0046] Table 1

[0047] This invention compares a method of directly mixing successful trajectories for distillation without using residual factor weighting with the residual factor-guided weighted supervised fine-tuning method proposed in this invention. Both methods use the exact same successful trajectory dataset; the only difference is whether or not residual factors are used as sample weighting signals. Results show that the method of directly mixing successful trajectories for distillation has an average success rate of 23.75%, while the residual factor-guided weighted supervised fine-tuning method achieves an average success rate of 85.0%, an improvement of 61.25 percentage points. This significant difference demonstrates that residual factors effectively capture the correction needs of the base model under different states and can serve as a reliable sample-level difficulty proxy signal to guide multi-task weighted updates.

[0048] Module 2 (Cross-Task Residual Normalization Module): As shown in Figure 3, the cross-task residual normalization module eliminates the heterogeneity of dynamic residual factors among different tasks. First, it groups the data by task and independently performs quantile statistics for each task i, calculating the 20th percentile q20 and 80th percentile q80 of the residual factors in the successful trajectory set. Then, it performs quantile normalization transformation:

[0049] This transformation converges the residual factor distribution of each task to a uniform [0,1] interval by removing subtle noise below q20 and extreme outliers above q80. The 20th and 80th percentiles were chosen as normalization anchors based on the following: the bottom 20% of samples correspond to time steps where the underlying model is close to optimal and residual requirements are extremely low, representing a noisy region with low information content; the top 20% of samples may contain extreme outliers caused by occasional estimation biases from the reviewer network. Using the middle 60% of the data range (q20 to q80) to define the effective scaling interval strikes a balance between data information utilization and outlier robustness compared to q10 / q90 (which may not adequately filter outliers) or q25 / q75 (which may excessively shrink the effective range). The above quantiles are preferred parameters; in practical applications, quantile settings can be adjusted according to task characteristics.

[0050] For tasks where factor values ​​are concentrated in a narrow range (i.e., tasks where the base model performs relatively evenly), linear scaling may amplify the original differences. However, this invention ensures training stability through a three-layer protection mechanism: First, the truncation mapping operation strictly restricts the normalized values ​​to the [0, 1] interval; second, the upper bound of the weights in subsequent module three is truncated (preferably...). The first layer limits the influence range of any single sample weight; the second layer, intra-batch weight normalization in module four, ensures stable batch-level gradient magnitude. These three mechanisms work synergistically to significantly reduce the sensitivity of the normalization method to the distribution of factors.

[0051] As an alternative implementation, the quantile normalization described above can also be replaced with a ranking-based normalization scheme. Specifically, for all samples within each task i, normalization is performed according to their original residual factors. The values ​​are sorted in ascending order, and the rank of each sample is linearly mapped to the interval [0, 1].

[0052] in The residual factor of sample k represents the ascending rank of all samples in task i, where N_i is the total number of samples in task i. This scheme is independent of the specific distribution of factor values ​​and is naturally robust to skewed distributions, multimodal distributions, or extreme outliers. In special task scenarios where the factor value distribution exhibits severely non-uniform characteristics, implementers can choose to use rank normalization instead of quantile normalization, while the subsequent course learning weight calculation, weight truncation, and intra-batch normalization processes remain unchanged. This invention prefers the quantile normalization scheme because it preserves the absolute magnitude information of the original factor values, which is beneficial for the comparability of the degree of correction required across tasks, and the calculation anchor point is fixed, eliminating the need for global re-sorting as the dataset changes.

[0053] By using the quantile normalization described above, we can reduce the impact of differences in residual scales between tasks on sample weighting, so that the learning weights of subsequent multi-task courses can be established on a comparable scale of correction requirements.

[0054] Module 3 (Residual-Aware Course Learning Weight Calculation Module): As shown in Figure 4, the residual-aware course learning weight calculation module updates the weights using a monotonically increasing strategy through the course peak scheduler. Based on the normalized residual factor, the Gaussian kernel weight is calculated for the k-th sample of the i-th task using the Gaussian kernel weight calculation module:

[0055] in To control the hyperparameters of the distribution width, The peak value is for non-stationary courses. The physical meaning is the focus width of course learning in the normalized residual factor space: When smaller, only with Very similar samples receive significant training weights, the course focuses on narrow interval samples, with high discrimination but few effective samples; When the sample size is large, more samples receive higher weights, resulting in smoother training but reduced difficulty and discrimination. The scaling factor needs to match the scale of the normalized range [0, 1]. Too small a value (e.g., 0.05) will result in insufficient effective samples within the window and unstable training; too large a value (e.g., above 0.5) will cause the window to cover almost all samples, losing its course discrimination effect. The preferred value is [value missing]. At this time, the distance Samples within approximately ±0.6 are assigned significant weights, ensuring sufficient sample coverage while maintaining good difficulty discrimination. In practical applications, the weights can be adjusted within the range of 0.1 to 0.5 depending on the number of tasks and data distribution. To prevent extreme samples from dominating training, the weights are further truncated using a weight upper bound truncation module. ,in The upper bound hyperparameter for the weights is preferably set to 2.0 to prevent a few values ​​from falling exactly within the bounds. Nearby samples receive excessive weight and dominate the gradient update direction.

[0056] Course scheduling strategy: The scheduling is monotonically increasing throughout the training process, and the optimal solution is linear scheduling.

[0057] Where t is the current training iteration step, This represents the total number of iterations. and These are the course start and end values, respectively. Preferred selection. , . The reason for using 0.2 instead of 0 is that samples with a value below 0.2 after normalization correspond to time steps with extremely low residual correction requirements, and their training value is limited, so they do not need to be used as the starting point of the course. The reason for using 0.8 instead of 1.0 is that samples with a normalization value close to 1.0 may contain a small number of extreme outliers. Terminating the training at 0.8 can prevent the model from being affected by extreme samples at the end of training. (In the early stages of training...) With a Gaussian kernel around 0.2, it assigns high weights to regions with low residual requirements, allowing the model to prioritize learning basic behaviors (i.e., operational steps that the basic vision-language-action model can already perform well); as training progresses... Gradually increase the size of the sample, shifting the course focus to samples with high residual requirements; towards the end of the training period. Approaching 0.8, the model focuses on challenging key correction behaviors, achieving progressive refinement from easy to difficult. In addition to linear scheduling, cosine scheduling (with slower changes in the early and late stages and faster changes in the middle stage) or segmented tiered scheduling (dividing the training process into several stages, with each stage using a fixed peak value) can also be used as alternatives.

[0058] Module 4 (Batch Normalization Regularization and Weighted Flow Matching Strategy Update Module): As shown in Figure 5, in the batch normalization regularization module, each task is first sampled equally through a balanced batch construction step, and then intra-batch weight normalization is performed to ensure gradient magnitude stability.

[0059] Where ε is the numerical stability constant. Based on normalized weights, in the weighted flow matching update module, backpropagation and parameter updates are performed through flow matching conditional path construction and weighted flow matching loss calculation, ultimately obtaining the updated shared vision-language-action model backbone:

[0060] in This is a learnable velocity field, meaning it shares the parameterized form of the visual-language-action model backbone within the flow matching framework, rather than being an additional network structure. Within the flow matching framework, the visual-language-action model is parameterized as a conditional velocity field: its input includes the coupling states. (Linear interpolation of noise and target action at flow matching time t ∈ [0,1]), the continuous-time variable t of flow matching, and multimodal conditional information c (visual observation, language commands, ontology perception, etc.), the output is a velocity vector, indicating how to adjust the action distribution towards the target direction at the current coupling state. The gradient of the flow matching loss is directly used to update the backbone parameters of the shared vision-language-action model. (Right now ). The target velocity field determined by the conditionally optimal transmission path can be analytically calculated under the linear conditional path as follows: . This refers to the final action (obtained by the synthesis strategy) corresponding to each sample in the successful trajectory.

[0061] This invention uses flow matching as the update paradigm for the backbone model because the preferred vision-language-action model backbone (including but not limited to...) is... Visual-language-action (VLA-OFT, etc.) models themselves use stream matching as their native training paradigm, and weighted supervised updates follow the native training paradigm of the backbone model to ensure seamless compatibility. If a visual-language-action model with other training paradigms is chosen as the backbone (e.g., a model using regression action heads), then the stream matching loss can be replaced with the corresponding weighted regression loss.

[0062] The core innovation of this invention lies not in the selection of a specific training paradigm, but in introducing a residual factor-driven sample-level weighting into the supervised update loss of the backbone model. Regardless of whether the underlying training paradigm is flow matching, diffusion model, or direct regression, the general form of the weighted loss can be uniformly expressed as:

[0063] in This is the loss for the b-th sample under the original training paradigm of the backbone model. This weighting mechanism assigns higher training weights to samples with high residual demand (time steps where the base model has large biases and requires significant correction), and lower weights to samples with low residual demand (time steps where the base model is close to optimal). This prioritizes the allocation of limited gradient update budgets to the regions where the model most needs improvement, driving the vision-language-action model backbone to learn how to adjust its action distribution under high residual demand conditions. This allows the backbone parameter level to digest the task-level corrections provided by the residual strategy, achieving a systematic fusion of multi-task interaction experience. This weighting mechanism is independent of specific training paradigms and can be used in conjunction with any vision-language-action model architecture.

[0064] refer to Figures 1 to 6 In specific implementations, the method of this invention can use various multi-degree-of-freedom robotic arms as the hardware platform. The specific implementation steps are as follows: Step 1: Multi-task synthesis strategy rollout and successful trajectory collection. M target tasks are set. First, a hybrid training set is constructed based on demonstration trajectories of multiple target tasks. Multi-task joint supervised fine-tuning is then performed on the general pre-trained vision-language-action model to obtain a shared multi-task vision-language-action backbone model, thus completing the joint pre-training of the multi-task base model. This shared backbone serves as the initialization base model for subsequent residual factor-guided continuous updates in this invention. The system calls the shared vision-language-action model backbone network and the M pre-trained task-specific residual policies. This process corresponds to the independent residual policy training and judgment for each task. During the independent training process in Stage I, each task-specific residual policy has, through comprehensive evaluation of the greedy value advantage and stagnation escape signal of residual intervention, synchronously calculated and recorded the dynamic residual factor at each time step of each interaction trajectory. The factor is defined as:

[0065] in The difference in Q-values ​​between the basic action and the synthesized action is obtained through gating mapping. The residual factor is derived based on the deviation between the actual value gain and the historical expected gain. In Phase II of this invention, the aforementioned residual factor is directly used as a known input signal, and the residual strategy parameters remain frozen. The aforementioned synthetic control strategy is used in parallel operation in a real physical or simulation environment, i.e., parallel interactive sampling of the synthetic strategy is performed. Subsequently, the successful trajectory filtering and collection module filters and collects trajectories that have successfully completed the task, constructing a successful trajectory dataset, thus completing the successful trajectory collection and residual factor extraction.

[0066] Step 2: Residual Factor Cross-Task Normalization. To address the differences in intensity and amplitude of actions across different tasks, the cross-task residual normalization module performs processing. This stage corresponds to cross-task residual quantile normalization. First, the task-grouping submodule divides the data by label. Then, the quantile statistics submodule calculates the quantile values ​​(preferably the 20th and 80th percentiles) of the residual factor for each task in its successful trajectory set. Finally, the quantile normalization transformation module maps and truncates the original residual factor to the [0, 1] interval.

[0067] As an alternative implementation, the quantile normalization transformation described above can also be replaced with a ranking-based normalization transformation: all samples within each task are sorted in ascending order of their original residual factor values, and the ranking is divided by the total number of samples in that task to obtain a normalized value in the interval [0, 1]. This alternative is suitable for special scenarios where the factor value distribution exhibits severe skewness or multimodal characteristics, and the subsequent processing steps three to five remain unaffected.

[0068] Step 3: Dynamic Course Learning Weight Calculation and Scheduling. The residual-aware course learning weight calculation module initializes the course scheduling parameters. This stage corresponds to the calculation and truncation of course learning weights. In each training step of the backbone fine-tuning, the course peak scheduler updates the weights according to a preset monotonically increasing strategy. Then, the course learning weights of each multimodal sample are calculated using the Gaussian kernel weight calculation submodule, and the upper limit is truncated using the weight upper bound truncation submodule to form a weighted multi-task dataset.

[0069] Step 4: Balanced Batch Sampling and Batch Normalization. During the data loading phase of the batch normalization and regularization module, balanced batch sampling and intra-batch weight regularization are performed. The balanced batch construction unit constructs balanced batches containing an equal number of samples from each task. Subsequently, the intra-batch weight normalization unit performs intra-batch normalization on the truncated weights of all samples within the batch to prevent gradient explosion caused by a few high-weight samples.

[0070] Step 5: Weighted Flow Matching Fine-tuning of the Backbone. In the weighted flow matching strategy update module, the parameters of the shared vision-language-action model backbone are unfrozen. This stage corresponds to weighted flow matching objective optimization and backbone parameter update. A flow matching conditional path construction submodule is used, with the final synthesized action corresponding to each time step in the successful trajectory as the regression objective. Based on the normalized sample weights, the weighted flow matching loss calculation submodule minimizes the prediction error in the velocity field space in a weighted manner. The backbone network parameters are iteratively updated through backpropagation and parameter update modules, and it is determined whether the multi-task fine-tuning has converged. If it has not converged, the course peak parameters are monotonically updated, and the process returns to Step 3 above to continue iterating; if convergence is determined, the process ends, and the updated shared vision-language-action model with stable multi-task generalization ability is finally obtained.

[0071] Compared with the prior art, the present invention has the following beneficial effects: Effect 1: Significantly improved multi-task joint training performance. In multi-task evaluation of four real robot tasks (weighted object insertion, tissue box pulling, balance placement, and toy suspension), the proposed method achieved an average success rate of 85.0%, significantly outperforming the baseline method (11.25%) and direct residual distillation (23.75%), and approaching the upper bound of residual reinforcement learning (90.0%). Notably, direct residual distillation and the proposed method use the exact same successful trajectory dataset; the only difference lies in whether a dynamic residual factor is used as the sample weighting signal. The 61.25 percentage point performance difference directly verifies the effectiveness of the residual factor-guided weighting mechanism. This demonstrates that the proposed method can effectively transfer single-task residual interaction experience to a shared vision-language-action model backbone.

[0072] Effect 2: Effectively mitigates gradient interference from multiple tasks. Through a sample weighting mechanism guided by residual factors, key samples and ordinary samples are automatically distinguished during the flow matching update process, making the gradient directions of each task more coordinated, avoiding mutual cancellation and overwriting between tasks, and achieving stable joint alignment of multiple tasks.

[0073] Effect 3: Automatic quantification of task-level update requirements. As a structured signal, the dynamic residual factor automatically reflects the degree of correction requirements of each task to the shared backbone under different states, providing task-specific guidance signals for backbone updates without the need for manual annotation or manual design of task weights.

[0074] Effect 4: Gradual course scheduling promotes high efficiency and refinement. Based on The monotonically increasing course scheduling strategy guides the model to first master basic behaviors and then deal with complex scenarios, which improves the training convergence stability and prevents the model from being exposed to high-difficulty samples too early, which would cause training oscillations.

[0075] Effect 5: Can be used in conjunction with any vision-language-action model architecture. The residual normalization, weight calculation, batch normalization, and flow matching update process of this invention does not depend on a specific vision-language-action model architecture, and can be seamlessly integrated with different backbones, exhibiting good versatility and scalability.

[0076] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0081] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A residual-guided based multi-task model fine-tuning method, characterized in that, Includes the following steps: Dynamic residual factor extraction steps: Using a multi-task base model, interact with the trained residual strategies corresponding to each task to obtain multiple successful trajectories, and extract the dynamic residual factor corresponding to each time step from the successful trajectories. The dynamic residual factor is used to quantify the degree of residual correction required by the multi-task base model at the corresponding time step. Cross-task normalization step: For the success trajectories of different tasks, based on the distribution of dynamic residual factors in each task, the dynamic residual factors are normalized respectively to eliminate the difference in the magnitude of the dynamic residual factors between different tasks and obtain normalized residual factors. Course learning weight calculation steps: Calculate the course learning weight for each sample in the successful trajectory based on the course scheduling parameters of the current training step and the normalized residual factor. Weighted fine-tuning steps: Construct a balanced batch containing each sample, normalize the course learning weights within the batch to obtain the normalized weights of each sample, and calculate the weighted loss based on the normalized weights to perform supervised fine-tuning of the multi-task basic model and update its model parameters.

2. The method of claim 1, wherein, In the dynamic residual factor extraction step, the parameters of the residual strategy are frozen, and the dynamic residual factor is obtained by weighted summation of the greedy value modulation term and the stagnant escape term.

3. The method of claim 2, wherein, The formula for calculating the dynamic residual factor is as follows: wherein, is an upper bound of the dynamic residual factor, and are weight coefficients of the greedy value modulation term and the stagnation escape term, respectively, is the greedy value modulation term, is the stagnation escape term, wherein the greedy value modulation term has a value range of [0, 1]; the stagnation escape term has a value range of [0, +∞).

4. The method of claim 1, wherein, In the cross-task normalization step, the normalization process includes quantile-based normalization, which specifically includes: for each task, selecting a first predetermined percentile and a second predetermined percentile of the dynamic residual factor in its successful trajectory set, wherein the second predetermined percentile is greater than the first predetermined percentile; based on the first predetermined percentile and the second predetermined percentile, performing linear scaling and truncation on the dynamic residual factor of each task, mapping it to the [0,1] interval, to obtain the normalized residual factor.

5. The method of claim 1, wherein, In the course learning weight calculation step, the course learning weight is calculated using a Gaussian kernel function, and the formula for calculating the Gaussian kernel function is as follows: wherein, is a course learning weight, is a normalized residual factor for the i-th task and the k-th sample, is a non-stationary course peak in the course schedule parameter, is a hyperparameter that controls the width of the Gaussian kernel.

6. The method of claim 5, wherein, After calculating the course learning weights using the Gaussian kernel function, an upper bound truncation process is applied: wherein is the course learning weight after the upper bound truncation process, is the course learning weight upper bound hyperparameter, and the non-stationary course peak is monotonically increasing from a starting value to an ending value during the training process.

7. The method of claim 1, wherein, In the weighting fine-tuning step, the weighted loss includes at least one of a weighted flow matching loss, a weighted diffusion loss, or a weighted regression loss; and the weighted loss has a form of where B is the balanced batch, b is a sample index in the batch, is a loss of the bth sample under a multi-task base model native training paradigm, is the normalized weight.

8. The method of claim 1, wherein, The multi-task basic model includes a vision-language-action model. 9.A multi-task model training system, characterized in that, include: The successful trajectory collection and dynamic residual factor extraction module is configured to use a multi-task base model to interact with the trained residual strategies corresponding to each task to obtain multiple successful trajectories, and extract the dynamic residual factor corresponding to each time step from the successful trajectories. The dynamic residual factor is used to quantify the degree of residual correction required by the multi-task base model at the corresponding time step. The cross-task residual normalization module is configured to normalize the dynamic residual factors based on the distribution of dynamic residual factors in each task for the success trajectory of different tasks, so as to eliminate the difference in the magnitude of dynamic residual factors between different tasks and obtain normalized residual factors. The residual-aware course learning weight calculation module is configured to calculate the course learning weight for each sample in the success trajectory based on the course scheduling parameters and the normalized residual factor. The update module is configured to construct a balanced batch containing all samples, normalize the course learning weights within the batch to obtain normalized weights for each sample, and calculate a weighted loss based on the normalized weights. This is used to supervise and fine-tune the multi-task basic model and update its model parameters.

10. An electronic device, comprising: It includes one or more processors; a memory for storing one or more computer programs; wherein, when the one or more computer programs are executed by the one or more processors, the electronic device causes the residual-guided multi-task model fine-tuning method as described in any one of claims 1 to 8.

Images