A multi-teacher joint distillation and plug-in model expansion method

Abstract

This invention provides a multi-teacher joint distillation and plug-in model expansion method, belonging to the field of data processing technology. Specifically, it includes: constructing a decoupled student model; performing joint distillation training on the decoupled student model using multiple teacher models, applying feature decoupling constraints to the common features output by the shared backbone network and the task-specific features output by the task-specific adapter during training; in the inference phase, dynamically calling the corresponding task-specific adapter based on the task identifier using the trained decoupled student model, jointly processing the input data with the shared backbone network to obtain the inference result corresponding to the task identifier; and performing plug-in incremental expansion, whereby, in response to the access of a new teacher model to introduce a new task, the shared backbone network remains unchanged, and a new task-specific adapter corresponding to the new teacher model is added and trained. This invention improves flexibility and adaptability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-teacher joint distillation and plug-in model expansion method. Background Technology

[0002] Currently, in existing technologies, multi-teacher joint distillation typically employs a weighted fusion approach to uniformly train student models by combining outputs or intermediate features from different teachers. When teacher models originate from different tasks or data distributions, this approach easily leads to feature expression conflicts and redundant storage within the student model, making it difficult for the model to form a stable, reusable, and universal representation structure. Furthermore, when the system needs to introduce new tasks or capabilities, it often requires a complete retraining of the student model or a large-scale parameter update, which not only incurs high training costs but also easily degrades the performance of already learned tasks, affecting the model's online scalability and continuous evolution capabilities.

[0003] It is evident that there is an urgent need for a flexible and adaptable multi-teacher joint distillation and plug-in model extension method. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a multi-teacher joint distillation and plug-in model extension method, which at least partially solves the problems of poor flexibility and adaptability in the prior art.

[0005] This invention provides a method for multi-teacher joint distillation and plug-in model extension, including: Step 1: Construct a decoupled student model, wherein the decoupled student model includes a shared backbone network and multiple task-specific adapters corresponding to multiple teacher tasks; Step 2: Use multiple teacher models to jointly distill and train the decoupled student model. During the training process, apply feature decoupling constraints to the common features output by the shared backbone network and the task-specific features output by the task-specific adapter to obtain the trained decoupled student model. Step 3: Perform plug-in inference. In the inference phase, the trained decoupled student model dynamically calls the corresponding task-specific adapter according to the task identifier, and processes the input data together with the shared backbone network to obtain the inference result corresponding to the task identifier. Step 4: Perform plug-in incremental expansion. When a new teacher model is connected to introduce a new task, in response to the connection of the new teacher model, the shared backbone network remains unchanged, and a new task-specific adapter corresponding to the new teacher model is added and trained to achieve incremental expansion of the model's capabilities.

[0006] According to a specific implementation of an embodiment of the present invention, step 1 specifically includes: Step 1.1: Construct a neural network with deep feature extraction capabilities as a shared backbone network to extract common features from the input data. ; Step 1.2: For multiple teacher tasks, construct lightweight task-specific adapters to extract the task-specific features of each teacher task. ; Step 1.3: Generate the final fusion feature for the i-th teacher task using an additive fusion method. ; in, and The feature dimensions remain consistent.

[0007] According to a specific implementation of an embodiment of the present invention, the step of performing joint distillation training includes: Based on the outputs of multiple teacher models on the same input sample, a backbone supervision signal is constructed to represent multi-teacher consensus knowledge. By constraining the feature representation of the shared backbone network with the backbone supervision signal, the shared backbone network can focus on learning general common knowledge across tasks.

[0008] According to a specific implementation of an embodiment of the present invention, the step of constructing the backbone supervision signal for characterizing multi-teacher consensus knowledge includes: Calculate the consensus center and divergence covariance matrix output by multiple teacher models, where the expression for the consensus center is: ; in, This represents the total number of teacher models introduced. The index number represents the teacher model, and its value is a natural number from 1 to n. This represents the feature vector output by the i-th teacher model for the same input sample x; The expression for the divergence covariance matrix is: ; Among them, superscript This represents the matrix transpose operation; Constructing a consensus projection operator based on the divergence covariance matrix ; in, It is the identity matrix. Represents trace operation. To prevent numerical stability constants with a denominator of zero; Based on the consensus projection operator, the output of the shared backbone network will be... After projecting onto the teacher consensus subspace and aligning it with the consensus center, a common distillation loss is constructed: .

[0009] According to a specific implementation of an embodiment of the present invention, the feature decoupling constraint is specifically a feature space orthogonal decoupling constraint, which achieves orthogonal decoupling of the two in the feature space by minimizing the projection component of the task characteristic features output by the task-specific adapter onto the common feature direction output by the shared backbone network.

[0010] According to a specific implementation of the present invention, the feature decoupling loss function corresponding to the orthogonal decoupling constraint of the feature space is: ; in, Indicates shared backbone features and the first The inner product between the outputs of each task-specific adapter is used to measure the strength of their correlation in the feature space; The scale factor represents the core characteristics and is used to eliminate the instability caused by scale changes.

[0011] According to a specific implementation of an embodiment of the present invention, the joint distillation training employs a joint loss function to optimize the decoupled student model, wherein the expression of the joint loss function is as follows: ; in, This represents the overall distillation loss used to constrain the consistency between the student model's final output and the teacher model's output. , , To balance the weights.

[0012] According to a specific implementation of an embodiment of the present invention, the step of dynamically invoking the corresponding task-specific adapter based on the task identifier includes: In resource-constrained deployment environments, parameters of the shared backbone network are kept in memory, while parameters of multiple task-specific adapters are stored in separate files on external storage media. Based on the task identifier of the input data, dynamically load the corresponding target task-specific adapter parameters from the external storage medium into memory; The input data is processed sequentially through a shared backbone network and a loaded task-specific adapter, and the inference result is obtained by fusing the output features of the two.

[0013] According to a specific implementation of an embodiment of the present invention, the step of adding and training a new task-specific adapter includes: Freeze the parameters of the already trained shared backbone network; Instantiate a new task-specific adapter corresponding to the new teacher model; Using training data from the new task, and with the output of the new teacher model as supervision, a new task-specific adapter is trained, and during the training process, the output of the new task-specific adapter and the output of the shared backbone network are kept in compliance with the feature decoupling constraint. The parameters of the newly trained task-specific adapter are stored independently and registered in the adapter plugin library.

[0014] The multi-teacher joint distillation and plug-in model expansion scheme in this embodiment of the invention includes: Step 1, constructing a decoupled student model, wherein the decoupled student model includes a shared backbone network and multiple task-specific adapters corresponding to multiple teacher tasks; Step 2, using multiple teacher models to perform joint distillation training on the decoupled student model, applying feature decoupling constraints to the common features output by the shared backbone network and the task-specific features output by the task-specific adapters during the training process, to obtain a trained decoupled student model; Step 3, performing plug-in inference, in the inference stage, using the trained decoupled student model to dynamically call the corresponding task-specific adapter according to the task identifier, jointly processing the input data with the shared backbone network, to obtain the inference result corresponding to the task identifier; Step 4, performing plug-in incremental expansion, when a new teacher model is connected to introduce a new task, in response to the connection of the new teacher model, keeping the shared backbone network unchanged, adding and training a new task-specific adapter corresponding to the new teacher model, so as to achieve incremental expansion of the model's capabilities.

[0015] The beneficial effects of the embodiments of the present invention are as follows: 1. It effectively solves the problems of feature aliasing and negative transfer in multi-teacher knowledge fusion, and improves the generalization ability of the model's underlying representation.

[0016] Existing technologies often suffer from disordered shared feature spaces due to gradient conflicts when integrating multi-teacher knowledge. This invention addresses this issue by employing a backbone commonality distillation mechanism. Utilizing the consensus centroid of multi-teacher output distributions as a supervisory signal, it forces the student model's backbone network to focus on cross-task commonality knowledge. This "convergence-seeking" strategy effectively filters out noise and bias specific to a single teacher's task, resulting in a backbone network with strong robustness and generalization ability, thus constructing a solid foundation of common features for multi-task learning.

[0017] 2. It achieves deep orthogonal decoupling of the feature space and maximizes parameter utilization, significantly reducing model redundancy.

[0018] To address the parameter waste inherent in traditional adapter techniques, where the adapter repeatedly learns core knowledge, this invention innovatively introduces a feature orthogonality decoupling constraint. This mathematically and geometrically forces the feature vectors generated by the task adapter to remain orthogonal to the core feature vectors, ensuring that the adapter stores only "pure" residual information that is linearly independent of the core. This mechanism enables the adapter to accurately represent specific tasks with a minimal parameter size, significantly improving the model's parameter representation efficiency and storage economy.

[0019] 3. It has the capability for low-cost incremental expansion with "zero catastrophic forgetting", supporting the continuous and lossless evolution of the model.

[0020] When faced with the need to expand to new tasks, this invention effectively alleviates the catastrophic forgetting problem common in traditional continuous learning methods by employing a strategy of freezing the backbone and incremental adaptation. Since the introduction of new capabilities relies entirely on newly added independent plugins, and the backbone parameters remain constant, the structure strictly guarantees zero regression in the performance of older tasks. Simultaneously, it eliminates the need for retraining the massive backbone network and replaying historical data, significantly reducing the computational cost and time required for model iteration, achieving truly low-cost and sustainable model evolution.

[0021] 4. Supports dynamic plug-in deployment for resource-constrained environments, significantly improving the flexibility of edge applications.

[0022] The decoupled "base + plug-in" architecture built upon this invention provides significant flexibility for engineering deployment. In resource-constrained environments such as mobile devices or embedded systems, the system does not need to load multiple complete heavy models; instead, it adopts a "one master, multiple slaves" on-demand loading mode. The inference engine only needs to reside on a single backbone network, dynamically switching between millisecond-level lightweight adapter files based on actual business requests. This storage-compute separation deployment significantly reduces runtime memory (RAM) usage and storage (ROM) overhead, enabling a single edge device to handle large-scale heterogeneous tasks. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a multi-teacher joint distillation and plug-in model expansion method provided in an embodiment of the present invention; Figure 2 A student model architecture diagram provided for an embodiment of the present invention; Figure 3 This invention provides a feature orthogonal decoupling training graph according to an embodiment of the invention. Figure 4 A multi-teacher joint distillation diagram provided as an embodiment of the present invention; Figure 5 This is a flowchart of plug-in inference and dynamic deployment provided for an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0028] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0029] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0030] With the widespread application of artificial intelligence models in perception, understanding, and decision-making, how to enable a single model to simultaneously possess multi-task and multi-domain processing capabilities under limited computing and storage resources has become a crucial technical challenge in the current model deployment and evolution process. Multi-teacher knowledge distillation, by introducing multiple teacher models with different functions, guides student models to learn comprehensive representations across tasks and is widely used for model compression and capability fusion.

[0031] In existing technologies, multi-teacher joint distillation typically employs a weighted fusion approach to uniformly train student models by combining outputs or intermediate features from different teachers. When teacher models originate from different tasks or data distributions, this approach can easily lead to feature expression conflicts and redundant storage within the student model, making it difficult for the model to form a stable, reusable, and universal representation structure. Furthermore, when the system needs to introduce new tasks or capabilities, it often requires a complete retraining of the student model or a large-scale parameter update. This not only incurs high training costs but also easily degrades the performance of already learned tasks, affecting the model's online scalability and continuous evolution capabilities.

[0032] Furthermore, existing continuous learning or multi-task compression methods generally rely on historical data playback or shared parameter fine-tuning to maintain the performance of old tasks. In engineering practice, this can lead to additional storage overhead, data management complexity, and potential data distribution shift risks, making it difficult for models to support plug-in deployment modes that allow for on-demand loading and on-demand scaling.

[0033] Therefore, how to effectively decouple and structure the shared information across tasks and the task-specific information by constructing a specific spatial constraint mechanism during the multi-teacher knowledge fusion process, thereby reducing interference between tasks and achieving low-cost incremental expansion and on-demand access to model capabilities, remains a problem to be solved in current multi-teacher distillation and continuous learning technologies.

[0034] Existing technologies include: 1. Based on multi-teacher knowledge distillation technology Existing technologies have proposed knowledge distillation schemes based on multi-teacher models. These schemes typically involve simultaneously incorporating the outputs or intermediate features of multiple teacher models for joint supervised training of the student model. Using soft-label fusion or feature matching, the student model achieves multi-task processing capabilities with a relatively small parameter scale. Its advantage lies in integrating domain knowledge from different teachers, but its drawbacks include: significant differences in knowledge distribution among different teachers, and existing schemes often employ simple weighted fusion, which easily leads to feature aliasing and knowledge conflicts within the student model. In heterogeneous task scenarios, the lack of an explicit extraction mechanism for the "greatest common divisor across tasks" makes it difficult for the model to form a stable and highly generalizable universal representation.

[0035] 2. Based on continuous learning and incremental training techniques Existing continuous learning methods primarily train new tasks sequentially while maintaining old ones, typically employing historical sample replay, regularization constraints, or parameter freezing strategies. These methods maintain the performance of older tasks through memory caching mechanisms, enabling gradual capability expansion. However, in engineering implementation, these methods are heavily reliant on historical data, increasing storage and management overhead. Furthermore, due to the high degree of parameter sharing and lack of structured isolation between different tasks, negative transfer interference is easily introduced when expanding new capabilities, making it difficult to achieve truly "residual" feature storage and lossless performance maintenance.

[0036] 3. Network architecture-based extension technology Architecture-based extension methods differentiate tasks by adding independent branches for different tasks. These methods stack multi-task modules on a shared backbone network, improving the model's adaptability to complex scenarios. However, existing methods often only achieve extension through physical stacking, lacking constraints on the correlation between backbone features and extension module features. This leads to adapter modules repeatedly learning features already covered by the backbone network, resulting in severe representation redundancy and wasted computational resources, and the inference path often becomes bloated as the number of tasks increases.

[0037] 4. Based on model fine-tuning and transfer learning techniques Traditional approaches typically involve updating the overall or partial parameters of a pre-trained model for a new task. While these approaches perform well in task-specific transfer learning, they lack fine-grained segmentation of the model's internal knowledge structure. When the system frequently introduces new tasks, overall fine-tuning can easily disrupt the original parameter distribution, leading to "catastrophic forgetting." Furthermore, existing fine-tuning methods cannot achieve smooth feature stacking through lightweight plugins without altering the core parameters, making it difficult to support flexible deployment modes that require on-demand loading.

[0038] It is evident that the shortcomings of existing technologies include: 1. Severe Knowledge Conflicts and Feature Aliasing: Existing multi-teacher distillation or multi-task fusion schemes lack effective mechanisms to distinguish between general common knowledge and task-specific characteristics. The knowledge of different teachers interferes with each other in the shared feature space of the student model, making it difficult for the model to extract low-level representations with high generalization ability, and easily leading to feature aliasing and representation instability.

[0039] 2. Redundant Feature Representation and Lack of Spatial Decoupling Mechanism: Existing architecture extension schemes do not systematically model the feature correlation between the shared network and task extension modules. Due to the lack of effective feature space decoupling constraints between modules, extension modules easily store redundant information that overlaps with the shared network, making it difficult to achieve efficient utilization of parameters and computing resources.

[0040] 3. High cost of model expansion and continuous evolution: Existing continuous learning and model fine-tuning methods often require large-scale parameter updates to the entire model or rely on historical data replay when introducing new tasks. The training overhead is high and it is easy to cause performance regression of existing tasks, which makes it difficult to meet the engineering requirements of rapid online expansion and lossless evolution of models.

[0041] 4. Lack of flexible plug-in deployment capabilities: Most solutions do not physically and logically decouple general capabilities from specific capabilities at the model architecture level, making it difficult to support the dynamic loading, unloading, or rapid replacement of specific functional modules according to actual application needs, thus limiting the deployment flexibility of the model on the edge or in resource-constrained environments.

[0042] This invention provides a multi-teacher joint distillation and plug-in model expansion method, which can be applied to the model training process in Internet scenarios.

[0043] See Figure 1 This is a flowchart illustrating a multi-teacher joint distillation and plug-in model expansion method provided by an embodiment of the present invention. Figure 1 As shown, the method mainly includes the following steps: Step 1: Construct a decoupled student model, wherein the decoupled student model includes a shared backbone network and multiple task-specific adapters corresponding to multiple teacher tasks; The student model is constructed using a decoupled architecture, consisting of a shared backbone and task-specific modules. The shared backbone is used to extract general representations across tasks, while the feature modules support differentiated capabilities for different tasks. This structural decoupling ensures the model maintains stable basic capabilities while providing scalable interfaces, offering a unified platform for subsequent multi-teacher distillation and plug-in-like extensions.

[0044] In practical implementation, this step aims to complete the initialization and decoupling design of the student model at the network topology level. Unlike the hard-sharing mechanism used in traditional multi-task models (i.e., all tasks share the underlying layer, only the top layer is separated), such as... Figure 2 As shown, this invention constructs a composite architecture combining a "globally shared backbone network" and a "multi-path parallel task-specific adapter." The design goal of this architecture is to provide physically separated storage media for multi-source knowledge. The backbone network serves as the foundation for carrying general knowledge, while the adapter acts as a lightweight plug-in module specifically designed to carry the differentiated residual features of specific teacher tasks. Through this structured design, the model possesses the physical basis for separating "commonality" and "specificity" even in the initial, untrained phase.

[0045] Shared backbone network architecture design: First, a neural network with deep feature extraction capabilities is constructed as a globally shared backbone. This backbone network is the core foundation of the entire system, used to process the raw input data and extract general representations with high generalization ability. In terms of structural design, the backbone network usually adopts mature deep convolutional neural networks, such as the ResNet series or Transformer architecture, to ensure that it has sufficient parameter capacity to capture the underlying common patterns across tasks.

[0046] In the data flow, input data After forward computation by the globally shared backbone network, it is mapped to a high-dimensional feature vector, denoted as . This feature vector This represents common information in the input data across all target tasks. To accommodate subsequent modular operations, the output layer dimension of the backbone network is set to a fixed value. , serving as the baseline dimension for the entire feature space.

[0047] 1.2 Construction of Task-Specific Adapter Module For the distillation Each teacher model task is instantiated in parallel within the student model. Each adapter is a separate, task-specific module. These adapters are designed as lightweight neural network architectures (e.g., bottleneck structures consisting of dimensionality reduction layers, non-linear activation layers, and dimensionality increase layers), with a much smaller number of parameters than the shared backbone network, to ensure efficient storage and computation when the model is expanded.

[0048] Each adapter corresponds to a specific teacher task. Its function is to capture "feature information" belonging to a specific task that the backbone network fails to cover. The adapter module receives the same input as the backbone network or intermediate layer features from the backbone network, and outputs a task-specific feature vector through independent parameter transformation, denoted as... .

[0049] To enable subsequent fusion, the adapter is forced to constrain its output dimension to share common features with the backbone. Dimensions Strictly maintain consistency.

[0050] 1.3 Feature Fusion Mechanism Based on Additive Residuals Define the final feature output logic of the student model for a specific task. This invention adopts an additive fusion strategy of "common features + characteristic residuals" rather than simple channel concatenation. For the first... For each task, the system will share the feature vector output by the backbone. With the corresponding first The feature vector output by each adapter The features are added together to generate the final fused features for this task. .

[0051] Its mathematical expression is as follows: .

[0052] This fusion mechanism establishes the core logic of this invention in terms of physical architecture: (Total ability) is determined by (General capabilities) and (Dedicated supplementary capabilities) together constitute this. This design not only facilitates subsequent constraints through the loss function. and The orthogonality of the system provides direct operands and also supports flexible invocation during the inference phase—that is, by switching between different... Plugins can be used on the same trunk. This generates completely different task outputs.

[0053] Step 2: Use multiple teacher models to jointly distill and train the decoupled student model. During the training process, apply feature decoupling constraints to the common features output by the shared backbone network and the task-specific features output by the task-specific adapter to obtain the trained decoupled student model. Multi-teacher joint distillation and feature alignment training introduce multiple teacher models to jointly guide student models. Common knowledge and task-specific information contained in the teacher models are mapped to the shared backbone and feature modules of the student models, respectively. Through a collaborative distillation mechanism, the student model integrates multi-task capabilities within a single framework, while avoiding mutual interference and performance degradation between different tasks.

[0054] In practical implementation, the core of this step lies in establishing a three-in-one training mechanism that includes "commonality extraction, feature alignment, and spatial decoupling." During the data input phase, training samples are simultaneously input into all teacher models and the student models to be trained. Unlike traditional methods that only focus on fitting the final output, this invention utilizes the output distribution characteristics of multi-source teachers to guide the knowledge flow within the student model. By constructing a composite loss function, it forces the student model to automatically complete the physical routing and hierarchical storage of common and feature knowledge during the training process.

[0055] 2.1 Extraction of common characteristics and core guidance among multiple teachers To avoid the problem of shared backbone learning conflicting information or task biases among teachers due to simple mean distillation, this invention further introduces a backbone guidance strategy based on a "teacher consistency subspace". During each forward propagation, the system simultaneously acquires the feature or Logits representations output by n teacher models for the same input sample. .

[0056] First, statistical aggregation is performed on the outputs from multiple teachers to calculate their consensus center: ; This consensus center characterizes the average cognitive position of different teachers regarding the input sample. Furthermore, to characterize the divergence structure among teachers across various feature dimensions, a teacher divergence covariance matrix is ​​introduced: ; The eigendirection of the covariance matrix reflects the degree of inconsistency in teacher outputs. To address the aforementioned divergence structure, this invention constructs a consensus projection operator: ; Where I is the identity matrix, Represents trace operation. To represent the numerical stability term, set it to a small constant, such as 10. 6. This operator is used to suppress feature directions with significant teacher disagreement, retaining only the stable subspace components with teacher consensus. Subsequently, the student models share the backbone output features. Project onto the teacher consensus subspace, and align the teacher consensus center within this space to construct the backbone commonality distillation loss: ; By minimizing this loss function, the shared backbone network is optimized and updated only within the teacher consistency subspace, thereby avoiding contamination of the backbone network by the specific bias of a single teacher or cross-task conflict information, and improving its cross-task general representation ability and generalization stability.

[0057] 2.2 Construction of Orthogonal Decoupling Constraints in Feature Space like Figure 3 As shown, in the multi-teacher joint distillation framework, although the shared backbone network and task adapter are physically separated through structural design, structural partitioning alone cannot completely avoid redundant representations at the semantic feature level. Without explicit constraints, the task adapter may repeatedly model common directions already covered by the backbone, thereby reducing feature utilization efficiency and affecting overall generalization ability.

[0058] To address this, this invention introduces an orthogonal decoupling constraint mechanism based on the geometric relationships of the feature space, aimed at reducing feature overlap between the task adapter and the shared backbone. The core idea is that the shared backbone is responsible for learning common representations across tasks, while the task adapter only supplements residual information not covered by the backbone.

[0059] Assume a shared backbone for input samples The output is , No. The output of each task branch after adapter mapping is: ; in Indicates the first Intermediate features of each task branch Indicates the corresponding adapter Transformation function.

[0060] Geometrically, adapter features It can be considered as a mixture of "common redundancy components" and "task residual components". To eliminate the redundancy, this invention does not directly maximize the feature angle, but instead explicitly calculates... In sharing backbone features The magnitude of the projection component in the direction is determined, and the square norm of the projection component is minimized, thereby forcing the adapter features to gradually migrate to the orthogonal complement space of the backbone features.

[0061] Based on this, the orthogonal decoupling loss in the feature space is constructed as follows: ; in, Indicates shared backbone features and the first The inner product between the outputs of each adapter is used to measure the strength of their correlation in the feature space; The scale factor represents the core characteristics and is used to eliminate the instability caused by scale changes. To prevent smooth terms with a denominator of zero.

[0062] This loss term is geometrically equivalent to minimizing the squared length of the projection vector of the adapter features onto the backbone direction: ; This inhibits the adapter from accumulating redundant expressions in the feature directions already occupied by the trunk.

[0063] Unlike traditional orthogonal constraints based on cosine similarity, this method uses projection component norm constraints for decoupling. When the adapter outputs... When the overall amplitude is small, its projection component in the main direction is naturally small, and the constraint effect is weakened accordingly; however, when the adapters overlap significantly in the main subspace, the penalty term will be significantly enhanced. Therefore, this constraint manifests as a soft orthogonality constraint, which can adaptively focus the optimization on the feature regions with "significant overlap" while avoiding excessive intervention in weak noise.

[0064] Through this orthogonal decoupling constraint of the feature space, the system can effectively remove redundant information between the shared backbone and the task adapter without suppressing the ability to express task characteristics. This allows the shared backbone to focus on cross-task commonality modeling, while the adapter is only responsible for supplementing task-related residual features.

[0065] 2.3 Optimization of Combined Distillation Objectives and Parameter Iteration like Figure 4 As shown, based on the established common guidance and decoupling constraints, and combined with the performance requirements for specific tasks, the final joint optimization objective is constructed. The system introduces a total distillation loss ( That is, to calculate the final fusion output of the student model. (Composed of the superposition of backbone features and adapter features) and the corresponding teacher model The difference between the outputs. This loss ensures that the combination of "backbone + adapter" can fully reproduce the teacher's professional skills.

[0066] Finally, the overall distillation loss, backbone commonality loss, and feature orthogonal decoupling loss are linearly combined using weighted coefficients to form the global total loss function. Using the backpropagation algorithm, the system calculates the gradient based on this total loss and simultaneously updates the shared backbone parameters and the parameters of each adapter. After training convergence, the student model achieves the ideal decoupling state where "the backbone stores high generalization commonality, and the adapters store high specificity residuals," laying a solid parameter foundation for subsequent plug-in expansion.

[0067] Its overall optimization objective logic is as follows: ; in, , , The self-defined balancing weight hyperparameters for each loss term.

[0068] Step 3: Perform plug-in inference. In the inference phase, the trained decoupled student model dynamically calls the corresponding task-specific adapter according to the task identifier, and processes the input data together with the shared backbone network to obtain the inference result corresponding to the task identifier. The plug-in inference and deployment update features allow for joint inference during the input task phase, selecting appropriate feature plug-ins and the shared backbone based on the input task type, enabling flexible invocation of multi-task capabilities. It also supports dynamic loading and version updates of plug-ins, allowing the model to continuously evolve in the deployment environment and adapt to different application scenarios and business needs.

[0069] In practical implementation, this step involves deploying the trained decoupled model to a real-world application environment (such as an edge computing device or a cloud inference server) and executing inference tasks. Based on the "backbone-adapter" architecture built in the preceding steps, this invention no longer runs a large set of multiple models during the inference phase. Instead, it adopts a dynamic calling mode of "base always resident + plug-in on-demand loading". The system flexibly schedules the shared backbone and specific task adapters to combine according to the specific task instructions input from the outside. While ensuring the accuracy of multi-task processing, it greatly reduces the system's memory usage and computing resource consumption during runtime, achieving true lightweightness and flexibility.

[0070] Dynamic routing and feature synthesis based on task identifiers During inference execution, the system first receives input data (e.g., an image to be detected) and a specific task identifier (Task ID, indicating the specific task to be performed, such as "vehicle detection" or "lane segmentation"). The shared backbone network, acting as a general feature extractor, performs forward computation on the input data to generate a fixed backbone common feature vector. .

[0071] Subsequently, based on the received task identifier, the system indexes and activates the corresponding task from the adapter library. Each task adapter module receives backbone features or raw input, calculates and outputs residual feature vectors specific to that task. At this point, the system keeps adapters for other non-target tasks in a dormant or unloaded state to avoid unnecessary computation. Finally, the feature synthesis logic, consistent with the training phase, is executed to synthesize the common features of the backbone. With the currently active adapter features Element-wise addition is performed to obtain the final task characteristics. The results are then output through the corresponding task prediction header.

[0072] Its reasoning logic can be expressed as: ; Storage-compute separation and dynamic loading strategies in resource-constrained environments For deployment environments with limited memory or video memory resources (such as mobile devices and embedded chips), this invention adopts a "deployment-memory separation" strategy. The parameters of the shared backbone network are configured to reside in memory, serving as the foundational service base for all tasks; while each task-specific adapter is stored as an independent file on external storage media (such as Flash or hard disk).

[0073] When the system needs to switch task scenarios (e.g., from task A to task B), the control logic does not need to unload or reload the massive backbone network. It only needs to release the task A adapter parameters currently in memory and quickly read and load the lightweight adapter parameters for task B from external storage. This "hot-swappable" dynamic loading mechanism enables a single device to support polling processing of large-scale heterogeneous tasks with a very low memory budget, significantly improving the task throughput and adaptability of edge intelligent systems.

[0074] Step 4: Perform plug-in incremental expansion. When a new teacher model is connected to introduce a new task, in response to the connection of the new teacher model, the shared backbone network remains unchanged, and a new task-specific adapter corresponding to the new teacher model is added and trained to achieve incremental expansion of the model's capabilities.

[0075] With the introduction of new teachers and the expansion of plug-in capabilities, when a new teacher task is introduced, the shared backbone structure of the student model remains stable, and corresponding feature plug-in modules are built only for the new task. Through lightweight alignment and distillation training with the new teacher model, the knowledge of the new task is injected into the plug-in modules and mounted onto the student model, achieving incremental expansion of the model's capabilities without having to retrain the entire original task.

[0076] In specific implementation, such as Figure 5 As shown, this step primarily addresses the challenge of introducing new tasks during the model's lifecycle management. When business scenarios change, requiring deployed student models to acquire entirely new capabilities (i.e., integrating a new teacher model), this invention departs from the traditional full-data hybrid retraining model. Instead, it employs a "frozen backbone + incremental adaptation" strategy based on the already constructed decoupled architecture. This strategy fully leverages the generalization capabilities of the shared backbone network, confining the learning process for new tasks within the newly added lightweight adapter. This mathematically eliminates any overwriting or interference with existing task knowledge (i.e., "catastrophic forgetting"), achieving extremely low-cost model capability evolution.

[0077] 4.1 Main Parameter Freezing and New Adapter Initialization When initiating the new task expansion process, the parameters of the already trained global shared backbone network in the student model are first frozen. This means that during subsequent training for the new task, gradient updates of the backbone network weights are completely blocked, and their mapping relationship in the feature space remains constant. This operation is based on the premise that, after multi-teacher joint distillation in step 2, the backbone network has extracted sufficiently robust cross-task common features, which are sufficient to serve as the underlying support for the new task.

[0078] At the same time, based on the definition of the new task, the system instantiates a new task-specific adapter module (NewAdapter). This adapter inherits the network structure configuration of the standard adapter (such as input / output dimensions, bottleneck layer ratio, etc.) and is initialized randomly or with a specific distribution, ready to carry the specific knowledge brought by the new teacher.

[0079] 4.2 Incremental Distillation Training Based on Orthogonal Constraints The training data for the new task is simultaneously input into both the new teacher model and the student model containing the frozen backbone. During training, the system performs forward propagation: the frozen backbone network outputs fixed common features. The new adapter being trained outputs task characteristics to be optimized. The two results are added together to generate a new task prediction result.

[0080] At this stage, the focus is on constructing and optimizing a hybrid loss function consisting of two parts: (1) New task distillation loss: Calculate the difference between the joint output and the output of the new teacher model, forcing the new adapter to quickly fit the business logic of the new task; (2) Incremental orthogonal decoupling loss: Although the backbone parameters are fixed, the system still needs to calculate and The orthogonal decoupling loss is calculated and backpropagated to the new adapter. This step forces the new adapter to actively avoid the existing characteristic directions of the main branch when searching for the optimal solution, and only search for parameters in the "residual subspace" orthogonal to the main branch.

[0081] Through this constrained optimization process, the new adapter is trained to store only the "pure incremental parts" of the new teacher's knowledge that cannot be interpreted by the backbone network.

[0082] 4.3 Seamless integration of capabilities and updates to the plugin library Once the new adapter training converges, the system independently serializes and stores its parameter files, and registers them in the model's "adapter plugin library," thus officially launching the capability. Since the parameters of the shared backbone network remain unchanged throughout the expansion process, and the adapter files for the original tasks were not used in training, the inference results for all original tasks (Task 1 to Task n) are mathematically guaranteed to be completely consistent with those before the expansion. This mechanism achieves true "lossless expansion," enabling the system to continuously add new adapters as business needs evolve. Plugins linearly increase model skills without the risk of model performance degradation or revalidating old features.

[0083] The multi-teacher joint distillation and plug-in model extension method provided in this embodiment constructs a student model architecture that decouples a shared backbone network from task-specific adapters. It introduces a backbone common distillation mechanism based on teacher consensus projection, enabling the backbone network to effectively extract cross-task common knowledge from multiple heterogeneous teacher models. This avoids feature aliasing and gradient conflicts caused by differences in teacher outputs in traditional multi-teacher distillation, significantly improving the generalization ability and training stability of the student model's underlying representations. During training, an orthogonal decoupling constraint is applied to the feature space. By minimizing the projection components of adapter features onto the backbone feature direction, each task adapter is forced to learn only residual information linearly independent of the backbone, fundamentally eliminating feature redundancy between the backbone and adapters. This system achieves efficient parameter utilization. Simultaneously, the soft orthogonal mechanism retains the degrees of freedom required for the adapter to express task specificity while ensuring a clear geometric separation between common and specific features, laying a structural foundation for subsequent plug-in extensions. During the inference phase, the system dynamically loads the corresponding lightweight adapter based on the task identifier, and outputs the result after fusing it with the shared features of the resident backbone. In resource-constrained environments, the storage-computation separation strategy significantly reduces runtime memory usage and enhances multi-task processing capabilities on the edge side. When introducing new tasks, incremental training is performed by freezing the backbone and adding new adapters. Orthogonal constraints ensure that the new adapter only supplements residual knowledge not covered by the backbone, thus mathematically guaranteeing zero performance regression of old tasks and achieving low-cost, continuous, lossless evolution.

[0084] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multi-teacher joint distillation and plug-in model expansion, characterized in that, include: Step 1: Construct a decoupled student model, wherein the decoupled student model includes a shared backbone network and multiple task-specific adapters corresponding to multiple teacher tasks; Step 2: Use multiple teacher models to jointly distill and train the decoupled student model. During the training process, apply feature decoupling constraints to the common features output by the shared backbone network and the task-specific features output by the task-specific adapter to obtain the trained decoupled student model. Step 3: Perform plug-in inference. In the inference phase, the trained decoupled student model dynamically calls the corresponding task-specific adapter according to the task identifier, and processes the input data together with the shared backbone network to obtain the inference result corresponding to the task identifier. Step 4: Perform plug-in incremental expansion. When a new teacher model is connected to introduce a new task, in response to the connection of the new teacher model, the shared backbone network remains unchanged, and a new task-specific adapter corresponding to the new teacher model is added and trained to achieve incremental expansion of the model's capabilities.

2. The method according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Construct a neural network with deep feature extraction capabilities as a shared backbone network to extract common features from the input data. ; Step 1.2: For multiple teacher tasks, construct lightweight task-specific adapters to extract the task-specific features of each teacher task. ; Step 1.3: Generate the final fusion feature for the i-th teacher task using an additive fusion method. in, and The feature dimensions remain consistent.

3. The method according to claim 1, characterized in that, The steps for conducting combined distillation training include: Based on the outputs of multiple teacher models on the same input sample, a backbone supervision signal is constructed to represent multi-teacher consensus knowledge. By constraining the feature representation of the shared backbone network with the backbone supervision signal, the shared backbone network can focus on learning general common knowledge across tasks.

4. The method according to claim 3, characterized in that, The step of constructing the backbone supervision signal for representing multi-teacher consensus knowledge includes: Calculate the consensus center and divergence covariance matrix output by multiple teacher models, where the expression for the consensus center is: in, This represents the total number of teacher models introduced. The index number represents the teacher model, and its value is a natural number from 1 to n. This represents the feature vector output by the i-th teacher model for the same input sample x; The expression for the divergence covariance matrix is: Among them, superscript This represents the matrix transpose operation; Constructing a consensus projection operator based on the divergence covariance matrix in, It is the identity matrix. Represents trace operation. To prevent numerical stability constants with a denominator of zero; Based on the consensus projection operator, the output of the shared backbone network will be... After projecting onto the teacher consensus subspace and aligning it with the consensus center, a common distillation loss is constructed: 。 5. The method according to claim 4, characterized in that, The feature decoupling constraint is specifically a feature space orthogonal decoupling constraint, which achieves orthogonal decoupling between the two in the feature space by minimizing the projection components of the task characteristic features output by the task-specific adapter onto the common feature direction output by the shared backbone network.

6. The method according to claim 5, characterized in that, The feature decoupling loss function corresponding to the orthogonal decoupling constraint in the feature space is: in, Indicates shared backbone features and the first The inner product between the outputs of each task-specific adapter is used to measure the strength of their correlation in the feature space; The scale factor represents the core characteristics and is used to eliminate the instability caused by scale changes.

7. The method according to claim 6, characterized in that, The joint distillation training employs a joint loss function to optimize the decoupled student model, wherein the expression for the joint loss function is as follows: in, This represents the overall distillation loss used to constrain the consistency between the student model's final output and the teacher model's output. , , To balance the weights.

8. The method according to claim 1, characterized in that, The step of dynamically invoking the corresponding task-specific adapter based on the task identifier includes: In resource-constrained deployment environments, parameters of the shared backbone network are kept in memory, while parameters of multiple task-specific adapters are stored in separate files on external storage media. Based on the task identifier of the input data, dynamically load the corresponding target task-specific adapter parameters from the external storage medium into memory; The input data is processed sequentially through a shared backbone network and a loaded task-specific adapter, and the inference result is obtained by fusing the output features of the two.

9. The method according to claim 1, characterized in that, The steps of adding and training a new task-specific adapter include: Freeze the parameters of the already trained shared backbone network; Instantiate a new task-specific adapter corresponding to the new teacher model; Using training data from the new task, and with the output of the new teacher model as supervision, a new task-specific adapter is trained, and during the training process, the output of the new task-specific adapter and the output of the shared backbone network are kept in compliance with the feature decoupling constraint. The parameters of the newly trained task-specific adapter are stored independently and registered in the adapter plugin library.

Images