A multi-modal continual learning based multi-task method and system
By employing a multimodal continuous learning approach, combined with a decoupled low-rank adaptation mechanism and a structurally self-tuning sparse expert network, the adaptability problem of multimodal models under task evolution and complex heterogeneous modalities is solved, achieving efficient multi-task processing.
Patent Information
- Application Number
- CN202610499939.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-15
- Publication Date
- 2026-07-14
AI Technical Summary
Existing multimodal large models struggle to retain old knowledge and adapt to new tasks when faced with dynamic evolution of task distribution and complex heterogeneous modes. They suffer from poor modal adaptability, continuous parameter growth, and a lack of dynamic structural adjustment capabilities.
We employ a multimodal continuous learning approach, combining a decoupled low-rank adaptation mechanism with a structurally self-adjusting sparse expert network. Through sparse hybrid expert layers and a decoupled low-rank adaptation mechanism, we dynamically optimize the model structure, prune cold experts and grow new experts, freeze historical task directions, and only update the magnitude of new tasks.
It enables efficient and continuous learning under complex, heterogeneous, and multimodal data conditions, completing classification, question answering, and generation tasks, improving the model's adaptability and efficiency, and avoiding catastrophic forgetting.
Smart Images
Figure CN122389936A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal data processing technology, and in particular to a multi-task method and system based on multimodal continuous learning. Background Technology
[0002] With the development of artificial intelligence, breakthroughs have been made in technologies such as computer vision, speech recognition, and natural language processing. Large models based on deep learning have demonstrated superior performance in multiple unimodal tasks. In recent years, multimodal learning has gradually become a research hotspot. Its core goal is to integrate and understand information from different modalities (such as images, text, speech, and video) to improve the model's cognitive ability in complex scenarios. Especially in practical applications such as dialogue systems, intelligent question answering, and medical assisted diagnosis, joint modeling and reasoning of multi-source heterogeneous modal data such as images, text, and speech are often involved, and traditional unimodal systems are difficult to meet the requirements.
[0003] Meanwhile, the task distribution in practical applications is not static and fixed, but evolves continuously over time, with task types and modal inputs constantly expanding. For example, an educational robot may initially only need to handle image classification tasks, but later it may also need to support functions such as image-text question answering, speech recognition, and multi-turn dialogue. This places a demand on the model's ability to learn continuously, meaning that the model must dynamically adapt to new tasks and modalities without forgetting old knowledge, while maintaining a consistent structure and efficient reasoning capabilities.
[0004] Existing multimodal large models, such as CLIP, BLIP, and Flamingo, have made some progress in unified representation learning and modality alignment, but they mostly adopt static model structures and offline training methods, lacking the ability to adapt to task evolution and complex heterogeneous modalities. Meanwhile, current research on continuous learning largely focuses on single-modal domains (especially image processing), primarily mitigating catastrophic forgetting through sample replay, regularization constraints, or incremental parameter expansion.
[0005] Although existing methods have achieved some success, they are difficult to directly extend to multimodal and multi-task scenarios. They often suffer from problems such as poor adaptability to complex heterogeneous multimodal scenarios, the forgetting of old tasks often accompanying the training of new tasks, the continuous growth of model parameters in multi-task and multimodal scenarios, and the lack of ability to dynamically adjust the structure according to the complexity of tasks. Summary of the Invention
[0006] The purpose of this invention is to provide a multi-task method and system based on multimodal continuous learning, which integrates a decoupled low-rank adaptation mechanism and a structured self-tuning sparse expert network, enabling the model to complete tasks such as classification, question answering and generation in a highly efficient and continuous learning manner under complex and heterogeneous multimodal data conditions, thus overcoming the bottlenecks of poor modality adaptability and single task in traditional multimodal models.
[0007] To achieve the above objectives, this invention provides a multi-task method based on multimodal continuous learning, comprising the following steps: S1. Acquire multimodal data and construct a multimodal dataset; S2. Map the multimodal dataset obtained in S1 to the unified language model representation space to obtain a unified semantic feature representation of the multimodal data; S3. Input the unified semantic feature representation of the multimodal data obtained in S2 into the sparse hybrid expert layer, and then have the router perform token-level sparse activation before... Each expert performs forward computation through self-attention and expert feedforward neural networks; S4. During the continuous learning process, the activation frequency and average gradient norm of experts are statistically analyzed regularly, and long-term low-activity cold experts are automatically pruned or new experts are grown to dynamically optimize the structure of the self-adjusting sparse hybrid expert layer. S5. Adopt a decoupled low-rank adaptation mechanism to freeze the historical task direction and only update the learning direction and magnitude for new tasks. S6. Output the prediction results for classification, question answering, or generation tasks in a unified manner.
[0008] Preferably, S1 is as follows: We obtain raw modal data (image, speech, and text) from publicly available multimodal datasets; and construct a multimodal dataset from this raw data, as shown below: ; in, A dataset representing multimodal data; This indicates that the modality is a subset of images. This represents a subset of samples whose modality is text. This represents a subset of samples whose modality is speech.
[0009] Preferably, the unified semantic feature representation of the multimodal data in S2 is as follows: ; in, A unified semantic feature representation for multimodal data. To unify and integrate the feature dimensions; A built-in modality alignment module for multimodal visual language models.
[0010] Preferably, S3 is as follows: S31. Through multi-head self-attention and normalization, the unified semantic feature representation obtained in S2 is processed. Feature processing is performed to generate intermediate features after multi-head self-attention enhancement; as shown below: ; in, Indicates the first Intermediate features after multi-head self-attention enhancement; This indicates the focus of the bulls; It is a normalization layer; Indicates the first The output features of the layer; S32. Perform a linear transformation on the intermediate features generated in S31 after multi-head self-attention enhancement using the router weight matrix, and then apply the transformation based on the token-level sparse activation conditions. Several experts participated in the process, and [the following was obtained] The selection probability vectors corresponding to each expert are shown below: ; in, Indicates the first A vector of the selection probabilities of each expert; This means selecting the expert with the highest activation probability from all experts. indivual; Normalization softmax operate; This represents the router weight matrix, with dimension 1. , For the number of experts, For feature dimensions; S33, Activated Each expert processes the intermediate features after multi-head self-attention enhancement through a single feedforward neural network to generate a forward output feature. This forward output feature is then weighted and summed using the expert's selection probability vector as weights to obtain a fused result. The fusion output features after individual experts are shown below: ; in, Indicates fusion The fusion output characteristics after individual experts; Indicates the first Forward output features of an expert; Indicates the first The expert's forward output features for the input features are specifically represented as follows: , Indicates the first The weight matrix of a feedforward neural network for an expert. Represents the linear rectification activation function; S34, Integration The fusion output features after the first expert are processed through residual connections and layer normalization to obtain the first... The final output features of the first layer are also used as the input features of the next layer, and so on, until the process reaches the [number]th layer. The last layer is the layer from which we obtain the first layer. The final output features of the layer, where the first layer... The final output features of the layer are shown below: ; in, Indicates the first The final output features of the layer.
[0011] Preferably, S4 is as follows: S41. Calculate the activation frequency and gradient saturation of selected experts in a fixed number of rounds to construct a comprehensive activation score index, as shown below: ; in, Experts The comprehensive activation score index; The weighting coefficient adjusts the trade-off between activation frequency and gradient contribution; Experts Activation frequency in past rounds; Experts The average gradient norm; S42. Set a scoring threshold and perform dynamic structural adjustments on the experts to dynamically optimize the self-adjusting sparse hybrid expert layer. (1) If the first k An expert Below the threshold If so, it is determined to be a cold expert, and cold expert trimming is performed; (2) If the first k An expert Above the threshold This generates new experts, thereby enabling automatic adjustment of the sparse hybrid expert layer. The weight matrix of the fully connected layer of the new expert is initialized as follows: ; in, This represents the weight matrix of the initial fully connected layer for new experts; Indicates inheritance; Indicates a Gaussian distribution. This represents the mean; Indicates variance.
[0012] Preferably, S5 is as follows: S51, regarding the first For each expert, the task is to construct a decoupled low-rank update weight matrix, as shown below: ; in, Experts In the mission The resulting decoupled low-rank update weight matrix; This indicates the direction learned from historical tasks; Indicates the current task that can be learned. and the scope of historical tasks; Indicates a new direction for the current task; S52. Learn the direction and scope of the current task, adjust the scope of historical tasks, and output expert results. The fine-tuned forward output features are shown below: ; in, Experts Fine-tuning the forward output features; Original experts The original weight matrix has been frozen; Original experts The forward output features are specifically represented as follows: ; Indicates the current task The cumulative decoupling low-rank adaptation updates for all historical tasks are specifically represented as follows: ; S53. By inputting the fine-tuned forward output features into the self-adjusting sparse hybrid expert layer of the structure and completing the remaining forward inference process, the final fine-tuned output features are obtained.
[0013] Preferably, S6 is as follows: The final fine-tuned output features are fed into the multi-task head to extract task-specific information and output the corresponding classification, question-answering, or generation task results, as shown below: ; in, Represents a multi-tasking head Learnable parameters; Represents the result of the task; This represents the final fine-tuned output characteristics.
[0014] A multi-task system based on multimodal continuous learning includes: a multimodal data acquisition module for acquiring raw multimodal data of images, videos, and text from a public multimodal dataset; The modality alignment module receives the raw multimodal data output by the multimodal data acquisition module, maps the raw multimodal data to the unified language model representation space, and outputs a unified semantic feature representation of the multimodal data. The self-tuning sparse hybrid expert routing network module includes a sparse hybrid expert layer for receiving the unified semantic feature representation output by the modality alignment module, and then routing it through a router based on token-level sparse activation. Each expert performs forward computation on the unified semantic feature representation; at the same time, during the training process, the activation frequency and average gradient norm of the experts are monitored, cold experts are pruned and new experts are introduced to form an adaptive expert structure. The decoupled low-rank adaptation mechanism fine-tuning module is used to receive the output features of the sparse hybrid expert layer after the core parameters of the expert structure of the self-adjusting sparse hybrid expert routing network module are frozen; by decoupling the direction of freezing the low-rank adaptation mechanism, the module fine-tunes the parameters for the new task, updates the magnitude, and outputs the fine-tuned features adapted to the new task. The unified output module is used to input the fine-tuned features into the multi-task head built into the unified output module, and uniformly output the prediction results of multimodal classification, question answering or generation tasks.
[0015] Therefore, this invention adopts the above-mentioned multi-task method and system based on multimodal continuous learning, which integrates the decoupled low-rank adaptation mechanism and the structured self-tuning sparse expert network, so that the model can complete tasks such as classification, question answering and generation in an efficient and continuous learning manner under complex and heterogeneous multimodal data conditions, breaking through the bottleneck of poor modal adaptability and single task of traditional multimodal models.
[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0017] Figure 1 This is a flowchart of a multi-task method based on multimodal continuous learning according to the present invention; Figure 2 This is a structural diagram of a multi-task system based on multimodal continuous learning according to an embodiment of the present invention. Detailed Implementation
[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0019] This invention discloses a multi-task method based on multimodal continuous learning, comprising the following steps: S1. Acquire multimodal data and construct a multimodal dataset; S2. Map the multimodal dataset obtained in S1 to the unified language model representation space to obtain a unified semantic feature representation of the multimodal data; S3. Input the unified semantic feature representation of the multimodal data obtained in S2 into the sparse hybrid expert layer, and then have the router perform token-level sparse activation before... Each expert performs forward computation through self-attention and expert feedforward neural networks; S4. During the continuous learning process, the activation frequency and average gradient norm of experts are statistically analyzed regularly, and long-term low-activity cold experts are automatically pruned or new experts are grown to dynamically optimize the structure of the self-adjusting sparse hybrid expert layer. S5. Adopt a decoupled low-rank adaptation mechanism to freeze the historical task direction and only update the learning direction and magnitude for new tasks. S6. Output the prediction results for classification, question answering, or generation tasks in a unified manner.
[0020] Example like Figure 1 As shown, the present invention provides a multi-task method based on multimodal continuous learning, comprising the following steps: S1. Obtain multimodal data and construct a multimodal dataset, the details of which are as follows: S11. Obtain raw modal data such as images, speech, and text from public multimodal datasets; among which, public multimodal datasets include CMU-MOSEI, ImageBind, VALOR, etc.
[0021] S12. The raw modal data such as images, speech, and text obtained in S11 are used to construct a multimodal dataset, as shown below: ; in, A dataset representing multimodal data; This indicates that the modality is a subset of images. This represents a subset of samples whose modality is text. This represents a subset of samples whose modality is speech.
[0022] S2. Map the multimodal dataset obtained in S1 to a unified language model representation space to obtain a unified semantic feature representation of the multimodal data, achieving cross-modal alignment. The unified semantic feature representation of the multimodal data is as follows: ; in, A unified semantic feature representation for multimodal data; To unify and integrate the feature dimensions; A built-in modality alignment module for multimodal visual language models.
[0023] S3. Input the unified semantic feature representation of the multimodal data obtained in S2 into the sparse hybrid expert layer. The router then performs sparse activation based on the modal features. Each expert performs forward computation through self-attention and expert feedforward neural networks.
[0024] S31. Through multi-head self-attention and normalization, the unified semantic feature representation obtained in S2 is processed. Feature processing is performed to generate intermediate features after multi-head self-attention enhancement, as shown below: ; in, Indicates the first Intermediate features after multi-head self-attention enhancement; This indicates the focus of the bulls; It is a normalization layer; Indicates the first Output characteristics of the layer.
[0025] S32. Perform a linear transformation on the intermediate features generated in S31 after multi-head self-attention enhancement using the router weight matrix, and then apply the transformation based on the token-level sparse activation conditions. Several experts participated in the process, and [the following was obtained] The selection probability vectors corresponding to each expert are shown below: ; in, Indicates the first A vector of the selection probabilities of each expert; This means selecting the expert with the highest activation probability from all experts. indivual; The softmax operation represents normalization. This represents the router weight matrix, with dimension 1. , For the number of experts, For feature dimensions.
[0026] S33, Activated Each expert processes the intermediate features after multi-head self-attention enhancement through a single feedforward neural network to generate forward output features. These forward output features are then weighted and summed using the expert's selection probability vector as weights to obtain a fused result. The fusion output features after individual experts are shown below: ; in, Indicates fusion The fusion output characteristics after individual experts; Indicates the first k Forward output features of an expert; Indicates the first k The expert's forward output features for the input features are specifically represented as follows: ;in, Indicates the first The weight matrix of a feedforward neural network for an expert; This represents the linear rectification activation function.
[0027] S34, Integration The fusion output features after the first expert are processed through residual connections and layer normalization to obtain the first... The final output features of the first layer are also used as the input features of the next layer, and so on, until the process reaches the [number]th layer. The last layer is the layer from which we obtain the first layer. The final output features of the layer, where the first layer... The final output features of the layer are shown below: ; in, Indicates the first The final output features of the layer As input features for the next layer, inference continues with S3 and S4 until the final layer obtains the... The final output features of the layer .
[0028] By using residual connections and layer normalization, we provide a computational foundation for efficient processing of multimodal data while maintaining training stability.
[0029] S4. During the continuous learning process, the activation frequency and average gradient norm of experts are statistically analyzed periodically, and long-term low-activity cold experts or new experts are automatically pruned to dynamically optimize the self-adjusting sparse hybrid expert layer.
[0030] During continuous learning, the sparse hybrid expert layer often encounters issues where selected experts are not used for extended periods or experience congestion when handling certain tasks. Therefore, firstly, the activation frequency and gradient saturation of the selected experts are calculated over a fixed number of rounds to construct a comprehensive activation score index, as shown below: ; in, Experts The comprehensive activation score index; The weighting coefficient adjusts the trade-off between activation frequency and gradient contribution; Experts Activation frequency in past rounds; Experts The average gradient norm.
[0031] Secondly, a scoring threshold is set, and the expert structure is dynamically adjusted to optimize the self-adjusting sparse hybrid expert layer.
[0032] (1) If the first k An expert Below the threshold If the result is negative, it is considered a "cold expert" and "cold expert cutout" will be performed.
[0033] (2) If the first k An expert Above the threshold This generates new experts, thereby enabling automatic adjustment of the sparse hybrid expert layer. The weight matrix of the fully connected layer of the new expert is initialized as follows: ; in, This represents the weight matrix of the initial fully connected layer for new experts; This represents the inheritance coefficient, which controls the ratio of the weight matrix of the fully connected layer to that of the new and old experts. Indicates a Gaussian distribution. Represents the mean; Represents variance.
[0034] S5. A decoupled low-rank adaptation mechanism is adopted to freeze the historical task direction and only update the learning direction and magnitude for new tasks.
[0035] In the fine-tuning phase, to accelerate the training process, all parameters of the multimodal visual language model, including all experts, are frozen, and a decoupled low-rank adaptation mechanism is used to activate each expert. In the fine-tuning phase, direction plays a more critical role than amplitude. Traditional low-rank adaptation mechanisms, with their coupling of amplitude and direction, often update synchronously, leading to catastrophic forgetting. The core idea of the decoupled low-rank adaptation mechanism is to decouple amplitude and direction, fixing the direction learned from historical tasks, and only learning the direction and amplitude for the current task, while adjusting the amplitude of historical tasks to reduce catastrophic forgetting and improve adaptability to complex tasks.
[0036] S51, regarding the first One expert, task Construct the decoupled low-rank update weight matrix as follows: ; in, Experts In the mission The resulting decoupled low-rank update weight matrix; This indicates that the direction learned from historical tasks remains fixed. Indicates the current task that can be learned. t and the scope of historical tasks; This indicates a new direction for the current task.
[0037] The expert layer parameter update process of the decoupled low-rank adaptation mechanism keeps the original weight matrix and the direction of the historical task fixed, and only needs to learn the direction and magnitude of the current task and adjust the magnitude of the historical task.
[0038] S52. Learn the direction and scope of the current task, adjust the scope of historical tasks, and output expert results. k The fine-tuned forward output features are shown below: ; in, Experts k Fine-tuning the forward output features; Original experts k The original weight matrix has been frozen; Original experts k The forward output features are specifically represented as follows: ; Indicates the current task The cumulative decoupling low-rank adaptation updates for all historical tasks are specifically represented as follows: .
[0039] S53. By inputting the fine-tuned forward output features into the self-adjusting sparse hybrid expert layer and completing the remaining forward inference process, the final fine-tuned output features are obtained. .
[0040] S6. Output the prediction results for classification, question answering, or generation tasks in a unified manner.
[0041] The final fine-tuned output characteristics will be Incoming D Multi-tasking head In order to extract task-specific information, in this embodiment, the multi-task head uniformly outputs the corresponding classification, question and answer, or generated task results according to the current task type. The formula for this process is expressed as: ; in, Represents a multi-tasking head Learnable parameters.
[0042] like Figure 2 As shown, the present invention provides a multi-task system based on multimodal continuous learning, comprising: The multimodal data acquisition module is used to acquire raw multimodal data such as images, videos, and text from multimodal datasets.
[0043] The modality alignment module receives the raw multimodal data output by the multimodal data acquisition module, maps the raw multimodal data to the unified language model representation space, and outputs a unified semantic feature representation of the multimodal data.
[0044] The self-tuning sparse hybrid expert routing network module includes a sparse hybrid expert layer for receiving the unified semantic feature representation output by the modality alignment module, and then routing it through a router based on token-level sparse activation. Each expert performs forward computation on the unified semantic feature representation; at the same time, during training, the activation frequency and average gradient norm of the experts are monitored, cold experts are pruned and new experts are introduced to form an adaptive expert structure.
[0045] The decoupled low-rank adaptation mechanism fine-tuning module is used to receive the output features of the sparse hybrid expert layer after the core parameters of the expert structure of the self-adjusting sparse hybrid expert routing network module are frozen. It freezes the historical task direction through the decoupled low-rank adaptation mechanism, fine-tunes the parameters of the new task, updates the magnitude, and outputs the fine-tuned features adapted to the new task.
[0046] The unified output module takes the fine-tuned features as input to the multi-task head of the unified output module and outputs them in a unified manner for tasks such as multimodal classification, question answering, or generation.
[0047] Therefore, this invention adopts the above-mentioned multi-task method and system based on multimodal continuous learning, which integrates the decoupled low-rank adaptation mechanism and the structured self-tuning sparse expert network, so that the model can complete tasks such as classification, question answering and generation in an efficient and continuous learning manner under complex and heterogeneous multimodal data conditions, breaking through the bottleneck of poor modal adaptability and single task of traditional multimodal models.
[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A multi-task method based on multimodal continuous learning, characterized in that, Includes the following steps: S1. Acquire multimodal data and construct a multimodal dataset; S2. Map the multimodal dataset obtained in S1 to the unified language model representation space to obtain a unified semantic feature representation of the multimodal data; S3. Input the unified semantic feature representation of the multimodal data obtained in S2 into the sparse hybrid expert layer, and then have the router perform token-level sparse activation before... Each expert performs forward computation through self-attention and expert feedforward neural networks; S4. During the continuous learning process, the activation frequency and average gradient norm of experts are statistically analyzed regularly, and long-term low-activity cold experts are automatically pruned or new experts are grown to dynamically optimize the structure of the self-adjusting sparse hybrid expert layer. S5. Adopt a decoupled low-rank adaptation mechanism to freeze the historical task direction and only update the learning direction and magnitude for new tasks. S6. Output the prediction results for classification, question answering, or generation tasks in a unified manner.
2. The multi-task method based on multimodal continuous learning according to claim 1, characterized in that, S1 specifically refers to: We obtain raw modal data (image, speech, and text) from publicly available multimodal datasets; and construct a multimodal dataset from this raw data, as shown below: ； in, Represents a multimodal dataset; This indicates that the modality is a subset of images. This represents a subset of samples whose modality is text. This represents a subset of samples whose modality is speech.
3. The multi-task method based on multimodal continuous learning according to claim 2, characterized in that, The unified semantic feature representation of multimodal data in S2 is shown below: ； in, A unified semantic feature representation for multimodal data. This represents the unified and integrated feature dimensions; This refers to the modality alignment module built into the multimodal visual language model.
4. The multi-task method based on multimodal continuous learning according to claim 3, characterized in that, S3 specifically refers to: S31. Through multi-head self-attention and normalization, the unified semantic feature representation obtained in S2 is processed to generate intermediate features enhanced by multi-head self-attention, as shown below: ； in, Indicates the first Intermediate features after multi-head self-attention enhancement; This indicates the focus of the bulls; It is a normalization layer; Indicates the first The output features of the layer; S32. Perform a linear transformation on the intermediate features generated in S31 after multi-head self-attention enhancement using the router weight matrix, and then apply the transformation based on the token-level sparse activation conditions. Several experts participated in the process, and [the following was obtained] The selection probability vectors corresponding to each expert are shown below: ； in, Indicates the first A vector of the selection probabilities of each expert; This means selecting the expert with the highest activation probability from all experts. indivual; Normalization softmax operate; This represents the router weight matrix, with dimension 1. , For the number of experts, For feature dimensions; S33, Activated Each expert processes the intermediate features after multi-head self-attention enhancement through a single feedforward neural network to generate forward output features. These forward output features are then weighted and summed using the expert's selection probability vector as weights to obtain a fused result. The fusion output features after individual experts are shown below: ； in, Indicates fusion The fusion output characteristics after individual experts; Indicates the first Forward output features of an expert; Indicates the first The expert's forward output features for the input features are specifically represented as follows: , Indicates the first The weight matrix of a feedforward neural network for an expert. Represents the linear rectification activation function; S34, Integration The fusion output features after the first expert are processed through residual connections and layer normalization to obtain the first... The final output features of the first layer are also used as the input features of the next layer, and so on, until the process reaches the [number]th layer. The last layer is the layer from which we obtain the first layer. The final output features of the layer, where the first layer... The final output features of the layer are shown below: ； in, Indicates the first The final output features of the layer.
5. A multi-task method based on multimodal continuous learning according to claim 4, characterized in that, S4 specifically refers to: S41. Calculate the activation frequency and gradient saturation of selected experts in a fixed number of rounds to construct a comprehensive activation score index, as shown below: ； in, Experts The comprehensive activation score index; The weighting coefficient adjusts the trade-off between activation frequency and gradient contribution; Experts Activation frequency in past rounds; Experts The average gradient norm; S42. Set a scoring threshold and perform dynamic structural adjustments on the experts to dynamically optimize the self-adjusting sparse hybrid expert layer. (1) If the first An expert Below the threshold If so, it is determined to be a cold expert, and cold expert trimming is performed; (2) If the first An expert Above the threshold Then, a new expert is generated, where the weight matrix of the fully connected layer of the new expert is initialized as follows: ； in, This represents the weight matrix of the initial fully connected layer for new experts; Indicates inheritance; Indicates a Gaussian distribution. This represents the mean; Indicates variance.
6. The multi-task method based on multimodal continuous learning according to claim 5, characterized in that, S5 specifically refers to: S51, regarding the first For each expert and task, a decoupled low-rank update weight matrix is constructed as follows: ； in, Experts In the mission The resulting decoupled low-rank update weight matrix; This indicates the direction learned from historical tasks; This represents the current task t and the historical task range that can be learned; Indicates a new direction for the current task; S52. Learn the direction and scope of the current task, adjust the scope of historical tasks, and output expert results. The fine-tuned forward output features are shown below: ； in, Experts Fine-tuning the forward output features; Original experts The original weight matrix has been frozen; Original experts The forward output features are specifically represented as follows: ; Indicates the current task The cumulative decoupling low-rank adaptation updates for all historical tasks are specifically represented as follows: ; S53. By inputting the fine-tuned forward output features into the self-adjusting sparse hybrid expert layer of the structure and completing the remaining forward inference process, the final fine-tuned output features are obtained.
7. A multi-task method based on multimodal continuous learning according to claim 6, characterized in that, S6 specifically refers to: The final fine-tuned output features are fed into the multi-task head to extract task-specific information and output the corresponding classification, question-answering, or generation task results, as shown below: ； in, Represents a multi-tasking head Learnable parameters; Represents the result of the task; This represents the final fine-tuned output characteristics.
8. A multi-task system based on multimodal continuous learning, characterized in that, The system is used to execute a multi-task method based on multimodal continuous learning as described in any one of claims 1-7, the system comprising: The multimodal data acquisition module is used to acquire raw multimodal data, including images, videos, and text, from publicly available multimodal datasets. The modality alignment module receives the raw multimodal data output by the multimodal data acquisition module, maps the raw multimodal data to the unified language model representation space, and outputs a unified semantic feature representation of the multimodal data. The self-tuning sparse hybrid expert routing network module includes: a sparse hybrid expert layer, which receives the unified semantic feature representation output by the modality alignment module and, through the router of the sparse hybrid expert layer, performs sparse activation based on the token level. Each expert performs forward computation on the unified semantic feature representation; at the same time, during the training process, the activation frequency and average gradient norm of the experts are monitored, cold experts are pruned and new experts are introduced to form an adaptive expert structure. The decoupled low-rank adaptation mechanism fine-tuning module is used to receive the output features of the sparse hybrid expert layer after the core parameters of the expert structure of the self-adjusting sparse hybrid expert routing network module are frozen. It then uses the decoupled low-rank adaptation mechanism to freeze the direction, fine-tunes the parameters for the new task, updates the magnitude, and outputs the fine-tuned features adapted to the new task. The unified output module is used to input the fine-tuned features into the multi-task head built into the unified output module, and uniformly output the prediction results of multimodal classification, question answering or generation tasks.