Multi-granularity task adaptive sharing multi-task collaborative adaptation method
By constructing a multi-granularity task information combination tree and a task attention module, the optimization difficulties and performance instability caused by differences between tasks in multi-task learning are solved, achieving efficient sharing and accurate adaptation of information between tasks, and improving the performance and stability of multi-task learning.
Patent Information
- Application Number
- CN202511559642.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-29
- Publication Date
- 2026-02-10
AI Technical Summary
In multi-task learning, especially in the field of computer vision, the large differences between tasks lead to optimization difficulties and performance instability. Existing technologies are unable to effectively share and balance knowledge information from different tasks, resulting in frequent negative transfer phenomena.
By constructing a multi-task collaborative adaptation method with multi-granularity task adaptive sharing, an information combination tree is established, and a task attention module is used to extract and fuse feature information at different granularity levels. Intra-layer and inter-layer soft prompts are designed to alleviate the optimization difficulties and performance instability caused by differences between tasks.
It effectively improves the efficiency of information interaction and adaptation accuracy between tasks, significantly enhances the generalization performance and collaborative adaptation effect of multi-task learning, and is applicable to a variety of visual perception tasks and pre-trained model structures, with an average accuracy improvement of more than 3%.
Smart Images
Figure CN121503582A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-task learning, and specifically to a multi-task collaborative adaptation method for multi-granularity task adaptive sharing. Background Technology
[0002] Multitask learning is an important technological paradigm in deep learning, aiming to improve the generalization ability of a single task and the overall performance of the model by simultaneously learning multiple related tasks. Especially in deep neural network applications, due to the complexity of the network structure and the large number of parameters, a large amount of labeled data is needed for training to acquire sufficient task-related knowledge. However, data collection conditions for many real-world tasks are stringent or prohibitively expensive, making it difficult to collect sufficiently large datasets for training. This results in insufficient data for a single task to support the model in fully mastering the required knowledge. Therefore, multitask learning was proposed to address the data gap problem by sharing domain knowledge and key features from different tasks and utilizing additional information provided by other tasks.
[0003] However, multi-task learning also faces optimization difficulties and performance instability due to significant differences between tasks. Especially in computer vision (CV), where visual information contains more complex semantic structures, the information differences between different tasks are more pronounced, easily leading to negative transfer—that is, joint training of multiple tasks may perform worse than independent training of a single task. This makes effectively sharing and balancing knowledge information from different tasks one of the core issues in multi-task learning. In current technologies, most multi-task learning research in the vision field focuses on multi-task paradigms with single datasets, paying less attention to multi-task scenarios with more complex inter-task relationships and more relaxed constraints across multiple datasets. How to design more efficient multi-task information sharing and collaboration mechanisms in multi-dataset environments has become an urgent technical challenge. Summary of the Invention
[0004] In view of the above problems, the present invention provides a multi-task collaborative adaptation method with multi-granularity task adaptive sharing. The present invention establishes multi-granularity task sharing feature information by building an information combination tree, and designs a task attention module to extract feature information that is beneficial to specific tasks from the tree. This solves the problems of optimization difficulties and performance instability caused by large differences between tasks in existing multi-task learning, and effectively alleviates the negative transfer phenomenon.
[0005] This invention provides a multi-task collaborative adaptation method with multi-granularity task adaptive sharing, the specific steps of which are as follows: S1. Construct a multi-granularity information combination tree for visual task sharing; S2. Extract combined task feature information at different granularity levels of the multi-granularity information combination tree, and obtain intra-layer soft cues and cross-layer soft cues by performing intra-layer fusion and inter-layer fusion through the task attention module. S3. Obtain the final soft cue based on the intra-layer soft cue and cross-layer soft cue, and concatenate the final soft cue to the input of the visual pre-trained model to obtain the prediction result on the corresponding visual task. Optionally, the visual tasks include video action recognition, semantic segmentation, object detection, and instance segmentation.
[0006] Optionally, the multi-granularity task information combination tree includes multiple levels and nodes, with nodes including non-leaf nodes and leaf nodes; each node includes a set of learnable cues.
[0007] Optionally, the leaf nodes of the multi-granularity task information combination tree are located at the lowest level. Layer, number of leaf nodes and number of tasks Correspondingly; the number of nodes in the remaining layers is half that of the layer below it; the top layer contains only a single node.
[0008] Alternatively, gradient descent can be used to minimize the total loss of the task. The optimization of the multi-granularity information combination tree and task attention module for task sharing is expressed as follows:
[0009] in, This represents the total loss across all tasks; It is the first The loss of each task.
[0010] Optionally, in step 2, at the... In the leaf nodes of a layer, each node independently provides task prompts and stores its own task's feature information. In each of the remaining layers, the task-shared prompts of nodes in the same layer jointly store the shared information of all tasks. The task-shared prompts of nodes in the same layer are aggregated into in-layer soft prompts through the task attention module.
[0011] Optionally, in step 2, all generated intra-layer soft cues and task-independent cues are fused between layers through the task attention module to obtain cross-layer soft cues.
[0012] Optionally, in step 3, the cross-layer soft hints and the task-independent hints are added together to obtain the final soft hint.
[0013] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The multi-task collaborative adaptation method of the present invention, through the collaborative action of the multi-granularity task information combination tree and the task attention module, can effectively integrate and extract multi-granularity features between different tasks, and effectively improve the information interaction efficiency and task adaptation accuracy between different tasks. (2) The task attention module of the present invention automatically judges and quantifies the degree of interest of each task in the information of the tree nodes through task vectors and attention mechanism, implicitly realizes task clustering and information sharing, realizes adaptive learning and optimization of refined task combination information, and effectively alleviates the negative transfer phenomenon. (3) The present invention introduces a task semantic prior information embedding method, which uses a large language model to generate task-specific semantic embeddings as task vector initialization, which can significantly improve the semantic perception ability of task vectors and enhance the adaptive ability of task attention modules, thereby further improving the generalization performance and collaborative adaptation effect of each task in the multi-task learning process. (4) The multi-task collaborative adaptation method of the present invention with multi-granularity task adaptive sharing has wide applicability and can flexibly adapt to a variety of different visual perception tasks and different pre-trained model structures. Compared with the single-task adaptation method, the average accuracy is improved by more than 3%. Attached Figure Description
[0014] Figure 1 This is a schematic diagram of the multi-granularity information combination tree of the present invention; Figure 2 This is a schematic diagram of the task attention module of the present invention. Detailed Implementation
[0015] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.
[0016] A specific embodiment of the present invention, such as Figures 1-2 A multi-task collaborative adaptation method with multi-granularity task adaptive sharing is disclosed. The specific steps are as follows: S1. Construct a multi-granularity information combination tree for task sharing; Specifically, the multi-granularity task information combination tree includes multiple levels and nodes, with nodes including non-leaf nodes and leaf nodes.
[0017] Furthermore, each node includes a set of learnable cues and aggregates multi-task feature information of different granularities at different levels. The cues are shaped as follows: The real parameter matrix is defined as ,in It is a real number. Indicates the length of the prompt. The dimension of the hint is the same as the dimension of the hidden layer of the adapted model.
[0018] In one embodiment of the present invention, the adapter for the multi-task information combination tree is vit-base-patch16-224, the dimension of the hidden layer is 768, and the length of the prompt is 100. Therefore, the dimension of the prompt corresponding to each node in the tree is 100x768.
[0019] Furthermore, the top level of the multi-granularity task information combination tree, i.e., the level where the root node is located, is the [missing information]. A layer contains only a single node. The lowest layer, where leaf nodes reside, is the [layer name missing]. Layer, number of leaf nodes and number of tasks In this context, each leaf node corresponds to a unique task, representing the independent characteristics of that task, which are not shared with other tasks.
[0020] Furthermore, from the multi-granularity task information combination tree, the first... Starting with each layer, the number of nodes in each layer is half the number of nodes in the previous layer. The specific recursive definition is as follows:
[0021] in, Represents the multi-granularity information combination tree. The number of nodes in the layer; Represents the multi-granularity information combination tree. Number of nodes in the layer; This indicates the total number of levels in the task information combination tree.
[0022] Furthermore, The total number of nodes in the multi-granularity task information combination tree is approximately equal to .
[0023] Furthermore, the target task set is as follows: ,in, Indicates the first One task.
[0024] For example, the target task It includes a variety of visual tasks, including but not limited to image classification, video action recognition, semantic segmentation, object detection, and instance segmentation.
[0025] Furthermore, gradient descent is used to minimize the total loss of the task. For the training set Optimize the multi-granularity information combination tree for task sharing and the task attention module, and improve the training set. The set of hints contained in a node on the multi-granularity task information combination tree is expressed as:
[0026] in, This represents the total loss across all tasks; It is the first The loss of each task.
[0027] In one embodiment of the present invention, the task consists of two types of classification tasks, referred to as a classification task group. Specifically, the task group comprises 19 image classification tasks and 1 video behavior recognition task. The multi-granularity task information tree has 6 levels, with each level containing 1, 2, 3, 5, 10, and 19 nodes from top to bottom.
[0028] In one embodiment of the present invention, the task consists of three dense detection tasks, referred to as a dense detection task group.
[0029] Specifically, the task group consists of one semantic segmentation task, one object detection task, and one instance segmentation task. The multi-granularity task information tree has three levels, with each level having 1, 2, and 3 nodes from top to bottom.
[0030] S2. Extract combined task feature information at different granularity levels of the multi-granularity information combination tree, and obtain intra-layer soft cues and cross-layer soft cues by performing intra-layer fusion and inter-layer fusion through the task attention module. Furthermore, the present invention extracts combined task feature information in two stages on the multi-granularity task information combination tree: intra-layer fusion (intra-layer soft hints) and inter-layer fusion (cross-layer soft hints).
[0031] Specifically, a task-independent task attention module is used to perform two weighted fusions of shared task feature information of nodes at different levels in a multi-granularity task information combination tree, both intra-layer and inter-layer, to obtain cross-layer soft cues, thereby achieving task-adaptive feature information extraction and sharing.
[0032] First, in the In each layer's leaf nodes, the prompts independently store the feature information of their respective tasks (task-independent prompts). In each of the remaining layers, the task-shared prompts of nodes within the same layer collectively store shared information on all task features. These task-shared prompts are aggregated into intra-layer soft prompts through the task attention module, expressed as:
[0033] in, Representing the The task in the multi-granularity information combination tree Intra-layer soft hints formed by layer aggregation; Representing the Task attention operations for each task; Represents the multi-granularity information combination tree. The first layer Each prompt includes task characteristic information; Represent real numbers; Indicates the length of the prompt. This indicates that the dimension of the prompt is the same as the dimension of the hidden layer of the adapted model.
[0034] Secondly, the soft suggestions generated within each layer and the first Independent hints for the t-th task in the layer Inter-layer fusion is performed again through the task attention module to obtain cross-layer soft cues. The expression is: .
[0035] Specifically, the task attention module introduces task-independent and learnable task vectors to capture the specific concerns of each task, enabling the accurate extraction and sharing of combined task feature information. The specific concerns of each task are the attention weights of each cue on the multi-granularity task information sharing tree obtained by the task attention module, measured using the dot product attention score of the task vector and the cue. Attention operations are performed between the task vector and the cue input from the task attention module, which can be used for fusion of nodes at the same level or for fusion of nodes across different levels.
[0036] Furthermore, the set of learnable task vectors is ,in, It is a real number, where N represents the number of tasks in the task group, and D represents the dimension of the task vector and the dimension of the hidden layer of the adapted model; the task vector Each task corresponds to a specific task, representing the task's affinity to information in the multi-granularity task information tree.
[0037] For example, if there are 10 tasks in a task group and the dimension of the task vector is 768, then the set of task vectors is a 10x768 learnable parameter matrix.
[0038] Furthermore, in the task attention module, various prompts are input to obtain attention scores, expressed as follows:
[0039] in, Representing the The first prompt Attention scores for each vector; Indicates the first The first prompt The first vector One value; Indicates the first The task vector corresponding to each task; This indicates that the dimension of the prompt is the same as the dimension of the hidden layer of the adapted model; Indicates the length of the prompt sequence.
[0040] Then, SoftMax is used to normalize the attention scores to obtain the attention weights for each set of cues, expressed as:
[0041] in, The input task attention module represents the first... The first prompt Attention weights for each vector.
[0042] Furthermore, the attention weights reflect the degree of attention each task pays to different cues. The task attention module uses these attention weights to weightedly fuse the corresponding cues, resulting in soft cues, expressed as: ;
[0043] in, Indicates a soft prompt The Middle One vector; Indicates the first The first group prompt There are 1 learnable vectors.
[0044] Furthermore, for task vectors that are task-independent and learnable, a task semantic prior information embedding method is used. By leveraging a large language model, the semantic information of the task is injected as prior knowledge into the multi-task collaborative adaptation method, thereby improving the effectiveness and stability of multi-task adaptation.
[0045] This invention employs the aforementioned method based on manually defined task data sources and knowledge extraction to extract natural language descriptions of data sample distribution and task objectives, thereby obtaining task-independent and learnable task vectors. This overcomes the challenge that in multi-task learning, the representation of each task typically consists of two key elements: data sample distribution and task objective. Sharing a feature space requires effectively fusing information from these two aspects. However, due to significant domain shifts between different task datasets in reality, it is difficult to directly generate universally applicable machine features from the data distribution. In its specific implementation, this invention first analyzes and organizes relevant literature on various publicly available datasets involved in the target task, forming natural language knowledge representations for each task from two dimensions: task target knowledge and data scenario knowledge. Subsequently, sentence embedding technology is used to convert these natural language descriptions into corresponding semantic feature embedding representations to obtain task-independent and learnable task vectors.
[0046] S3. Obtain the final soft cue based on the intra-layer soft cue and cross-layer soft cue, and concatenate the final soft cue to the input of the visual pre-trained model to obtain the prediction result on the corresponding visual task. For example, the output of an image classification task corresponds to the prediction result on a visual task as the category of an object; the output of a video action recognition task is the corresponding action category in a video sequence; the output of a semantic segmentation task is the label of each pixel in an image; the output of an object detection task is the category of an object in an image and its spatial location (usually a bounding box); and the instance segmentation task further outputs the precise pixel-level region of each independent instance based on object detection.
[0047] Furthermore, to reduce the loss of task-independent suggestions and increase their weight in the fusion process, this invention adds cross-layer soft suggestions to task-independent suggestions (leaf node suggestions) to obtain the final soft suggestion, expressed as:
[0048] in, Indicates task The final soft tip; Indicates the first Each task has its own task prompt.
[0049] The composite tree of this invention fully considers the problem of negative migration caused by differences between tasks. If only the root node is used for aggregation, negative migration may occur due to large differences between tasks, reducing overall performance. Therefore, this invention adopts a secondary aggregation multi-granularity task information tree to alleviate the negative migration phenomenon by fusing information of different granularities. From top to bottom, the composite tree... The layer nodes aggregate information from all tasks, and all have a focus level of [missing information]. As the node hierarchy increases, the number of nodes grows exponentially, and the attention weights generated by the task attention module at each node become more differentiated, resulting in a gradually finer granularity of information combination.
[0050] Exemplary, this invention uses different training batch sizes for different tasks. In one example of this invention, for the classification task group, the batch size is 32 for image classification and 64 for video action recognition. For the dense detection task group, the batch size is 32 for object detection and semantic segmentation, and 8 for instance segmentation. Within the same task group, all tasks have the same number of training epochs, and the number of iterations in each epoch depends on the number of iterations of the task with the fewest iterations. Specifically, let the tasks in the task group... The batch size is Its training set size is Then the number of iterations in one round is In each iteration, the dataset is randomly and uniformly sampled without replacement.
[0051] In one embodiment of the invention, four pre-trained visual models are adapted, specifically three Transformer-based network structures and one convolutional network structure, including ViT-Base / 16 based on supervised learning pre-training, ViT-MAE / 16 based on self-supervised learning pre-training, the heterogeneous ViT network SWin-Transformer designed using window attention and sliding window, and the convolutional neural network ResNet50.
[0052] In one embodiment of the invention, a cue is inserted at the first layer input of the pre-trained visual model. Specifically, for ViT-type pre-trained visual models, the cue is inserted after the patch embedding layer; for convolutional pre-trained models, the cue is inserted before the first main network layer. Taking ResNet-50 as an example, the first main network layer follows immediately after the initial convolutional and max-pooling layers, and contains multiple basic network layers with residual connections.
[0053] In one embodiment of the present invention, the prompt length setting is shown in Table 1.
[0054] Table 1. Prompt length on different pre-trained models
[0055] In ViT, ViT-MAE, and Swin, the learnable cue is appended to the input token of the first layer. For ResNet50, the learnable cue is directly added to the feature map, and the cue has the same shape as the feature map of the interpolated layer.
[0056] The method described in this invention improves performance on various pre-trained backbone networks and various tasks compared to single-task adaptation.
[0057] Specifically, in one embodiment of this invention, the average accuracy of the classification task group was improved by more than 3%. The performance of both tasks was measured using micro-averaging. The image classification task used the VTAB dataset, which contains 19 tasks; the accuracy in the table is the average accuracy of these 19 tasks. Video action recognition included one task, using the Kinetics-200 dataset.
[0058] The method described in this invention improves performance on various backbone networks and dense detection datasets. Specifically, in one instance of this invention, the average accuracy is improved by more than 3% on the dense detection task group. For both the baseline method and the method described in this invention, mean average precision (mAP) is used for evaluation on object detection and instance segmentation tasks, while mean intersection over union (mIoU) is used for evaluation on semantic segmentation. The average accuracy of the three tasks is selected as the multi-task performance evaluation metric. Object detection includes one task using the COCO dataset; semantic segmentation includes one task using the Cityscapes dataset; and instance segmentation includes one task using the Penn-Fudan dataset.
[0059] To ensure uniform numerical scale across different cues and facilitate fair comparison and fusion, this invention requires all cues to undergo layer normalization (LayerNorm) before being input into the task attention module. Layer normalization eliminates numerical discrepancies caused by scale differences between cues, accurately reflecting the actual correlation between the task vector and the cues. Furthermore, cues with a uniform scale facilitate the interaction and integration of feature information between different tasks during the fusion process.
[0060] The task attention module implicitly clusters similar tasks by calculating the degree of attention each task pays to tree nodes. The closer the attention distributions of tasks are, the more similar the implicitly formed task group information is.
[0061] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment. The specific implementation steps are as follows: This invention employs a method based on human-defined task data sources and knowledge extraction to extract natural language descriptions of data sample distribution and task objectives.
[0062] In its specific implementation, this invention first analyzes and organizes the relevant literature of each public dataset involved in the target task, and forms a natural language knowledge expression for each task from two dimensions: task target knowledge and data scenario knowledge.
[0063] As an example, in one embodiment of the present invention, the natural language knowledge representations for each task are shown in Table 2. Table 2 Natural Language Knowledge Representation for Some Tasks in VTAB
[0064] Subsequently, sentence embedding technology is used to convert these natural language descriptions into corresponding semantic feature embedding representations.
[0065] As an example, in one instance of the present invention, the text-embedding-3-large model published by OpenAI is used as the embedding extraction model to generate embedding vectors for each task.
[0066] The generated task semantic embedding vectors are used to initialize the task vectors in the aforementioned task attention modules. This initialization strategy is based on the following key assumptions: tasks with similar semantics require higher similarity in their cues, and the attention distribution of task attention modules on similar tasks should also be more similar. Therefore, the distance between embedding vectors can effectively measure the semantic similarity between tasks. To quantitatively evaluate the similarity of the embedding representations between tasks...
[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-task collaborative adaptation method with multi-granularity task adaptive sharing, characterized in that, The specific steps are as follows: S1. Construct a multi-granularity information combination tree for visual task sharing; S2. Extract combined task feature information at different granularity levels of the multi-granularity information combination tree, and obtain intra-layer soft cues and cross-layer soft cues by performing intra-layer fusion and inter-layer fusion through the task attention module. S3. Obtain the final soft cue based on the intra-layer soft cue and cross-layer soft cue, and concatenate the final soft cue to the input of the visual pre-trained model to obtain the prediction result on the corresponding visual task.
2. The multi-task collaborative adaptation method according to claim 1, characterized in that, Visual tasks include video action recognition, semantic segmentation, object detection, and instance segmentation.
3. The multi-task collaborative adaptation method according to claim 1, characterized in that, The multi-granularity task information combination tree includes multiple levels and nodes, with nodes including non-leaf nodes and leaf nodes; each node includes a set of learnable cues.
4. The multi-task collaborative adaptation method according to claim 3, characterized in that, The leaf nodes of the multi-granularity task information combination tree are located at the lowest level. Layer, number of leaf nodes and number of tasks Correspondingly; the number of nodes in the remaining layers is half that of the layer below it; the top layer contains only a single node.
5. The multi-task collaborative adaptation method according to claim 1, characterized in that, Gradient descent is used to minimize the total loss of the task. The optimization of the multi-granularity information combination tree and task attention module for task sharing is expressed as follows: in, This represents the total loss across all tasks; It is the first The loss of each task.
6. The multi-task collaborative adaptation method according to claim 4, characterized in that, In step 2, at the In the leaf nodes of a layer, each node independently provides task prompts and stores its own task feature information. In each of the remaining layers, the task-shared prompts of nodes in the same layer jointly store the shared information of all tasks. The task-shared prompts of nodes in the same layer are aggregated into in-layer soft prompts through the task attention module.
7. The multi-task collaborative adaptation method according to claim 4, characterized in that, In step 2, all generated intra-layer soft cues and task-independent cues are fused together through the task attention module to obtain cross-layer soft cues.
8. The multi-task collaborative adaptation method according to claim 1, characterized in that, In step 3, the cross-layer soft hints and the task-independent hints are added together to obtain the final soft hint.