Multi-task intensive prediction method and system based on task condition adapter
By freezing features, extracting the backbone encoder and introducing task-specific prompts, combining the task condition adapter of the channel and the spatial attention module, the problem of large amount of parameters and long training time in multi-task intensive prediction is solved, and more efficient resource utilization and performance is achieved optimization.
Patent Information
- Application Number
- CN202411174660.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-06
AI Technical Summary
The existing multi-task intensive prediction methods have problems such as large amount of parameters, long training time and large memory consumption, which makes it difficult to apply the model in resource-constrained environments, and the training process is time-consuming, which reduces the research efficiency.
Using a multi-task intensive prediction method based on task condition adapter, the backbone encoder is extracted by freezing features and introducing learnable task-specific prompts to achieve flexible switching between multi-tasks, reduce the number of parameters and computing resource consumption, and pass through channels and space. Attention module enhances the effectiveness of the adapter.
Compared with traditional methods, it achieves fewer training parameters, shorter training time and lower memory consumption, while improving task feature expression capabilities and model performance.
Smart Images

Figure CN119091282A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multi-task prediction method in the field of artificial intelligence technology, and in particular to a multi-task intensive prediction method and system based on a task condition adapter. Background Art
[0002] Multi-task dense prediction is crucial in the field of computer vision. It allows the model to simultaneously handle multiple tasks such as semantic segmentation, depth estimation, and edge detection, improving the depth and accuracy of scene understanding. Multi-task dense prediction improves model efficiency, reduces computing resource consumption, and enhances the robustness and flexibility of the model in practical applications such as autonomous driving and medical imaging by sharing network parameters and learning complementary task features.
[0003] Existing technologies in the field of multi-task intensive prediction have many shortcomings, mainly including large number of parameters, long training time and high memory consumption. Traditional multi-task intensive prediction methods design independent modules to handle the relationship between each pair of tasks, resulting in an exponential increase in the number of parameters as the number of tasks increases, greatly increasing the complexity of the model. In addition, in order to train these huge networks, a large amount of computing resources and storage space are required, which not only limits the application of the model in resource-constrained environments, but also increases the cost of research and development. Long training time is another key issue. Due to the large number of model parameters, a large amount of data and iterations are required to optimize, which makes the training process time-consuming. Long training time not only reduces research efficiency, but also increases the time cost of waiting for model training to complete. At the same time, the large number of parameters and complex network structure also lead to high memory consumption, which is particularly evident when dealing with large-scale datasets or performing high-resolution image analysis. On memory-constrained systems, this may make the training process impossible or inefficient. Summary of the invention
[0004] In order to solve the problems existing in the background technology, the present invention proposes a multi-task intensive prediction method and system based on task condition adapter. The present invention first adopts the training and reasoning method of task condition, that is, inputs an image and the task category to be predicted, outputs the corresponding intensive prediction result, introduces learnable task-specific prompts, and enables the network to flexibly switch between multiple tasks, thereby achieving the purpose of saving the number of parameters and computing resources.
[0005] In a first aspect, the present invention provides a multi-task intensive prediction method based on a task condition adapter, comprising:
[0006] Get the input image and the type of task to be performed;
[0007] Extracting multi-scale feature maps of an input image using a feature extraction backbone encoder combined with an adapter;
[0008] According to the type of task to be performed, the multi-scale feature map is decoded using a task decoder combined with an adapter to obtain a decoding result;
[0009] Multiple task-specific output heads process the decoding results according to the task type and output the prediction results of the corresponding tasks.
[0010] During the training process, the multi-scale dense feature extraction network in the feature extraction backbone encoder combined with the adapter is in a frozen state, and the task-conditional adapter is used to modulate each stage of the decoder during the decoding process of the task decoder combined with the adapter to strengthen the task-specific decoding process.
[0011] The feature extraction backbone encoder combined with the adapter includes a connected multi-scale dense feature extraction network and a multi-scale adapter layer. For each scale of the Transformer layer in the multi-scale dense feature extraction network, each Transformer layer is connected to the corresponding adapter layer, and each adapter layer includes multiple adapters. The output of the residual branch of each residual block in the Transformer layer of the current scale is used as the input of the corresponding adapter in the adapter layer. The output of the first adapter is added to the input of the adapter layer of the current scale as the first intermediate feature of the current scale. The output of each adapter is added to the previous intermediate feature to generate the next intermediate feature. The output of the Transformer layer of the current scale is added to the output of the adapter layer and then passed through a normalization layer to obtain a feature map of the current scale.
[0012] The adapter includes a linear layer, a convolutional block attention module and a task condition module. The first linear layer is used to reduce the dimension of the input features of the adapter, and then the convolutional block attention module is used to adapt the channel dimension and spatial dimension of the reduced features; then the task condition module performs task-specific enhancement modulation on the adapted features according to the task-specific coding to obtain task-specific coding features; finally, the second linear layer is used to increase the dimension of the task-specific features to obtain multi-scale image features modulated by the adapter.
[0013] The task condition module comprises a third and a fourth linear layer, wherein the third linear layer first normalizes the weight vector according to the task specific coding and the output of the convolutional block attention module, and the fourth linear layer learns the second normalized weight vector according to the task specific coding, thereby completing the task specific coding e t Modulate the output y″ of the convolutional block attention module.
[0014] Each of the output heads includes a convolution module, and the channel dimension of the convolution module is determined by the task type.
[0015] In a second aspect, the present invention further provides a multi-task intensive prediction system based on a task condition adapter, comprising:
[0016] The image and task type acquisition module is configured to: acquire images and task types;
[0017] The image preprocessing module is configured to: normalize the image pixel values so that the image adapts to the network input, and encode it into image features using a block coding network, and also add image position coding to represent pixel position information;
[0018] The task-specific coding module is configured to: select a correct task-specific coding for subsequent feature modulation according to an input task type;
[0019] The task-specific feature extraction module is configured to: extract multi-scale features of an input image according to the task-specific encoding;
[0020] The task condition modulation decoding module is configured to: add the task condition adapter to the existing multi-scale decoder, adapt the channel and spatial dimensions of the features in the decoding stage, and decode the features according to the task specific coding modulation to obtain the task specific features of the final output;
[0021] The task-specific output module is configured to: use the task-specific output convolution module to adapt the channel dimension of the task-specific features to obtain a dense prediction result.
[0022] In the task-specific feature extraction module, a frozen multi-scale dense feature extraction network is connected to a task condition adapter in parallel to extract multi-scale features of an input image.
[0023] The task condition adapter is configured to: use the convolutional block attention module to adapt the multi-scale features to the channel and spatial dimensions, and use the task condition module to further modulate the adapted features according to task-specific encoding to obtain better task-specific features.
[0024] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multi-task intensive prediction method based on a task condition adapter are implemented.
[0025] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multi-task intensive prediction described in the first aspect.
[0026] The present invention first extracts multi-scale feature maps of input images based on the deep learning feature extraction backbone encoder network Swin Transformer. On this basis, based on the channel and spatial attention mechanism, the present invention proposes a multi-task intensive prediction method based on task condition adapters, and connects it in parallel with the feature extraction backbone encoder network with frozen parameters, thereby realizing efficient extraction of frozen backbone encoder network parameters and further saving the number of parameters and computing resources. In the decoding stage, the present invention serially connects the task condition adapter to each decoder layer to achieve more direct and effective task-specific feature modulation. The method in the present invention performs unified end-to-end training on the adapter and the decoder, ensuring the effective update of the model parameters and the optimization effect of the task performance; through the method in the present invention, compared with the traditional multi-task intensive prediction method, it can be achieved with fewer training parameters, shorter training time and lower memory consumption while having comparable or even better performance.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] 1. The present invention adopts a task-condition training method. Compared with the traditional multi-task intensive prediction training method, this method allows the multi-task intensive prediction network to share more task-common parameters between different tasks. In addition, compared with most task-condition methods, the present invention shares most of the neural network parameters between different tasks, thereby achieving the purpose of saving the number of training parameters.
[0029] 2. The task condition adapter proposed in the present invention, on the one hand, further saves the number of training parameters by freezing the feature extraction backbone network. On the other hand, compared with the traditional adapter method, the present invention connects the adapter and the backbone network in parallel, which makes it possible for the gradient of back propagation during network training to not pass through the frozen backbone network. This not only reduces the number of training parameters, but also further reduces space occupancy and training time.
[0030] 3. Compared with the traditional adapter method, the task condition adapter proposed in the present invention combines the channel and spatial attention modules to enhance the effectiveness of the adapter in dense prediction tasks. At the same time, it adds a task condition module to perform task-specific modulation on the features according to the task-specific encoding, further enhancing the task feature expression capability.
[0031] 4. The present invention applies the adapter to both the encoder and the decoder to enhance the multi-task adaptation capability of the entire network, and performs unified end-to-end training on the adapter and the decoder, thereby ensuring the effective updating of the model parameters and the optimization of the task performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.
[0033] Figure 1 This is a flow chart of Embodiment 1 of the present invention;
[0034] Figure 2 This is a schematic diagram of the overall network structure of Embodiment 1 of the present invention;
[0035] Figure 3 This is a diagram showing the relationship between tasks revealed by the task-specific coding in Example 1 of the present invention;
[0036] Figure 4 This is a schematic diagram of a task condition adapter according to Embodiment 1 of the present invention;
[0037] Figure 5 This is a diagram showing the effect of the image features of Example 1 of the present invention being adaptively modulated by various parts of the task condition adapter;
[0038] Figure 6 This is a schematic diagram of the connection between the frozen backbone feature extraction network and the task condition adapter module of Example 1 of the present invention;
[0039] Figure 7 A schematic diagram of a decoder network adapted to the task conditions of embodiment 1 of the present invention;
[0040] Figure 8 This is a diagram of the final multi-task prediction effect of Example 1 of the present invention. DETAILED DESCRIPTION
[0041] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0042] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0043] The goal of multi-task intensive prediction is to simultaneously process multiple tasks that require pixel-by-pixel prediction of images, such as semantic segmentation, depth estimation, normal vector estimation, edge detection, etc. By sharing part or all of the parameters of the network, or complementing the relevant information between different tasks, the multi-task intensive prediction method can usually save training parameters and make more efficient use of data and computing resources. It is often used in autonomous driving, robot navigation, virtual reality and augmented reality technology to achieve more accurate and efficient visual task processing. The multi-task intensive prediction method involved in the present invention is introduced below through an embodiment.
[0044] Example 1
[0045] like Figure 1 As shown, the present invention proposes a multi-task intensive prediction method based on a task condition adapter, comprising:
[0046] Get the monocular input image and the execution task type;
[0047] In this embodiment, the process of obtaining a monocular input image and executing a task type is as follows:
[0048] For each dense prediction task, a different, learnable task-specific encoding is assigned to each task type.
[0049] A feature extraction backbone encoder combined with an adapter is used to extract multi-scale feature maps of a monocular input image;
[0050] According to the type of task to be performed, the multi-scale feature map is decoded using a task decoder combined with an adapter to obtain a decoding result;
[0051] The decoding process of the task decoder combined with the adapter is:
[0052] Decode the encoder’s output features using the Swin Transformer block;
[0053] Using a task-conditioned adapter to modulate the decoded features according to the task-specific encoding, high-quality task-specific decoded features are obtained;
[0054] The task-specific decoding features are upsampled using the upsampling module to obtain higher resolution decoding features.
[0055] Connect the high-resolution decoded features with the encoder output features of the corresponding resolution to obtain the next layer of decoded input;
[0056] The output features of each layer of the decoder are fused at multiple scales to obtain the decoded image features. The output heads corresponding to multiple execution tasks process the decoding results according to the task type and output the prediction results of the corresponding tasks. Each output head includes a convolution module, and the channel dimension of the convolution module is determined by the task type.
[0057] During the training process, the multi-scale dense feature extraction network in the feature extraction backbone encoder combined with the adapter is in a frozen state, that is, each Transformer layer is in a frozen state, and the task decoder combined with the adapter uses the task-conditional adapter to modulate each stage of the decoder during the decoding process to strengthen the task-specific decoding process.
[0058] like Figure 6As shown, the feature extraction backbone encoder combined with the adapter includes a connected multi-scale dense feature extraction network and a multi-scale adapter layer, the task encoding is used as the input of each adapter, for each scale Transformer layer in the multi-scale dense feature extraction network, each Transformer layer is connected to the corresponding adapter layer, each adapter layer includes multiple adapters, the output of the residual branch of each residual block in the Transformer layer of the current scale is used as the input of the corresponding adapter in the adapter layer, each residual block corresponds to the adapter one by one, the output of the first adapter is added to the input of the adapter layer of the current scale (that is, the input image or the result of the last intermediate feature downsampling of the previous scale) as the first intermediate feature of the current scale, the output of each adapter is added to the previous intermediate feature to generate the next intermediate feature, that is, each residual block is connected to the corresponding adapter in parallel, the output of the Transformer layer of the current scale (that is, the output of the last residual block) is added to the output of the adapter layer (that is, the last intermediate feature) and then passed through the normalization layer to obtain the feature map of the current scale.
[0059] In the task decoder combined with the adapter, each residual block corresponds to the adapter one by one, and the output of the residual branch of each residual block is used as the final output of the residual branch after passing through the adapter, that is, serial connection.
[0060] like Figure 4 As shown, the adapter is a task-conditional adapter, which includes a linear layer, a convolutional block attention module and a task-conditional module. The input features of the adapter are reduced in dimension by using a trainable first linear layer, and then the convolutional block attention module is used to adapt the channel dimension and spatial dimension of the reduced features. Specifically, the reduced-dimensional features are used as the input of the channel attention layer, the output of the channel attention layer is used as the input of the spatial attention layer, and the result of the jump connection between the input of the spatial attention layer and its output is used as the adapted features; then the task-conditional module performs task-specific enhanced modulation on the adapted features according to the task-specific coding to obtain high-quality task-specific coding features, and the task-conditional module is connected to the second linear layer after the activation layer; finally, the trainable second linear layer is used to increase the dimension of the task-specific features to obtain multi-scale image features modulated by the adapter, that is, the output of the adapter.
[0061] The multi-scale dense feature extraction network is a universal multi-scale feature extraction backbone network. It uses a large-scale pre-training dataset to train the backbone network with image classification as the goal, freezes the pre-trained backbone network, uses the images in the target multi-task dense prediction training set as input images, and uses the pre-trained backbone network to extract the corresponding multi-scale features.
[0062] The multi-task dense prediction method is described in detail by applying the method to the public multi-task dense prediction dataset images; the specific scheme is as follows:
[0063] like Figure 2 As shown, the overall network structure of this embodiment is composed of a multi-scale encoding-decoding network. In order to solve the problem of too many training parameters in the existing multi-task dense prediction network, this embodiment adopts a task-conditional training method, that is, input an image and the type of task to be performed, and output the corresponding task prediction result graph. By sharing most of the network parameters between tasks, this embodiment reduces the number of parameters that need to be trained.
[0064] First, the input image and the type of task to be performed are obtained. Specifically, the existing technology is used for image acquisition, and the corresponding learnable task-specific encoding is selected according to the type of task to be performed. The task-specific encoding of each task is t are all 128-dimensional feature column vectors, where t represents the task type. The correlation between task-specific encodings of different tasks is as follows Figure 3 As shown, the figure shows that the task-specific encoding proposed in this embodiment can effectively model different tasks, and the correlation between the encodings can also reflect the relationship between different tasks.
[0065] like Figure 2 As shown, the frozen feature extraction backbone network in this embodiment is composed of a SwinTransformer network without the classification layer. For an input image I of size H×W, the image preprocessing module first uses an image encoding network to encode it into a size The image code is then added to the relative position code of the same size to introduce pixel position information into the code. After that, the image code is input into the frozen Swin Transformer backbone coding network to extract Features of four scales.
[0066] The adapter module A in the existing method consists of a feature dimension reduction module, a GeLU nonlinear module and a feature dimension increase module, namely:
[0067] A(x)=U(GeLU(D(x)))+x (1)
[0068] Where x is the input feature of n×k dimensions, n is the spatial dimension after flattening, and k is the channel dimension; D() is k×r, and GeLU(·) is the GeLU nonlinear module.
[0069] dimensional feature dimensionality reduction module, while U() is an r×k dimensional feature dimensionality increase module, and r is the kernel dimension of the adapter after dimensionality reduction.
[0070] The task condition adapter proposed in this embodiment has two main functions: first, it efficiently extracts and utilizes the features of the frozen backbone encoder network to obtain feature maps suitable for various dense prediction tasks; second, it modulates the output features of the network to adapt them to different dense prediction tasks. Figure 4 As shown, in addition to the existing modules, this embodiment additionally introduces a convolutional block attention module and a task condition module into the adapter.
[0071] For dense prediction tasks, both channel dimension features and spatial dimension features are crucial to prediction accuracy. However, existing adapters proposed for dense prediction tasks only focus on channel dimension features, ignoring the important role of spatial features in dense prediction tasks. Therefore, the convolutional block attention module K is introduced in this embodiment. The convolutional block attention module consists of a channel attention module and a spatial attention module. They use average pooling and maximum pooling in spatial and channel dimensions to aggregate information in two dimensions, and finally diffuse the channel and spatial dimension attention information into the original features by multiplying their weight matrices with the corresponding elements of the feature matrix. Formally, let r l is the kernel dimension of the lth layer, H l , W l are the height and width of the input feature map, respectively, and the input is r l ×H l ×W l The convolutional block attention module can be expressed as:
[0072]
[0073] Among them, M c Yes l ×1×1 dimensional channel attention map, M s Yes l ×H l ×W l dimensional spatial attention map, It is the multiplication of the corresponding elements of the matrix. Channel attention map M c It can be calculated by the following formula:
[0074] M c (y) = σ(MLP(AvgPool c (y))+MLP(MaxPool c (y))) (4)
[0075] Among them, AvgPool c (·) is spatial average pooling, MaxPool c (·) is the spatial maximum pooling.
[0076] The channel attention map Ms It can be calculated by the following formula:
[0077] M s (y) = σ(f 7×7 ([AvgPool s (y); MaxPool s (y)])) (5)
[0078] Where σ(·) represents the sigmoid activation function, f 7×7 represents a convolutional layer with a kernel size of 7×7, [·;·] represents matrix connections along the channel dimension, c and s represent pooling layers in the channel and spatial dimensions, respectively, MLP(·) is a multi-layer perceptron, AvgPool s (·) is channel average pooling, MaxPool s (·) is the channel maximum pooling. After the convolution block attention module, the feature integrates the attention information of the channel and spatial dimensions and is input into the task-specific module.
[0079] In this embodiment, the task condition module is responsible for modulating the adapter features according to the task type. The input of the task condition module is the output y″ of the convolutional block attention module, whose dimension is r l ×H l ×W l , and a 128-dimensional column vector task-specific encoding e t , which represents the task t being executed. The task condition module TCM uses two l ×128-dimensional linear layer W γ , W β To learn two normalized weight vectors, we can encode e according to the task specific t Modulate the output y″ of the convolutional block attention module:
[0080]
[0081] The task condition module enables the adapter parameters to switch between different tasks, so that the network can better extract task-specific features and improve the network's comprehensive understanding of the environment. The entire adapter can be written as:
[0082] A(y,e t )=U(GeLU(TCM(K(D(y))),e t ))+y (7)
[0083] Where K is the convolutional block attention module introduced above. In this embodiment, the task condition adapter is added to both the encoder and the decoder, so that the entire network can switch between different tasks efficiently and economically. In this embodiment, the effect of image features being adapted and modulated by various parts of the task condition adapter is shown in the figure below. Figure 5 shown.
[0084] Figure 6 The figure shows the connection between the frozen backbone feature extraction network and the task condition adapter module in this embodiment. The existing methods usually insert the adapter module directly into the Transformer block and connect the adapter to the backbone network in series. However, the encoder in this embodiment adopts a parallel connection adapter structure, such as Figure 6 The adapter in this embodiment extracts features from each attention module and fully connected module of the backbone feature extraction network and encodes them according to the task-specific encoding t The features of the backbone network are adapted and modulated using the convolutional block attention module and task-specific module mentioned above to increase the adaptability of the features to various tasks. The task-specific adaptation features output by each adapter will then be added to a feature path parallel to the frozen backbone network and summed with the output features of the frozen backbone network to mimic the residual connection between the adapter and the backbone network module in the existing method. This parallel path will serve as a "fast path" for gradient backpropagation. During the gradient backpropagation process, only the gradient of the parameters on this parallel "fast path" needs to be calculated, which avoids calculating the gradient of the huge frozen backbone encoder parameters during training. This method further reduces storage usage and training time on the basis of reducing the number of training parameters. The output of the adapter module is used as the final multi-scale feature output of the encoder after passing through the normalization layer independent of each scale.
[0085] Figure 7 The network structure of the task-specific decoder in this embodiment is shown. The decoder in this embodiment consists of 4 stages, and each stage is used to process the output features of the encoder at different scales. Each stage is connected by a block expansion module, which is used to upsample the small-scale features and fuse them with the large-scale features of the next layer to achieve the purpose of multi-scale feature fusion. Unlike the backbone encoder, the decoder in this embodiment does not have a pre-trained model, so the decoder needs to be fine-tuned. Therefore, the decoder does not need a gradient backpropagation "fast path" to shorten the gradient backpropagation path. In addition, using a parallel adapter structure in the decoder stage that requires fine-tuning will weaken the modulation effect of the adapter on the decoder, resulting in suboptimal information reconstruction results in the decoder output. So, Figure 7As shown, the decoder of this embodiment adopts the same structure as the existing adapter method, that is, the adapter is serially connected to the decoder network, and the adapter is directly inserted into the Transformer block of the decoder, and serially connected after each attention module and fully connected module. This connection method can modulate the layers of the decoder more directly and finely, thereby generating higher quality task-specific features based on task-specific information.
[0086] After upsampling the output features to the original image size of H×W, this embodiment inputs them into the task-specific independent convolution layer according to the execution task category to obtain the final output prediction image with dimensions of K×H×W, where K is the task-specific output dimension. Figure 8 As shown, the prediction results of this embodiment are significantly better than those of the traditional multi-task dense prediction network TSNs in terms of texture details and object edges, and are comparable to the existing optimal full-parameter training network Taskprompter.
[0087] In view of the shortcomings of existing multi-task intensive prediction methods such as a large number of training parameters, large storage space occupation, and long training time, this embodiment proposes a multi-task intensive prediction method based on a task condition adapter, designs a task condition adapter based on channel and spatial attention, and applies it to the encoder and decoder at the same time, sharing most of the network parameters between different tasks. At the same time, the backbone encoder network is frozen, further reducing the number of training parameters. In addition, the adapter is connected in parallel with the encoder during the encoding stage, and the purpose of saving computing consumption and training time is achieved by truncating the back propagation of the gradient in the frozen backbone network. Compared with the existing methods, this embodiment can achieve multi-task intensive prediction results that are equivalent to or even more accurate than the full-parameter training network with a very small amount of training parameters, computing consumption, and training time due to the structural improvement of the adapter and the innovative adapter connection method.
[0088] Example 2
[0089] This embodiment provides a multi-task intensive prediction system based on a task condition adapter, including:
[0090] The image and task type acquisition module is configured to: acquire images and task types;
[0091] The image preprocessing module is configured to: normalize the image pixel values so that the image adapts to the network input, and encode it into image features using a block coding network, and also add image position encoding to represent pixel position information;
[0092] The task-specific encoding module is configured to: select a correct task-specific encoding for subsequent feature modulation according to an input task type;
[0093] The task-specific feature extraction module is configured to: extract multi-scale features of the input image according to the task-specific encoding;
[0094] The decoding module of task-condition modulation is configured as follows: based on the existing multi-scale decoder, a task-condition adapter is added to adapt the channel and spatial dimensions of the features in the decoding stage, and the decoding features are modulated according to the task-specific coding to obtain the task-specific features of the final output;
[0095] The task-specific output module is configured to: use the task-specific output convolution module to adapt the channel dimension of the task-specific features to obtain a dense prediction result.
[0096] In the task-specific feature extraction module, the frozen multi-scale dense feature extraction network is connected to the task conditional adapter in parallel to extract multi-scale features of the input image.
[0097] The task condition adapter is configured to: use the convolutional block attention module to adapt the multi-scale features to the channel and spatial dimensions, and use the task condition module to further modulate the adapted features according to the task-specific encoding to obtain better task-specific features.
[0098] Example 3
[0099] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the multi-task intensive prediction method based on the task condition adapter in the first aspect are implemented.
[0100] Example 4
[0101] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the multi-task intensive prediction in the first aspect are implemented.
[0102] The above is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.
Claims
1. A multi-task intensive prediction method based on task condition adapter, characterized in that: include: Get the input image and the type of task to be performed; Extracting multi-scale feature maps of an input image using a feature extraction backbone encoder combined with an adapter; According to the type of task to be performed, the multi-scale feature map is decoded using a task decoder combined with an adapter to obtain a decoding result; Multiple task-specific output heads process the decoding results according to the task type and output the prediction results of the corresponding tasks.
2. A multi-task intensive prediction method based on task condition adapter according to claim 1, characterized in that: During the training process, the multi-scale dense feature extraction network in the feature extraction backbone encoder combined with the adapter is in a frozen state, and the task-conditional adapter is used to modulate each stage of the decoder during the decoding process of the task decoder combined with the adapter to strengthen the task-specific decoding process.
3. The multi-task intensive prediction method based on task condition adapter according to claim 1, characterized in that: The feature extraction backbone encoder combined with the adapter includes a connected multi-scale dense feature extraction network and a multi-scale adapter layer. For each scale of the Transformer layer in the multi-scale dense feature extraction network, each Transformer layer is connected to the corresponding adapter layer, and each adapter layer includes multiple adapters. The output of the residual branch of each residual block in the Transformer layer of the current scale is used as the input of the corresponding adapter in the adapter layer. The output of the first adapter is added to the input of the adapter layer of the current scale as the first intermediate feature of the current scale. The output of each adapter is added to the previous intermediate feature to generate the next intermediate feature. The output of the Transformer layer of the current scale is added to the output of the adapter layer and then passed through a normalization layer to obtain a feature map of the current scale.
4. The multi-task intensive prediction method based on task condition adapter according to claim 1, characterized in that: The adapter includes a linear layer, a convolutional block attention module and a task condition module. The first linear layer is used to reduce the dimension of the input features of the adapter, and then the convolutional block attention module is used to adapt the channel dimension and spatial dimension of the reduced features; then the task condition module performs task-specific enhancement modulation on the adapted features according to the task-specific coding to obtain task-specific coding features; finally, the second linear layer is used to increase the dimension of the task-specific features to obtain multi-scale image features modulated by the adapter.
5. The multi-task intensive prediction method based on task condition adapter according to claim 4 is characterized in that: The task condition module comprises a third and a fourth linear layer, wherein the third linear layer first normalizes the weight vector according to the task specific coding and the output of the convolutional block attention module, and the fourth linear layer learns the second normalized weight vector according to the task specific coding, thereby completing the task specific coding e t Modulate the output y″ of the convolutional block attention module.
6. The multi-task intensive prediction method based on task condition adapter according to claim 1, characterized in that: Each of the output heads includes a convolution module, and the channel dimension of the convolution module is determined by the task type.
7. A multi-task intensive prediction system based on task condition adapter, characterized in that: include: The image and task type acquisition module is configured to: acquire images and task types; The image preprocessing module is configured to: normalize the image pixel values so that the image adapts to the network input, and encode it into image features using a block coding network, and also add image position coding to represent pixel position information; The task-specific coding module is configured to: select a correct task-specific coding for subsequent feature modulation according to an input task type; The task-specific feature extraction module is configured to: extract multi-scale features of an input image according to the task-specific encoding; The task condition modulation decoding module is configured to: add the task condition adapter to the existing multi-scale decoder, adapt the channel and spatial dimensions of the features in the decoding stage, and decode the features according to the task specific coding modulation to obtain the task specific features of the final output; The task-specific output module is configured to: use the task-specific output convolution module to adapt the channel dimension of the task-specific features to obtain a dense prediction result.
8. The multi-task intensive prediction system based on task condition adapter according to claim 7, characterized in that: In the task-specific feature extraction module, a frozen multi-scale dense feature extraction network is connected to a task condition adapter in parallel to extract multi-scale features of an input image.
9. The multi-task intensive prediction system based on task condition adapter according to claim 7, characterized in that: The task condition adapter is configured to: use the convolutional block attention module to adapt the multi-scale features to the channel and spatial dimensions, and use the task condition module to further modulate the adapted features according to task-specific encoding to obtain better task-specific features.
10. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a multi-task intensive prediction method based on a task condition adapter as described in any one of claims 1 to 5 are implemented.
