Image instance segmentation and semantic segmentation task joint optimization method, equipment and medium
By employing coupled backbone networks and affinity-guided sharing strategies in semantic segmentation and instance segmentation tasks, gradient conflicts are mitigated, task sharing parameters are optimized, task performance is improved, and more efficient image segmentation results are achieved.
Patent Information
- Application Number
- CN202511104240.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-07
- Publication Date
- 2025-11-18
Smart Images
Figure CN120976544A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of computer vision. BACKGROUND
[0002] Semantic segmentation and instance segmentation are important tasks in the field of computer vision, and are widely used in the fields of autonomous driving, medical image analysis, video surveillance, etc. For these two tasks, the traditional method is to train and optimize through different neural network architectures, and to use separate models to process each task. However, although semantic segmentation and instance segmentation pursue different final goals, their output results have significant similarities, i.e. assigning class labels to each pixel in the image. Based on this, researchers have proposed a multi-task learning (MTL) framework to share part of the network's parameters in order to improve the performance of both. However, existing multi-task learning methods have the following problems: the coupled learning process of parameter sharing ratio and distribution is unreasonable, and there is a gradient conflict problem. SUMMARY
[0003] The present application is to solve the problem of unreasonable coupled learning process of parameter sharing ratio and distribution in existing multi-task learning methods when facing semantic segmentation and instance segmentation. The present application provides a multi-task learning based image instance segmentation and semantic segmentation task joint optimization method, device and medium.
[0004] The first aspect of the present application provides an image instance segmentation and semantic segmentation task joint optimization method, which adopts two independent coupled backbone networks and two independent heads to respectively perform the instance segmentation task and the semantic segmentation task, and uses affinity-guided sharing strategy to jointly use the two independent coupled backbone networks.
[0005] In one possible design, in the two independent coupled backbone networks, an element-level gradient direction correction algorithm is executed in the back propagation process to alleviate the gradient direction conflict caused by competing task sharing parameters between the instance segmentation task and the semantic segmentation task.
[0006] In one possible design, the affinity-guided sharing strategy includes:
[0007] First stage: calculate the inner product of the gradients between the instance segmentation task and the semantic segmentation task of each linear layer in the coupled backbone network, obtain the affinity value of each linear layer, and then normalize the affinity values of all linear layers to obtain the proportion of task sharing parameters of each linear layer;
[0008] Second stage: introduce a perception score in the channel dimension for each linear layer, and according to the size relationship between the perception score and the threshold, activate the task sharing weight or the task specific weight, so that the coupled backbone network can adaptively determine the task sharing parameter distribution.
[0009] In one possible design, calculating the gradient inner product between the instance segmentation task and the semantic segmentation task of each linear layer in the coupled backbone network to obtain the affinity value of each linear layer includes:
[0010] During backpropagation, the gradients of the instance segmentation task and the semantic segmentation task are calculated respectively, and the obtained gradients are flattened into one-dimensional vectors. The inner product of the corresponding one-dimensional vectors of the instance segmentation task and the semantic segmentation task is calculated to obtain the affinity.
[0011] In one possible design, activating task-shared weights or task-specific weights based on the relationship between the perceived score and the threshold includes:
[0012] When the perceived score is less than or equal to the threshold, the task-shared weight is activated.
[0013] When the perceived score is greater than the threshold, the task-specific weights are activated.
[0014] In one possible design, the element-level gradient direction correction algorithm includes:
[0015] The gradient vectors of the image instance segmentation task and the semantic segmentation task are flattened into one-dimensional vectors;
[0016] Determine whether there is a conflict in the gradient directions of the two tasks based on the one-dimensional gradient vectors of the two tasks;
[0017] When a conflict exists, the weighting coefficients are calculated based on the gradient descent rate, and the gradient vector is then updated.
[0018] In one possible design, the step of calculating the weighting coefficients based on the gradient descent rate and then updating the gradient vector includes:
[0019] Calculate the weight coefficient of each task based on the gradient change rate of each task;
[0020] The corresponding gradient vector is updated using the weight coefficients.
[0021] In one possible design, calculating the weight coefficients for each task based on the gradient change rate of each task includes:
[0022] The expression for the gradient rate of change is:
[0023] ,
[0024] in, For the first The gradient rate of change for each task. and The first Subsequent In the nth iteration the first linear layer of the i-th task the first linear layer of the i-th task the first linear layer of the i-th task denotes the 2-norm;
[0025] The expression of the weight coefficient is:
[0026] ,
[0027] wherein, is the weight coefficient of the i-th task.
[0028] The second aspect of the present application provides an image instance segmentation and semantic segmentation task joint optimization device, which comprises a processor and a memory, and at least one instruction is stored in the memory. The at least one instruction is loaded and executed by the processor to realize the image instance segmentation and semantic segmentation task joint optimization method as described above.
[0029] The third aspect of the present application provides a computer storage medium, wherein at least one instruction is stored in the computer storage medium. The at least one instruction is loaded and executed by the processor to realize the image instance segmentation and semantic segmentation task joint optimization method as described above.
[0030] The beneficial effects of the present application are:
[0031] The image instance segmentation and semantic segmentation task joint optimization method based on multi-task learning constructed by the present application effectively solves the task shared parameter learning and gradient conflict problems in multi-task learning through affinity guidance sharing strategy and fine-grained gradient correction strategy without complicating the network structure, and improves the performance of instance segmentation and semantic segmentation tasks. BRIEF DESCRIPTION OF DRAWINGS
[0032] Fig. 1 FIG. 1 is a structural schematic diagram of an image instance segmentation and semantic segmentation task joint optimization method;
[0033] Fig. 2 FIG. 5 is a schematic diagram of an affinity guidance sharing strategy;
[0034] Fig. 3 FIG. 8 is a schematic diagram of an element-level gradient direction correction algorithm. DETAILED DESCRIPTION
[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0036] For the coupled learning process of unreasonable parameter sharing ratio and distribution: existing parameter sharing schemes can be divided into two kinds, namely layer sharing and channel sharing. Extensive research shows that the channel sharing strategy achieves better performance. However, this method requires the network to simultaneously learn the proportion and distribution of task-shared parameters at each layer, which greatly increases the complexity of the optimization process during training.
[0037] For the gradient conflict problem: since semantic segmentation and instance segmentation are essentially different tasks, the gradient directions of their task-shared parameters will interfere with each other, causing the network to update in a suboptimal direction during optimization, thereby affecting performance. Therefore, how to more effectively optimize the two tasks simultaneously and reduce the gradient conflict during network optimization has become an important challenge in research.
[0038] Therefore, the embodiments of the present application provide an image instance segmentation and semantic segmentation task joint optimization method based on multi-task learning to solve the above problems. Referring to Figs. 1 to 3 To specifically describe the embodiments, the image instance segmentation and semantic segmentation task joint optimization method based on multi-task learning described in the embodiments is applied to image segmentation tasks in computer vision to improve the performance of image instance segmentation and semantic segmentation tasks, which includes:
[0039] As shown in Fig. 1 The joint optimization network is built based on Swin-Transformer, including two coupled backbone networks and two independent heads. The head network of Mask R-CNN is used to perform the instance segmentation task, and the head network of UPerNet is used to perform the semantic segmentation task.
[0040] The input image provided for the instance segmentation and semantic segmentation tasks , the embodiment adopts two parallel backbone networks for processing to generate intermediate feature representations .
[0041] The affinity-guided sharing strategy based on gradient similarity (AGFS) is applied in the backbone network. As shown in Fig. 2 The affinity-guided sharing strategy based on gradient similarity is divided into two stages:
[0042] Phase 1: Executing an affinity-guided sharing strategy based on gradient similarity. The first phase involves learning the parameter sharing ratio, including: calculating the inner product of the gradients between the semantic segmentation task and the instance segmentation task in each linear layer of the backbone network to obtain the affinity value. Then, the affinity layers of all linear layers are normalized to obtain the proportion of shared parameters for each task. Details are as follows:
[0043] For the backbone network with weights of The linear layer calculates the gradients for instance segmentation and semantic segmentation during backpropagation, flattens the gradients into a one-dimensional vector, and calculates the inner product to obtain the affinity. :
[0044] ,
[0045] in, Flattening operation, and Let represent the gradients for instance segmentation and semantic segmentation, respectively. and Representing feature dimension, Indicates the inner product.
[0046] Then the affinity of all linear layers was determined. Normalization to Scope, quantifying the proportion of shared parameters across different task layers. :
[0047] ,
[0048] in, and These represent the minimum and maximum affinity, respectively.
[0049] The second stage: Execute the second stage of AGFS, parameter sharing distribution learning, including: introducing a channel dimension perceptual score for each of the linear layers, transforming the distribution generation problem into a learnable problem, and activating task-shared weights or task-specific weights according to the relationship between the perceptual score and the threshold, so that the network adaptively determines the appropriate task-shared parameter distribution.
[0050] With the first Taking an MLP (Multilayer Perceptron) layer as an example, its original weights are considered as task-shared weights. and define task-specific weights. and perceptual scores Use indicator functions right Process and generate , for activation weight Define based on perception scores With threshold Size relationship, activate task shared weight or task-specific weights :
[0051] ,
[0052] ;
[0053] in, for The first in Number, .
[0054] like Fig. 3 As shown, this embodiment proposes an element-level gradient direction correction algorithm. During backpropagation, the element-level gradient direction correction algorithm (FGRS) is executed to flatten the gradient vectors of semantic segmentation and instance segmentation tasks into one-dimensional vectors. The sign of each element is compared, and the weighting coefficients are calculated based on the gradient descent rate to update conflicting elements. This alleviates gradient direction conflicts between instance segmentation and semantic segmentation tasks caused by competing tasks sharing parameters in an element-level manner.
[0055] For the The 1st FGRS iteration The task is defined as the first... The task-shared parameters for each linear layer are: ,calculate gradient , the gradient Flattened into a one-dimensional vector , .
[0056] This implementation then defines a mask. To determine Does each component in the gradient have a consistent sign across all tasks? That is, for the gradient... The Each component If the gradient components of two tasks are simultaneously or ,but ;otherwise, .
[0057] Therefore, if If so, then it is considered that there is no gradient conflict between the instance segmentation task and the semantic segmentation task. If the gradient signs are opposite between the instance segmentation task and the semantic segmentation task, it is believed that gradient direction conflicts will occur, and the gradient needs to be adjusted.
[0058] When the gradient directions conflict, the embodiment calculates the weight coefficient according to the gradient change rate of the first task to solve the gradient conflict problem, and the calculation formula is as follows:
[0059]
[0060]
[0061] wherein, ||.||2 represents the 2-norm.
[0062] Then, the embodiment uses the gradient of the first task to weight the update:
[0063]
[0064] is the first component of the single-dimensional gradient vector of the first linear layer in the first task in the updated first iteration.
[0065] Finally, the embodiment defines the final gradient of the task-shared parameter as follows:
[0066] wherein, represents converting a one-dimensional vector with a dimension of into a two-dimensional matrix.
[0067]
[0068] The comprehensive optimization framework is trained and tested in the instance segmentation task of the COCO dataset and the semantic segmentation task of the ADE20K dataset, and its performance is evaluated.
[0069] Embodiment
[0070] In the experimental stage, the backbone network of the image instance segmentation and semantic segmentation task joint optimization method based on multi-task learning is initialized using the Swin Transformer weight pre-trained on the ImageNet-1k dataset. The embodiment uses 8 NVIDIA GeForce RTX 3090 GPUs to simultaneously train the instance segmentation and semantic segmentation tasks, with a total of 200,000 iterations.
[0071] In the instance segmentation task, the present embodiment uses the COCO dataset for experiments. The present embodiment processes images using MMDetection and adopts 、 、 、 、 and to evaluate the instance segmentation performance.
[0072] In the semantic segmentation task, the present embodiment uses the ADE20K dataset for experiments. The present embodiment processes images using MMSegmentation and selects and as evaluation indicators.
[0073] The performance comparison of the image instance segmentation and semantic segmentation task joint optimization method based on multi-task learning proposed in the present embodiment with other methods in the COCO dataset and the ADE20K dataset is shown in Table 1 and Table 2.
[0074] Table 1: Performance comparison in instance segmentation dataset
[0075]
[0076] As shown in Table 1, the image instance segmentation and semantic segmentation task joint optimization method based on multi-task learning (COFS) proposed in the present embodiment shows impressive performance. COFS outperforms the baseline Swin-T by (1.0%, 1.0%, 1.1%) in 、 and respectively. In addition, COFS is also higher than Swin-T by (0.9%, 0.9%, 1.2%) in , and respectively, which proves the effectiveness of the decoupled learning stage in simultaneously alleviating the gradient direction conflict. Compared with other multi-task learning methods, the COFS series achieves outstanding accuracy in all indicators. Compared with ViT-MVT-T, COFS-T shows superior performance, with an increase of 3.3%, 4.2%, 4.3%, 2.4%, 3.7% and 2.8% respectively.
[0077] Table 2: Performance comparison in semantic segmentation dataset
[0078]
[0079] As shown in Table 2, COFS achieves competitive performance. Specifically, COFS outperforms Swin-T by (0.9%, 0.9%) in and The metrics were 1.8% and 1.2% higher than the baseline Swin-T, respectively. In addition, despite the ViT-MVT-T integrating additional modules in the backbone network, COFS achieved performance improvements (0.7%, 0.7%) with significantly lower FLOPs.
[0080] Specific implementation two: the image instance segmentation and semantic segmentation task joint optimization equipment described in the embodiment, the image instance segmentation and semantic segmentation task joint optimization equipment includes a processor and a memory, at least one instruction is stored in the memory, the at least one instruction is loaded and executed by the processor to realize the image instance segmentation and semantic segmentation task joint optimization method as described in embodiment one.
[0081] Specific implementation three: a computer storage medium is described in the embodiment, at least one instruction is stored in the computer storage medium, the at least one instruction is loaded and executed by the processor to realize the image instance segmentation and semantic segmentation task joint optimization method as described in embodiment one.
[0082] Although the present application is described herein with reference to particular embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present application. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be devised without departing from the spirit and scope of the application as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways other than the original claims describe. It can also be understood that features described in connection with individual embodiments can be used in other described embodiments.
Claims
1. A joint optimization method for image instance segmentation and semantic segmentation tasks, characterized in that: Two independent coupled backbone networks and two independent heads are used to perform the instance segmentation task and the semantic segmentation task respectively. The two independent coupled backbone networks are joined together using an affinity-guided sharing strategy.
2. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 1, characterized in that, In the two independent coupled backbone networks, an element-level gradient direction correction algorithm is executed during backpropagation to alleviate gradient direction conflicts caused by competing tasks sharing parameters between the instance segmentation task and the semantic segmentation task.
3. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 1 or 2, characterized in that, The affinity-guided sharing strategy includes: The first stage involves calculating the gradient inner product between the instance segmentation task and the semantic segmentation task of each linear layer in the coupled backbone network to obtain the affinity value of each linear layer. Then, the affinity values of all linear layers are normalized to obtain the proportion of shared parameters among the tasks of each linear layer. The second stage involves introducing channel-dimensional perceptual scores to each linear layer. Based on the relationship between the perceptual scores and the threshold, task-shared weights or task-specific weights are activated, enabling the coupled backbone network to adaptively determine the distribution of task-shared parameters.
4. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 3, characterized in that, The step of calculating the gradient inner product between the instance segmentation task and the semantic segmentation task of each linear layer in the coupled backbone network to obtain the affinity value of each linear layer includes: During backpropagation, the gradients of the instance segmentation task and the semantic segmentation task are calculated respectively, and the obtained gradients are flattened into one-dimensional vectors. The inner product of the corresponding one-dimensional vectors of the instance segmentation task and the semantic segmentation task is calculated to obtain the affinity.
5. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 3, characterized in that, The step of activating task-shared weights or task-specific weights based on the relationship between the perceived score and the threshold includes: When the perceived score is less than or equal to the threshold, the task-shared weight is activated. When the perceived score is greater than the threshold, the task-specific weights are activated.
6. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 2, characterized in that, The element-level gradient direction correction algorithm includes: The gradient vectors of the image instance segmentation task and the semantic segmentation task are flattened into one-dimensional vectors; Determine whether there is a conflict in the gradient directions of the two tasks based on the one-dimensional gradient vectors of the two tasks; When a conflict exists, the weighting coefficients are calculated based on the gradient descent rate, and the gradient vector is then updated.
7. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 6, characterized in that, The step of calculating weighting coefficients based on the gradient descent rate and then updating the gradient vector includes: Calculate the weight coefficient of each task based on the gradient change rate of each task; The corresponding gradient vector is updated using the weight coefficients.
8. The joint optimization method for image instance segmentation and semantic segmentation tasks according to claim 7, characterized in that, The calculation of the weight coefficients for each task based on the gradient change rate of each task includes: The expression for the gradient rate of change is: ， in, For the first The gradient rate of change for each task. and The first Subsequent In the nth iteration The first task The first linear layer single-dimensional gradient vector of the nth linear layer One portion, This indicates a search for the 2-norm; The expression for the weighting coefficient is: ， in, For the first The weight coefficients of each task.
9. An image instance segmentation and semantic segmentation task joint optimization device, characterized in that the image instance segmentation and semantic segmentation task joint optimization device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image instance segmentation and semantic segmentation task joint optimization method as described in any one of claims 1 to 8.
10. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement the joint optimization method for image instance segmentation and semantic segmentation tasks as described in any one of claims 1 to 8.