A weakly supervised image semantic understanding method based on multi-task learning

Abstract

The application discloses a kind of weakly supervised image semantic understanding methods based on multi-task learning, comprising the following steps: obtaining task missing image, constructing multi-level task sharing encoder, extracting high-level semantic information layer by layer, input corresponding decoder branch;Construct public space-task space feature mapping module, through the unaligned task fusion module and task interaction mapping module, update each subtask feature by mapping;Task adaptive feature update module is constructed, and multi-level iterative update unaligned task feature;Task adaptive weakly supervised image semantic understanding framework is constructed, model loss function is established, image data with task missing is input into model, and obtains multi-task prediction result such as semantic segmentation, depth estimation, surface normal estimation.The application is according to the data information of task label unaligned, through the mapping interaction of public space and task space, fully fuses unaligned task feature, iteratively generates high-quality multi-task prediction result, can effectively handle weakly supervised problem with task missing, and simultaneously improves each task prediction accuracy.

Description

Technical Field

[0001] This invention belongs to the field of pixel-level image semantic understanding, and more specifically, relates to a weakly supervised image semantic understanding method based on multi-task learning. Background Technology

[0002] Complex image visualization and analysis tasks in real-world scenarios, particularly pixel-level visualization, can typically be decomposed into a set of related tasks, such as semantic segmentation, depth estimation, and surface normal estimation. Traditional methods usually build separate models for each task and then synthesize the results from multiple models into a final output. With the rapid development of deep learning, pixel-level image understanding has seen significant improvements in both efficiency and accuracy. Since the pixel-level image information from multiple tasks is complementary in terms of features such as edges, texture, and depth, some works have established unified multi-task learning frameworks to explore the correlations and complementarities between tasks, simultaneously fulfilling the requirements of multiple tasks and improving the accuracy and efficiency of image semantic understanding.

[0003] However, these methods largely rely on large-scale training with precise annotations. When faced with complex situations involving numerous subtasks and a multitude of task labels for each image, this significantly increases annotation costs and reduces model efficiency. For weakly supervised image semantic understanding, numerous methods have emerged for single-task tasks, such as image-level labeled classification and segmentation models, and generative adversarial models. However, in multi-task learning, the types of subtasks are numerous, and the degree of subtask missingness varies across different datasets. Appropriately designing multi-task learning models to address the problem of task misalignment in weak supervision remains a challenge in pixel-level image multi-task learning. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a weakly supervised image semantic understanding method based on multi-task learning. This method uses feature mapping interaction between the common space and the task space to iteratively update multi-task features and solves the problem of multi-task misalignment in weakly supervised pixel-level semantic understanding through task adaptive learning.

[0005] To achieve the above objectives, this invention provides a pixel-level image semantic understanding method based on multi-task learning, comprising the following steps:

[0006] Step 1: Obtain the source image and missing task labels, construct a multi-level task-shared encoder, extract high-level semantic information and input it into the corresponding decoder branch, including the following sub-steps:

[0007] (1.1) Input the color image into a shared multi-task feature encoder, which includes multi-level feature extraction and pooling operations;

[0008] (1.2) The I levels of the encoder generate encoder features at different scales. This feature is not task-specific, but only related to the original image, and is an abstract representation of the features of the original image at different scales;

[0009] (1.3) Based on the results of steps (1.1) and (1.2), the features at each level are combined through direct connections and cross-layer connections and input into each decoder branch. Each decoder branch corresponds to an unaligned subtask. The features obtained by the subtask branch from the shared encoder are represented as follows: Where i represents the number of levels in the task branch, M is the total number of subtasks contained in all images of the dataset, and any image contains m random misaligned tasks (this value is determined by the task missing status of each image).

[0010] Step 2: Construct a public space-task space feature mapping module. Update the features of each sub-task by fusing the mapped public space, including the following sub-steps:

[0011] (2.1) Based on the subtask branches generated in step one, construct an unaligned task fusion module to fuse the feature spaces of multiple subtasks. Define the mapping function from the subtask space to the common space as follows: The interaction function between the two features is f. Applying function f sequentially to the misaligned task features yields the overall fusion function σ, which can be written as:

[0012]

[0013] When m is 2, σ requires only two task interactions; when m is 1, σ is directly taken. The value is sufficient.

[0014] (2.2) Based on the fusion features extracted in (2.1), they are mapped to a shared public space for multiple tasks, forming a public feature fusion space O. (i) , represented as:

[0015]

[0016] in, The function is obtained through unaligned task fusion and spatial mapping, and is expressed as:

[0017]

[0018] Where g is a nonlinear mapping function containing a general activation function. Common space O (i) By applying complementary features of task fusion to pattern enhancement in each subtask, pattern propagation between tasks is achieved.

[0019] (2.3) Construct a task interaction mapping module to map the public space O(i) Remapped into the spaces of each subtask. (i) Containing feature fusion information for all tasks in the image, for m misaligned task branches, O (i) The mapping is performed back to the original task space according to certain rules. Specifically, using the strategy learned by the network model, the mapping function from the common space to the specific sub-task space is defined as θ. t Then the mapped features are represented as:

[0020]

[0021] Where, θ t λ is the nonlinear mapping function learned by the network model. t is a network hyperparameter, representing the feature map absorption coefficient of task t.

[0022] Step 3: Construct a task-adaptive feature update module to iteratively update misaligned task features at multiple levels, including the following sub-steps:

[0023] (3.1) Public space O (i) Features mapped to subtask space and features within each level and task After further merging and convolutional transformation, the subtask features are updated as follows: Process writing:

[0024]

[0025] Where δ is the convolution function used for training the network model. This indicates that the summation method is used for fusion, and the result generates the feature output of the i-th level decoder.

[0026] (3.2) The output features of the i-th layer of the m decoder branches The input is then continued to the (i+1)th layer of the decoder, thus establishing the task-adaptive feature update module. The iterative process is represented as:

[0027]

[0028]

[0029] Step four involves constructing a task-adaptive image semantic understanding framework, establishing a task-specific model loss function, and training the model to obtain pixel-level image semantic prediction results for semantic segmentation, depth estimation, surface normal estimation, and other functions. This includes the following sub-steps:

[0030] (4.1) Based on the common space-task space feature mapping module and the task-adaptive feature update module, a task-adaptive image semantic understanding framework is constructed. The multi-level task feature sharing network serves as the bottom encoder part, and the feature mapping module and feature update module serve as the intermediate decoder part of the framework. Finally, the results are fed into a multi-task independent prediction classifier to generate the final prediction result.

[0031]

[0032] Where, γ t This represents the prediction function of the last layer of the t-th task branch of the decoder, which is implemented as a convolution function.

[0033] (4.2) Based on the task-adaptive image semantic understanding framework, design a task-specific loss function for each sub-task of actual training, and establish the overall loss function of the framework.

[0034] Let the subtasks M be of three types: depth estimation, semantic segmentation, and surface normal estimation, i.e., when m = 1, 2, 3, for the three types of subtasks T seg ,T dep ,T nor Specify the loss function for each image. Suppose the training data contains X misaligned color images. For the x-th image, if it has a semantic segmentation task label Y... seg This forms the semantic segmentation task branch, generating prediction results. At this point, cross-entropy loss L is used. s , ζ s The cross-entropy loss function is:

[0035]

[0036] If it has a depth estimation task label Y dep This forms the depth estimation task branch, which generates prediction results. At this point, calculate the loss L. d , ζ d For the berHu loss function:

[0037]

[0038] If it has a surface normal estimation task label Y nor This forms the surface normal estimation task branch, generating prediction results. At this point, calculate the loss L. n , ζ n L1 loss function:

[0039]

[0040] Therefore, for the x-th image in the training data as input, the total loss function ζ (x) Represented as:

[0041]

[0042] Where, μ x Let μ be a 1×3 unaligned task weight matrix. In this binary matrix, if a task is missing, the corresponding column value is 0; otherwise, it is 1. When the number of unaligned tasks is m, μ... x The weight matrix will be expanded to a size of 1×m. For X images as input to the training data, the overall training loss function ζ is:

[0043]

[0044] (4.3) Image data missing from the task was input into the model framework for training. Training was divided into two phases: initialization and fine-tuning. The network model used the SGD optimizer for gradient optimization. To increase the diversity of training data, four data augmentation methods—scaling, flipping, cropping, and rotation—were also employed. After the model converged, during the testing phase, a color image was input into the trained weakly supervised multi-task learning model to obtain prediction results for the image on multiple tasks, including semantic segmentation, depth estimation, and surface normal estimation.

[0045] Step (1.3) specifically involves determining the minimum scale feature S of the deepest layer for each level of generated features at different scales. (I) The most general feature representation is directly connected to each subsequent subtask branch; at the same time, the high-level semantic information S is extracted layer by layer. (i) The i = 1, 2, ... I will be connected across layers corresponding to the layers with the same scale in the multi-level decoder, thereby compensating for the loss of detail in the image during the encoder pooling process.

[0046] Step (2.1) specifically involves identifying misaligned task features. Transformed to the same spatial representation through mapping. The function form must hold for any m, therefore, common features are fused by sequentially interacting with each other.

[0047] The feature interaction function f is a fusion convolutional network containing a non-linear mapping function. First, two tasks are selected from the misaligned tasks to interact, generating preliminary interaction features. Then, the remaining misaligned task features are sequentially applied to the interaction features with the function f to obtain the fused features. σ is the iteration of the interaction function f, and it inherits the original feature scale. This method can fuse misaligned task features regardless of the value of M.

[0048] Step (3.2) specifically involves the following step: In the decoder layers i = 1, 2, ... I, the task-adaptive feature update module will combine the common space-task space feature mapping at each layer to generate a common feature fusion space O. (i) It is then mapped to the original task space and fused to generate new task features. Iterate and update the next level.

[0049] This module holds true for any misaligned task m and is adaptive to the randomly missing multi-task labels for each image, thus enabling task-adaptive feature iteration updates. As the feature scale is progressively recovered, the interaction between the common space and the misaligned task occurs progressively across multiple scales.

[0050] Step (4.1) specifically involves the low-level feature-sharing encoder extracting high-level semantic information of the image layer by layer and inputting it into the corresponding decoder branch; the decoder part of the feature mapping and feature update adaptively integrates the misaligned task features. Mapping to the common feature fusion space O (i) In the middle, and mapped back to the original task space to update and generate. It uses cross-layer connections to mine the potential features of each task from coarse to fine, forming a multi-level iterative update; the top-level independent prediction classifier generates the final prediction result independently in multiple task branches through the convolutional layers of the prediction functions of each task in the top layer of the decoder.

[0051] Compared with the prior art, the present invention has the following beneficial effects through the above-described technical solutions:

[0052] 1) The model of this invention is not constrained by missing tasks, can handle weakly supervised data with misaligned tasks, and can mine the supervision information contained in training samples of various tasks, thereby reducing the requirements for the quantity and quality of multi-task labeled data.

[0053] 2) This invention uses the mapping of multiple task spaces to extract information from other tasks to promote the learning of the target task. The model fitting is constrained by different tasks, which can reduce overfitting and avoid getting trapped in local optima.

[0054] 3) This invention establishes a task adaptive iterative model based on a common space-task space mapping mechanism. By fusing misaligned task features in the common space and updating the mapping, it can achieve better performance on a variety of tasks, while reducing model complexity, avoiding repeated training and prediction, and showing significant advantages in real-time prediction. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the weakly supervised image semantic understanding framework based on multi-task learning in this invention.

[0056] Figure 2This is a detailed flowchart of step two in this invention.

[0057] Figure 3 This is a schematic diagram illustrating the effect of step four in this invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] This invention uses several publicly available multi-task datasets as materials, and the operating environment is CUDA10-Python3.

[0060] like Figure 1 As shown, the weakly supervised image semantic understanding method based on multi-task learning of the present invention includes the following steps:

[0061] Step 1: Obtain the source image and misaligned multi-task labels, construct a multi-level task-sharing encoder, extract high-level semantic information of the image layer by layer, and input it into the corresponding decoder branch, including the following sub-steps:

[0062] (1.1) For the input model image and misaligned task labels, the color image is first input into a shared multi-task feature encoder. This multi-task feature encoder contains multiple levels of feature extraction and pooling operations, where I represents the total number of levels of the encoder, and the scale of the feature representation at each level is continuously reduced;

[0063] (1.2) The I levels of the encoder generate encoder features at different scales. The encoder features are not task-specific, but only related to the original image, and are abstract representations of the features of the original image at different scales;

[0064] (1.3) Based on the results of steps (1.1) and (1.2), the features generated by the encoder at each level are combined through direct connection and cross-layer connection and input into the subsequent decoder branches. Each branch of the decoder corresponds to a subtask.

[0065] Specifically, the multi-task shared encoder outputs scaled-down feature representations as the layer deepens, i.e., H i <H i-1 W i <W i-1 For features generated at different scales at each level, the smallest scale feature S in the deepest layer... (I) The most general feature representation is directly connected to each subsequent subtask branch; at the same time, the high-level semantic information S is extracted layer by layer. (i)The i = 1, 2, ... I will be connected across layers corresponding to the layers with the same scale in the multi-level decoder, thereby compensating for the loss of detail in the image during the encoder pooling process.

[0066] When an image has m subtask labels, the feature representation of the subtask branch obtained from the shared encoder is as follows: Here, M is the total number of subtasks contained in all images of the dataset, and any image contains m randomly selected misaligned tasks (this value is determined by the task missing status of each image). In practical applications of pixel-level image semantic understanding, the subtasks corresponding to the decoder branches include, but are not limited to, depth estimation, semantic segmentation, and surface normal estimation. The m subtasks differ significantly, therefore, an adaptive decoder model needs to be constructed for m randomly selected different subtasks of the input image.

[0067] Step 2: Construct a public space-task space feature mapping module. Through the misaligned task fusion module and the task interaction mapping module, the public space mapping updates the features of each sub-task.

[0068] like Figure 2 As shown, each decoder branch jointly constructs a single-level common space-task space feature mapping module, which specifically includes the following sub-steps:

[0069] (2.1) For the m sub-task branches generated in step one, construct an unaligned task fusion module to fuse the feature representation spaces of multiple sub-task branches;

[0070] At this point, the characteristics of the m subtask branches are: Where i represents the number of levels in this task branch. Feature maps extracted from different subtask branches are typically located in different spaces, and their data distributions differ. To fuse the feature information from different subtasks, the features... Transformation to the same spatial representation through mapping. Define the mapping function as follows: Its input consists of m task features specific to a single image and at a single level. Since the value of m is not fixed in misaligned task fusion, The function form must hold for any m to effectively fuse any number of multi-task features. Specifically, common features will be fused using a sequential interactive approach.

[0071] In the i-th layer decoder branch, the misaligned task features are respectively First, we define the feature interaction function f as a fusion convolutional network containing a non-linear mapping function. We select two tasks from the misaligned tasks to interact, generating preliminary interaction features to explore the complementarity between the pairs of tasks, while inheriting the scale of the original features. Then, we apply the function f to the remaining misaligned task features sequentially with the interaction features to obtain the fused features. σ is the iteration of the interaction function f, which can be written as:

[0072]

[0073] When m is 2, σ requires only two task interactions; when m is 1, σ is directly taken. The value of M is sufficient. Therefore, this method can be used to fuse misaligned task features regardless of the value of M.

[0074] (2.2) The fusion features extracted using the misaligned task fusion module are mapped to a common feature space shared by multiple tasks to form a common feature fusion space O. (i) , represented as:

[0075]

[0076] in, The function is obtained through unaligned task fusion and spatial mapping, and is expressed as:

[0077]

[0078] Where g is a nonlinear mapping function containing a general activation function. The function projects different subtask spaces into a shared common space through iterative fusion and spatial mapping, thereby enabling the common space to capture multi-angle, deep-level features and high-level semantic representations of the image. Common space O (i) Then, it will interact with the features of each subtask and map them together, thereby applying the complementary features of task fusion to the pattern enhancement of each subtask and realizing pattern propagation between tasks.

[0079] (2.3) Based on the public feature fusion space O generated in (2.2) (i) A task interaction mapping module is constructed to remap the fusion features of the public space to each sub-task space;

[0080] Misaligned task fusion features are mapped as O in the implicit common space. (i) That is, it contains the feature fusion information of all target tasks in the image. To explore feature complementarity and promote mutual feature learning, for m misaligned task branches, the common space O is... (i) Mapping back to the original task space according to certain rules, the feature representation of the task space can be updated and strengthened because the features of the common space have already incorporated complementary features from other tasks. Specifically, using a network model learning strategy, the mapping function from the common space to the specific sub-task space is defined as θ. t Then the mapped features are represented as:

[0081]

[0082] Where, θ t λ is the nonlinear mapping function learned by the network model. t is a network hyperparameter, representing the feature map absorption coefficient for task t.

[0083] Step 3: Construct a task-adaptive feature update module to iteratively update misaligned task features at multiple levels. This includes the following sub-steps:

[0084] (3.1) Based on the common space-task space feature mapping module, the mapping features of the common space and the features of the task space are further merged, and after convolution transformation, the feature output of the i-th level decoder is generated.

[0085] Use of public space O (i) Features mapped to subtask space Features within each level and task After further merging and convolutional transformation, the subtask features are updated as follows: writing:

[0086]

[0087] Where δ is the convolution function used for training the network model. This indicates a combination of summation methods.

[0088] (3.2) The output features of the i-th layer of the m decoder branches The input is then continued to the (i+1)th layer of the decoder, thus establishing the task-adaptive feature update module. The iterative process is represented as:

[0089]

[0090]

[0091] In the decoder layers i = 1, 2, ... I, the task-adaptive feature update module will combine the common space-task space feature mapping at each layer to generate a common feature fusion space O. (i) It is then mapped to the original task space and fused to generate new task features. Iterative updates to the next level. This module holds true for any misaligned task m and is adaptive to the randomly missing multi-task labels for each image, thus forming a task-adaptive feature iterative update.

[0092] Specifically, the common space is generated by mapping any number of misaligned subtasks, and then a shared feature representation and corresponding weights are provided for each misaligned subtask. As the feature scale is progressively recovered, the interaction between the common space and the misaligned tasks occurs progressively across multiple scales. The mapping of the space reduces the complexity of pairwise interactions between multiple features, intuitively and easily projecting the fused features into the spaces of each subtask, thereby reducing the interaction time between tasks and simplifying the shared model structure.

[0093] Step four involves constructing a task-adaptive image semantic understanding framework, establishing a model loss function, inputting the missing image data into the model, and obtaining pixel-level prediction results for semantic segmentation, depth estimation, surface normal estimation, and other functions. This includes the following sub-steps:

[0094] (4.1) Based on the common space-task space feature mapping module and the task adaptive feature update module, a task adaptive image semantic understanding framework is constructed. The multi-level task feature sharing network serves as the bottom encoder part, the feature mapping module and the feature update module serve as the intermediate decoder part of the framework, and finally the multi-task independent prediction classifier is fed into it.

[0095] The low-level feature-sharing encoder extracts high-level semantic information from the image layer by layer and inputs it into the corresponding decoder branch; the decoder part of the feature mapping and feature update adaptively integrates misaligned task features. Mapping to the common feature fusion space O (i) In the middle, and mapped back to the original task space to update and generate. It uses cross-layer connections to mine latent features for each task from coarse to fine, forming a multi-level iterative update; the top-level independent prediction classifier generates the final prediction result independently in multiple task branches through the convolutional layers of the prediction functions of each task in the top layer of the decoder.

[0096]

[0097] Where, γ t This represents the prediction function of the last layer of the t-th task branch of the decoder, which is implemented as a convolution function.

[0098] (4.2) Based on the task-adaptive image semantic understanding framework, design a task-specific loss function for each sub-task of actual training, and establish the overall loss function of the framework.

[0099] The overall loss function is specific to task T1, T2, ... T mThe loss function is composed of [various components]. In practical applications of pixel-level image semantic understanding, the subtasks corresponding to the decoder branches include, but are not limited to, depth estimation, semantic segmentation, and surface normal estimation. These subtasks differ significantly; therefore, task-specific loss functions are set for different semantic segmentation tasks. This step will focus on the most widespread application scenario, where subtask M specifically falls into three categories: depth estimation, semantic segmentation, and surface normal estimation. Specifically, when m = 1, 2, and 3, the loss function is applied to the three subtasks T. seg ,T dep ,T nor Specify the loss function respectively.

[0100] Suppose the training data contains X misaligned color images. For the x-th image, calculate the loss for each misaligned sub-task sequentially and then synthesize the total loss. If the x-th image has a semantic segmentation task label Y... seg This forms the semantic segmentation task branch, which generates prediction results through a classifier. At this point, cross-entropy loss L is used. s Among them, ζ s The cross-entropy loss function is:

[0101]

[0102] If the training image has a depth estimation task label Y dep This forms the depth estimation task branch, which generates prediction results through the classifier. At this point, calculate the loss L. d Among them, ζ d For the berHu loss function:

[0103]

[0104] If the training image has a surface normal estimation task label Y nor This forms the surface normal estimation task branch, which generates prediction results through the classifier. At this point, calculate the loss L. n Among them, ζ n L1 loss function:

[0105]

[0106] Therefore, for the x-th image in the training data as input, the total loss function ζ (x) It can be represented as:

[0107]

[0108] Where, μ xThis is a 1×3 unaligned task weight matrix, with column numbers assigned to task numbers. A column has a value of 1 if a task is not missing, and a value of 0 if a task is missing. This matrix is ​​adaptively generated during image input. When the number of unaligned tasks is m, μ... x The weight matrix will be expanded to a size of 1×m. Therefore, with X images input in the training data, the overall training loss function ζ can be expressed as:

[0109]

[0110] (4.3) Input the missing image data of the task into the model framework for training. The training is divided into two stages: initial and fine-tuning. The network model uses the SGD optimizer for gradient optimization.

[0111] The following training methods and parameter settings are only applicable to the example scenario involving three misaligned subtasks: depth estimation, semantic segmentation, and surface normal estimation. When the number and nature of the subtasks change, the training parameter settings need to be adjusted accordingly: The initial learning rate is set to 1e-4 for the pre-trained network and 0.01 for other network layers, and decays to 1e-5 and 0.01 during the fine-tuning phase. The momentum parameter and weight decay parameter are set to 0.9 and 1e-4, respectively. Furthermore, to increase the diversity of the training data, four data augmentation methods—scaling, flipping, cropping, and rotation—are employed.

[0112] After the model has converged during training, the color image is input into the trained weakly supervised multi-task learning model during the testing phase to obtain the prediction results of the image on multiple tasks such as semantic segmentation, depth estimation, and surface normal estimation.

[0113] This invention utilizes the mapping interaction between the common space and the task space to enable iterative learning of misaligned tasks within the same model framework, achieving weakly supervised pixel-level multi-task learning and improving the performance of each task. Misaligned multi-task data often leads to model incompatibility, while the input images with missing tasks are large in volume and have low annotation costs. This invention uses task-missing data to establish a weakly supervised learning framework, explores the potential commonalities and complementarities between tasks, and uses a common space mapping mechanism to assist single-task learning. The framework adopts a structure of a bottom-level shared encoder, an intermediate-level feature mapping decoder, and a top-level predictive classifier, using convolutional neural networks such as ResNet as the backbone to construct a task-adaptive weakly supervised multi-task learning framework for joint learning of misaligned tasks such as semantic segmentation, depth estimation, and surface normal estimation.

[0114] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A weakly supervised image semantic understanding method based on multi-task learning, characterized in that, Includes the following steps: Step 1: Obtain the missing task image, construct a multi-level task-shared encoder, and input the features into the corresponding decoder branch, including the following sub-steps: (1.1) Input color images into a shared multi-task feature encoder, which includes multi-level feature extraction and pooling operations; (1.2) Encoder I Each level generates encoder features at different scales. ; (1.3) Combine direct connections and cross-layer connections between features at each level and input them into the decoder branches of each misaligned subtask. The features of each branch are represented as follows: ,in, i This indicates the level number within the task branch. M It represents the total number of subtasks contained in all images of the dataset, with any image containing a random subtask. m indivual, Step 2: Construct a public space-task space feature mapping module to map and update the features of each sub-task, including the following sub-steps: (2.1) Construct an unaligned task fusion module to fuse the feature spaces of multiple subtasks. The mapping function from the subtask space to the common space is: The interaction function between the two features is f Multi-task features are applied sequentially. f The function is the overall fusion function. for: Among them, when m When it is 2, Only two tasks need to be interacted; when m When it is 1, Take directly value, (2.2) Map the integrated features to the shared public space to form a public feature integration space. , represented as: in, The function is obtained through unaligned task fusion and spatial mapping, and is expressed as: Where g is a nonlinear mapping function containing a general activation function. (2.3) Construct a task interaction mapping module to integrate the public space Mapped to the sub-task space, the strategy learned by the network model is applied to... m There are several misaligned task branches, and the mapping function from the common space to the specific subtask space is defined as follows: Then the mapped features are: in, The nonlinear mapping function learned by the network model. λ t Network hyperparameters are task-specific parameters. t The feature map absorption coefficient, Step 3: Construct a task-adaptive feature update module to iteratively update misaligned task features at multiple levels, including the following sub-steps: (3.1) Mapping the public space to sub-task space features At each level, the features are further merged with those within the task, and after convolutional transformation, the sub-task features are updated as follows: , represented as: in, The convolution function used for training the network model. To achieve the fusion of summation methods, the th... i Feature output of the hierarchical decoder (3.2) will m The decoder branch i The output features of the layer continue to be input into the decoder. i +1 layer, establish the task-adaptive feature update module, the iterative process is represented as follows: Step four involves constructing a task-adaptive image semantic understanding framework, establishing a model loss function, and obtaining multi-task prediction results such as semantic segmentation, depth estimation, and surface normal estimation after training. This includes the following sub-steps: (4.1) Construct a task-adaptive image semantic understanding framework, in which a multi-level task feature sharing network serves as the bottom encoder, the feature mapping module and the feature update module serve as the intermediate decoder, and finally the result is fed into a multi-task independent prediction classifier to generate the final prediction result: in, Indicates the decoder's first t The prediction function of the last layer in each task branch takes the form of a convolution function. (4.2) Based on the overall framework, establish a specific loss function for each subtask participating in the training to form the overall loss function; (4.3) Input the missing image data of the task into the model for training. The training is divided into two stages: initial and fine-tuning. The SGD optimizer is used for gradient optimization. After the model converges, the color image is input into the trained weakly supervised multi-task learning model to obtain the prediction results of multiple tasks such as semantic segmentation, depth estimation, and surface normal estimation.

2. The multi-task weakly supervised image semantic understanding method according to claim 1, characterized in that, Step (1.3) specifically involves determining the smallest scale feature of the deepest layer for each level of generated features at different scales. As a general feature representation, it is directly connected to each subsequent subtask branch; at the same time, high-level semantic information is extracted layer by layer. By connecting layers with the same scale as those in the multi-level decoder, the detail loss of the image during the encoder pooling process is compensated.

3. The multi-task weakly supervised image semantic understanding method according to claim 1, characterized in that, Step (2.1) specifically involves misaligning task features. Transformed to the same spatial representation through mapping Common features are fused using a sequential feature interaction method; feature interaction function. f For fused convolutional networks containing nonlinear mapping functions, First, select two tasks from the misaligned tasks to interact and generate preliminary interaction features. Then, apply the remaining misaligned task features to the interaction features in sequence. f The function yields the fused features. ,in, It is an interactive function f The iteration inherits the original feature scale.

4. The multi-task weakly supervised image semantic understanding method according to claim 1, characterized in that, Step (3.2) specifically involves, at the decoder layer... In this process, the task-adaptive feature update module combines the common space-task space feature mapping at each level to generate a common feature fusion space. It is then mapped to the original task space and fused to generate new task features. Iterate and update the next level. This module handles tasks with arbitrary value misalignment. m It is established that it is adaptive to the randomly missing multi-task labels of each image, thus forming a task-adaptive feature iteration update. With the feature scale being restored step by step, the interaction between the common space and the misaligned task is carried out step by step at multiple scales.

5. The multi-task weakly supervised image semantic understanding method according to claim 1, characterized in that, Step (4.1) specifically involves the low-level feature-sharing encoder extracting high-level semantic information of the image layer by layer and inputting it into the corresponding decoder branch; the decoder part of the feature mapping and feature update adaptively converts misaligned task features... Mapping to the public feature fusion space In the middle, and mapped back to the original task space to update and generate. It uses cross-layer connections to mine the potential features of each task from coarse to fine, forming a multi-level iterative update; the top-level independent prediction classifier generates the final prediction result independently in multiple task branches through the convolutional layers of the prediction functions of each task in the top layer of the decoder.

6. The multi-task weakly supervised image semantic understanding method according to claim 1, characterized in that, Step (4.2) specifically involves establishing a task-specific loss function for each subtask participating in training based on the overall framework, synthesizing the overall loss function, and then applying the loss function to the subtask. M The three categories are depth estimation, semantic segmentation, and surface normal estimation. At that time, for the three sub-tasks Specify the loss function respectively , Training data includes X A misaligned color image, for which the first... x Image, total loss function ζ (x) Represented as: in, for The misaligned task weight matrix is ​​defined such that a column value is 0 when a task is missing, and 1 otherwise. The matrix is ​​defined when the number of misaligned tasks is [missing information]. m hour, The weight matrix will be extended to Size, relative to training data X Image input, overall training loss function ζ for: 。 7. The specific task loss function according to claim 6, characterized in that, For the training data, the first x An image, if it has a semantic segmentation task label. This forms the semantic segmentation task branch, generating prediction results. , ζ s is the cross-entropy loss function, and the loss is... L s for: If it has a depth estimation task label This forms the depth estimation task branch, which generates prediction results. , ζ d is the berHu loss function, and the loss is... L d for: If it has a surface normal estimation task label This forms the surface normal estimation task branch, generating prediction results. , ζ n is L 1. Loss function, loss L n for: 。

Images