Multitasking methods, media and electronic devices

Abstract

This application relates to the field of image processing and discloses a multi-task processing method, medium, and electronic device that can improve the efficiency of multi-task interaction and the accuracy of multi-task prediction. The method includes: acquiring an input image; for the input image, acquiring at least two task-independent features corresponding to M tasks and at least two task-related features corresponding to each task through a neural network, wherein any two tasks are distinct, the at least two task-independent features are shared features during the processing of the M tasks, and each task-related feature is a feature used independently during the processing of its corresponding task; for each of the M tasks, stacking and fusing at least one task-independent feature from the at least two task-independent features with the corresponding at least one task-related feature in the channel dimension to obtain the backbone features corresponding to each task; and for the M tasks, using the backbone features corresponding to each task to obtain the output results corresponding to each task.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multi-task processing method, medium, and electronic device. Background Technology

[0002] Currently, deep learning-based computer vision models are widely used on terminal devices, and the types of applications are rapidly developing as application scenarios expand. Real-world applications often involve using multiple vision task models simultaneously. For example, in mobile phone photography, it's necessary to simultaneously identify the shooting scene and the subject to perform targeted image enhancement; in smart screen motion sensing scenarios, it's necessary to simultaneously identify the moving subject and its posture; and in augmented reality scenarios using smart glasses, it's necessary to simultaneously perform tasks such as object recognition and subject segmentation.

[0003] However, while multiple single-task models enrich computer vision applications, they also introduce a series of problems that severely impact user experience, such as increased bandwidth, computational load, latency, and power consumption, hindering the further expansion of vision task applications. Multi-task learning methods attempt to leverage the redundancy and complementarity of multiple tasks in the feature extraction process, fusing multi-task models with higher accuracy than individual single-task models, while significantly reducing computational load, latency, and power consumption. Therefore, how to achieve efficient multi-task learning in applications such as intelligent sensing has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a multitasking method, medium, and electronic device that can improve the efficiency of multitasking interaction and the accuracy of multitasking prediction.

[0005] In a first aspect, embodiments of this application provide a multi-task processing method for an electronic device, comprising: acquiring an input image; for the input image, acquiring at least two task-independent features corresponding to M tasks and at least two task-related features corresponding to each of the M tasks through a neural network, wherein the M tasks are all image processing tasks and any two tasks are different from each other, and the at least two task-independent features are features shared during the processing of the M tasks, and each task-related feature is a feature used independently during the processing of the corresponding task; for each of the M tasks, stacking and fusing at least one task-independent feature and the corresponding at least one task-related feature in the channel dimension to obtain the backbone features corresponding to each task; and for the M tasks, using the backbone features corresponding to each task to obtain the output results corresponding to each task.

[0006] Thus, in this embodiment, the multi-task processing method provided by this application can decompose convolutional layers into task-independent convolutions and task-related convolutions based on a partially shared structure, achieving end-to-end task decoupling. Furthermore, by utilizing the partially shared structure, the abstract features of the previous layer's task-independent convolutions are integrated into the current layer's task-related convolutions. This partially shared structure, as a basic structure, can be directly integrated into any visual task model, improving the adaptability of the multi-task processing in different scenarios and avoiding the need to calculate task-independent features for each task. In addition, the mask-based two-level interaction structure in this application allows the multi-task information interaction process to use task attention to supplement the backbone features of other tasks into the backbone features of the current task. This first transforms the initial prediction results of other tasks into task masks, and then fuses the masks with the initial prediction results of the current task to obtain multi-task prediction results, achieving efficient multi-task processing. Therefore, it improves the efficiency of multi-task interaction and the accuracy of multi-task prediction.

[0007] In one possible implementation of the first aspect described above, the neural network includes a backbone module, which comprises N partially shared convolutional layers (hereinafter referred to as N partially shared convolutional layers). Each partially shared convolutional layer includes a task-independent convolutional layer for acquiring task-independent features and a task-related convolutional layer for acquiring task-related features. In this way, the convolutional layers can be decomposed into task-independent and task-related convolutions based on the partially shared structure, achieving end-to-end task decoupling.

[0008] In one possible implementation of the first aspect above, the acquisition of at least two task-independent features corresponding to M tasks and at least two task-related features corresponding to each of the M tasks through a neural network includes: inputting the input image into the first layer of the partial shared convolution of the backbone module of the neural network, and outputting the first-level task-independent features and the first-level task-related features; inputting the (i-1)th level task-independent features into the task-independent convolution of the i-th layer of the partial shared convolution, and outputting the i-th level task-independent features; and stacking the (i-1)th level task-independent features and the (i-1)th level task-related features in the channel dimension and inputting them into the task-related convolution of the i-th layer of the partial shared convolution, and outputting the i-th level task-related features, until the N-th layer of the partial shared convolution is reached, where 2≤i≤N; wherein, the at least two task-independent features include the first to Nth level task-independent features, and the at least two task-related features corresponding to each task include the features corresponding to each task in the first to Nth level task-related features.

[0009] Thus, by utilizing a partially shared structure, this application can integrate the abstract features of the task-independent convolutions of the previous layer into the task-related convolutions of the current layer. This partially shared structure, as a basic structure, can be directly integrated into any visual task model, thereby improving the adaptability of the multi-task processing of this application in different scenarios.

[0010] In one possible implementation of the first aspect described above, at least one task-independent feature corresponding to each task includes one or more levels of task-independent features from levels 1 to N; at least one task-related feature corresponding to each task includes one or more levels of task-related features from levels 1 to N. This is beneficial for improving the efficiency of multi-task interaction.

[0011] In one possible implementation of the first aspect described above, the aforementioned N partially shared convolutional layers correspond to N preset stripping coefficients. Each stripping coefficient indicates the ratio of computational power of task-independent convolutions to that of task-related convolutions within a partially shared convolutional layer. Furthermore, the stripping coefficient increases with the model depth of the neural network. This balances the model capacity (i.e., the computational power of the corresponding convolutional layers) between task-independent and task-related features, establishing a process where the multi-task model capacity allocation gradually changes with model depth, achieving end-to-end progressive task decoupling.

[0012] In one possible implementation of the first aspect above, the initial convolutional layer in the N layers of partial shared convolutions includes M+1 first convolutional layers. The M first convolutional layers include M task-related convolutions corresponding to the tasks in sequence and 1 task-independent convolution. The input image is input into each of the M first convolutional layers. Each task-related convolution in the M first convolutional layers outputs the corresponding first-level task-related feature. The task-independent convolution in the M first convolutional layers outputs the first-level task-independent feature.

[0013] In one possible implementation of the first aspect above, the i-th partially shared convolutional layer includes M+1 second convolutional layers. The M+1 second convolutional layers include second task-related convolutions corresponding to M tasks in sequence and one second task-independent convolution. The j-th second task-related convolution takes into input the task-related features corresponding to the j-th task in the (i-1)-th level task-related features and the features in the (i-1)-th level task-independent features, and outputs the features in the task-related features corresponding to the j-th task in the i-th level task-related features, where 1≤j≤M. The second task-independent convolution in the i-th partially shared convolution takes into input the features in the (i-1)-th level task-independent features and outputs the features in the (i-1)-th level task-independent features.

[0014] In one possible implementation of the first aspect described above, the main module further includes N target network layers, which are batch normalization layers, activation layers, or pooling layers; the i-th target network layer includes M+1 target network layers, each of which includes a first target network layer corresponding to M tasks in sequence and a second target network layer that is unrelated to the task. The j-th first target network layer inputs the features of the task-related features corresponding to the j-th task in the (i-1)-th level task-related features and outputs the features of the task-related features corresponding to the j-th task in the i-th level task-related features, where 1≤j≤M; the second target network layer in the i-th target network layer inputs the features of the (i-1)-th level task-unrelated features and outputs the features of the (i-1)-th level task-unrelated features.

[0015] In one possible implementation of the first aspect above, the N-layer partial shared convolution of the backbone module further includes a group of fused convolutional layers; the group of fused convolutional layers includes M fused convolutional layers, the j-th fused convolutional layer among the M fused convolutional layers is used to input the task-related features corresponding to the j-th task in the task-related features of the N-th level and the features obtained by stacking the task-related features of the N-th level, 1≤j≤M; the j-th fused convolutional layer among the M fused convolutional layers is used to output the backbone features corresponding to the j-th task.

[0016] In one possible implementation of the first aspect above, the above-mentioned method of obtaining the output result of each task by utilizing the backbone features corresponding to each task for M tasks includes: using the backbone features of the j-th task to predict the primary prediction result of the j-th task, where 1≤j≤M; using the primary prediction result of the j-th task to obtain the secondary prediction result of the j-th task; and adding the primary prediction result of the j-th task and the secondary prediction result of the j-th task to obtain the output result of the j-th task.

[0017] In one possible implementation of the first aspect above, the above-mentioned prediction of the primary prediction result of the j-th task using the backbone features of the j-th task includes: for M features, fusing the backbone features corresponding to M-1 tasks other than the j-th task and stacking them into the backbone features of the current task, and using the stacked backbone features to predict the primary prediction result of the j-th task.

[0018] In one possible implementation of the first aspect described above, obtaining the secondary prediction result of the j-th task using the primary prediction result of the j-th task includes: for M features, converting the primary prediction results of M-1 tasks other than the j-th task into masks; multiplying the masks corresponding to the M-1 tasks other than the j-th task with the primary prediction result of the j-th task to obtain the corresponding masked prediction result; weighting and summing the masked prediction results corresponding to the M-1 tasks other than the j-th task according to their respective predefined weights, and then adding them to the primary prediction result of the j-th task to obtain the fused prediction result corresponding to the j-th task; and using the fused prediction result corresponding to the j-th task to predict the secondary prediction result of the j-th task. In this way, the primary prediction results of other tasks can be converted into task masks, and then the masks can be fused with the primary prediction result of the current task, improving the efficiency of multi-task interaction and the accuracy of multi-task prediction.

[0019] In one possible implementation of the first aspect described above, the neural network includes a first fusion layer, a task attention module, and a primary prediction module. The first fusion layer is a convolutional layer, the task attention module includes pooling layers and convolutional layers, and the primary prediction module includes convolutional layers, batch normalization layers, activation layers, and convolutional layers. The primary prediction result for the j-th task is generated as follows: the backbone features corresponding to M-1 tasks other than the j-th task are stacked and input into the first fusion layer to obtain a first fusion feature; the backbone features of the j-th task are input into the task attention module to obtain a channel-dimensional attention with a spatial dimension of 1×1; the channel-dimensional attention is multiplied by the first fusion feature and stacked with the backbone features of the j-th task to obtain a second fusion feature; the second fusion feature is input into the primary prediction module to output the primary prediction result for the j-th task. Thus, by utilizing the task attention mechanism, the correlation between the backbone features of other tasks and the current task is obtained, irrelevant features are filtered out, and the efficiency of multi-task interaction and the accuracy of multi-task prediction are improved.

[0020] In one possible implementation of the first aspect above, the neural network includes a secondary prediction module, which includes a convolutional layer, a batch normalization layer, an activation layer, and another convolutional layer; the secondary prediction result of the j-th task is the result obtained by inputting the fused prediction result corresponding to the j-th task into the secondary prediction module.

[0021] Secondly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the multitasking method described in the first aspect and any possible implementation thereof.

[0022] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the processors of the electronic device, for executing the multitasking method in the first aspect and any possible implementation thereof. Attached Figure Description

[0023] Figure 1 According to some embodiments of this application, a network architecture diagram for multitasking is shown;

[0024] Figure 2 According to some embodiments of this application, a network architecture diagram for multitasking is shown;

[0025] Figure 3 According to some embodiments of this application, a flowchart of a multi-task processing method is shown;

[0026] Figure 4 According to some embodiments of this application, a flowchart illustrating the output of the initial prediction result of the current task by a basic feature interaction submodule is shown.

[0027] Figure 5 According to some embodiments of this application, a flowchart illustrating a multi-task collaborative enhancement submodule for detecting the output of the current task is shown;

[0028] Figure 6 According to some embodiments of this application, a schematic diagram of a conventional HRNet18 structure is shown;

[0029] Figure 7 According to some embodiments of this application, a schematic diagram of a partially shared HRNet18 structure is shown;

[0030] Figure 8A According to some embodiments of this application, a schematic diagram of convolutional layer replacement is shown;

[0031] Figure 8B According to some embodiments of this application, a schematic diagram of convolution module replacement is shown;

[0032] Figure 9A According to some embodiments of this application, an input-output schematic diagram of a starting convolutional layer or convolutional module is shown;

[0033] Figure 9B According to some embodiments of this application, a structure and input / output schematic diagram of a fused convolutional group layer are shown;

[0034] Figure 10 According to some embodiments of this application, a structural schematic diagram of a basic feature interaction submodule is shown;

[0035] Figure 11 According to some embodiments of this application, a schematic diagram of the structure of a multi-task collaborative enhancement submodule is shown;

[0036] Figure 12 According to some embodiments of this application, a schematic diagram of a process for generating primary prediction results in a multi-task processing method is shown;

[0037] Figure 13 According to some embodiments of this application, a schematic diagram of the process for generating secondary prediction results in a multi-task processing method is shown;

[0038] Figure 14 According to some embodiments of this application, a schematic diagram of the structure of a mobile phone is shown. Detailed Implementation

[0039] The illustrative embodiments of this application include, but are not limited to, multitasking methods, media, and electronic devices.

[0040] The following explains some of the nouns and terms used in the embodiments of the application.

[0041] 1. Channel Dimension: In computer vision tasks, data often takes the form of 4-dimensional (4D) data. The first dimension is the batch processing dimension, which represents the number of data processed at one time; the second dimension (or the last dimension) represents the channel dimension, which corresponds to the red, green, blue (RGB) channels of the input image; the third and fourth dimensions (or the second and third dimensions) are spatial dimensions.

[0042] 2. The hierarchical structure of a neural network typically includes:

[0043] Convolutional Layer (Conv): Each convolutional layer consists of several convolutional units, and the parameters of each convolutional unit are optimized through the backpropagation algorithm. The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while more layers of the network can iteratively extract more complex features from low-level features.

[0044] Batch Normalization Layer (BN): Batch normalization involves inserting a normalization layer at the input of each layer in the network. This normalization process (normalizing to a mean of 0 and a variance of 1) is performed before the data enters the next layer. It is primarily used to accelerate network training and convergence, control gradient explosion to prevent gradient vanishing, and prevent overfitting.

[0045] Rectified Linear Unit Layer (ReLU): This function is applied to all elements of the input tensor without altering its spatial or depth information.

[0046] Pooling layers (Pool): Pooling layers can include max pooling, average pooling, global average pooling, global max pooling, and adaptive average pooling (Adaptive Avg Pooling). Average pooling calculates the average value of an image region as the pooled value for that region. Max pooling selects the maximum value of an image region as the pooled value for that region.

[0047] Convolution Blocks (Conv Blocks): Convolution blocks can be deployed after convolutional layers. Furthermore, a convolution block may include not only at least one convolutional layer, but also one or more of batch normalization layers, linear rectified unit layers, and pooling layers.

[0048] High Resolution Blocks (HR Blocks) are multi-scale feature fusion layers that select and fuse features from multiple layers before detection. There are two common types of multi-scale feature fusion layers: parallel multi-branch networks and serial skip-layer connection structures, both of which extract features within different receptive fields. For example, constructing image pyramids to obtain multi-scale features is a common approach in traditional object detection algorithms, such as the Deformable Parts Model (DPM) algorithm. Image pyramids scale the original image at different ratios to obtain input images of different sizes, and then perform feature extraction and prediction on the resulting images separately.

[0049] 3. Mean Intersection of Union (mIoU): This is a commonly used performance metric in object detection. For example, it can be used not only to identify positive and negative samples, but also to evaluate the distance between the predicted box and the ground truth.

[0050] 4. Root Mean Square Error (RMSE): It is the square root of the ratio of the square of the deviation between the observed value and the true value to the number of observations n. In actual measurement, the number of observations n is always finite, and the true value can only be replaced by the most reliable (best) value.

[0051] 5. High-Resolution Network (HRNet): A type of backbone network that can be applied to semantic segmentation, object detection, image classification, image segmentation, human pose estimation, face alignment, style transfer, image super-resolution, edge detection, and other network structures.

[0052] 6. Mask: A mask uses a selected image, graphic, or object to occlude (fully or partially) the image being processed, controlling the area or process of image processing. Image masking operations are commonly used in areas such as image smoothing, edge detection, and feature analysis. For example, performing a bitwise AND operation between the original image and the mask image results in an image that only contains the region of interest (ROI) of the original image.

[0053] 7. Shallow Image Features: These are features extracted by the shallow layer of the network. These features are closer to the input and contain more pixel-level information, such as fine-grained information like color, texture, edges, and corner details. Shallow image features are typically detail features, used to ensure the network captures more image details.

[0054] 8. High-level information of an image: These are features extracted by deep networks. These features are closer to the output and contain more abstract information, such as semantic information and other coarse-grained information. High-level information of an image is usually abstract features used to obtain overall information about the image.

[0055] 9. Part share (PS) refers to the sharing of feature parts in the backbone modules of a multi-task learning network.

[0056] In some embodiments, a neural network architecture for simultaneous detection and segmentation prediction is provided, such as a Real-Time Deep Network for Scene Understanding (also known as BlitzNet). This network architecture is designed based on the assumption that neural network models for different visual tasks undergo a process of moving from shallow features to high-level features and from detailed features to abstract features during feature extraction. Therefore, the basic feature extraction modules of neural network models for different tasks are redundant. The framework first includes a shared backbone neural network structure for multiple tasks; then, each task extracts image features from one or more locations within the backbone neural network; the output features of the backbone neural network, or the output features after simple processing, are fed into task-independent head structures to obtain the prediction results for each visual task. However, the backbone features in this method are completely shared, only considering task similarity and not task differences in backbone feature extraction, thus failing to obtain coherent, task-independent high-level features.

[0057] In other embodiments, a different network architecture diagram for multi-task processing is provided, such as a multi-task learning framework based on multi-scale task interaction. The multi-task learning algorithm used in this network architecture can reintegrate the prediction results into the multi-task learning process through distillation to obtain new prediction results, which are expected to be superior to the original prediction results. Thus, although this method performs secondary processing on the initial prediction results, incorporating task information with clear representational meaning as input into the multi-task model, it can significantly improve prediction accuracy. However, this multi-task processing method is a correlation method based on secondary prediction, directly feeding the primary original prediction results into the convolutional neural network, increasing the neural network size and computational burden of secondary prediction.

[0058] As mentioned above, in order to overcome the shortcomings of the two multi-task processing methods provided in the above embodiments, how to integrate multiple single-task models into a unified multi-task model, while maintaining task accuracy, and reducing the computing power and power consumption requirements of terminal devices has become the main problem to be solved.

[0059] To address the aforementioned issues, embodiments of this application provide a multi-task processing method, which offers a visual multi-task learning framework with a partially shared structure. First, this framework incorporates a novel multi-task backbone feature extraction module. This module balances the model capacity of task-irrelevant and task-related features, establishing a process where the multi-task model capacity allocation gradually changes with model depth, achieving end-to-end progressive task association decoupling. Second, the framework employs a mask-based two-level interaction structure. This structure mines complementary features from both the multi-task backbone features and the initial prediction results, improving information interaction efficiency and multi-task prediction accuracy.

[0060] Thus, compared with the aforementioned BlitzNet-based and multi-scale task interaction-based multi-task processing methods, the multi-task processing method based on a partially shared structure provided in this application differs in the following ways: Firstly, the partially shared structure provided in this application, while maintaining the same computational load, decomposes the multi-task backbone feature extraction module into layer-by-layer task-independent convolutions and task-related convolutions, achieving end-to-end task decoupling. Furthermore, utilizing the partially shared structure, the abstract features of the previous-level task-independent convolutions are integrated into the task-related convolutions of the current layer; computational power allocation dynamically changes with model depth; this partially shared structure, as a basic structure, can be directly integrated into any visual task model. Secondly, the mask-based two-level interaction structure in this application can divide the multi-task information interaction process into basic feature interaction and multi-task collaborative enhancement. Basic feature interaction uses task attention to supplement the backbone features of other tasks into the backbone features of the current task; while the multi-task collaborative enhancement layer first transforms the initial prediction results of other tasks into task masks, and then fuses the masks with the initial prediction results of the current task to obtain the multi-task prediction results.

[0061] In some embodiments, electronic devices suitable for the multitasking processing method of this application include, but are not limited to, intelligent terminal devices with relatively weak computing power such as mobile phones, automobiles, televisions, and smart glasses based on intelligent sensing, and can also be applied to cloud products such as cloud applications implemented based on cloud servers. As an example, the apparatus or module for executing the multitasking processing method in this application can be deployed in software form on the computing node of an electronic device to improve the running efficiency of the multitasking visual model. Therefore, the executing entity of the multitasking processing method provided in the embodiments of this application can be an electronic device, or software or functional modules within an electronic device used to execute the multitasking processing method.

[0062] In some embodiments, the application scenarios of the multi-task processing method provided in this application include, but are not limited to: parallel processing of multiple tasks such as image scene recognition, object detection, object segmentation, and edge detection in various application scenarios such as mobile phone photography, mobile phone image editing, autonomous driving scene perception, and security monitoring.

[0063] Reference Figure 1 The diagram shown is a network architecture diagram for multitasking provided in an embodiment of this application. Figure 1 The network architecture 30 includes an input image, a backbone module 31 based on a partially shared structure, a mask-based two-stage interaction module 32, and an output result. Furthermore, the network architecture 30 can be deployed in an electronic device executing the multi-task processing method of this application, thus serving as the execution entity of the multi-task processing method.

[0064] in, Figure 1 The input image can be an RGB image, but is not limited to this.

[0065] The backbone module 31, based on a partially shared structure, is used to extract multi-task shared features and task-independent features from the input image. It is understood that, in this embodiment, multi-task shared features can also be called task-independent features, and task-independent features can also be called task-related features.

[0066] As an example, for a multi-task task consisting of object detection, image segmentation, and image edge detection: task-independent features can be implicit image features that are commonly required by multiple tasks, such as color, texture, edge, and angular information of the image; while task-related features can be implicit image features required by a single task, such as semantic features of the image, for example, the category of the object of interest in the image.

[0067] The mask-based secondary interaction module 32, based on the task-independent features and task-related features of each task provided by module 31, performs mask-based secondary feature interaction between the task-independent features (or multi-task shared features) and the task-related features (or task-independent features) to obtain the output result.

[0068] Among them, the output results of the two-pole interaction module 32 based on the mask are the output results of the multiple tasks. For example, the output result of task 1 is the bounding box detection result, the output result of task 2 is the image segmentation result, and the output result of task 3 is the image edge detection result.

[0069] Furthermore, refer to Figure 2 right Figure 1 The backbone module 31 based on a partially shared structure and the secondary interaction module 32 based on a mask in the network architecture shown are described in detail.

[0070] like Figure 2 As shown, the backbone module 31, based on a partially shared structure, effectively extracts both task-related and task-independent backbone features. This module consists of multiple (e.g., N) partially shared convolutional layers. Each partially shared convolutional layer allocates computational power between task-related and task-independent convolutions according to a pre-defined stripping coefficient. The stripping coefficient monotonically increases with model depth, gradually reducing the computational power proportion of task-independent convolutions in the partially shared convolutional layers. Task-independent convolutions only accept the results of their parent task-independent convolutions as input; task-related convolutions stack the results of their parent task-related and task-independent convolutions as input. The module's function is to effectively extract both task-related and task-independent backbone features.

[0071] In some embodiments of this application, for multiple partially shared layers in a backbone module based on a partially shared structure, the initial layer stripping coefficient can be set to 0.05 and the final layer stripping coefficient to 0.33, but it is not limited to this.

[0072] As an example, taking multi-task processing involving two tasks as an example, assuming that the two tasks correspond to multiple partially shared layers that are divided into task-related convolutions and task-independent convolutions, and that the computing power is evenly distributed, then the stripping coefficient of each partially shared layer is taken as 0.33.

[0073] Furthermore, the backbone module 31 based on the partial shared structure may also include a fusion layer 1, which is used to stack the task-independent features output by one or more partial shared convolutions in the backbone module 31 based on the partial shared structure with the task-related features of the corresponding task in the channel dimension, and obtain the backbone features corresponding to each task through the fusion layer 1.

[0074] The mask-based secondary interaction module 32 is used to improve the efficiency of multi-task information interaction and the accuracy of multi-task prediction. This module consists of two sub-modules: a basic feature interaction sub-module 321 and a multi-task collaborative enhancement sub-module 322. For each task, the basic feature interaction sub-module 321 uses a task attention mechanism to fuse the backbone features of other tasks and stack them into the backbone features of the current task, which are then passed through a primary predictor to obtain the primary prediction result of the current task. For each task, the multi-task collaborative enhancement sub-module 322 first converts the primary prediction results of other tasks into spatial mask form. Each task mask is multiplied by the primary prediction result of the current task, and then multiplied and added using a set of learnable weight parameters, resulting in an enhanced prediction result that fuses the primary prediction results of multiple tasks. The enhanced prediction result is then input into the secondary predictor, and its result is added to the primary prediction result to obtain the final prediction result of the current task.

[0075] It should be noted that, Figure 2 The structure of a single task is represented in the image. In some embodiments, the mask-based secondary interaction module 32 described above includes multiple basic feature interaction submodules 321 and multiple multi-task collaborative enhancement submodules 322, with each task corresponding to one basic feature interaction submodule 321 and one multi-task collaborative enhancement submodule 322. For example, Figure 2 The basic feature interaction submodule 321 and the multi-task collaboration enhancement submodule 322 shown can correspond to any one of task 1, task 2 or task 3.

[0076] More specifically, the basic feature interaction submodule 321 includes task attention 33, fusion layer 2, and primary predictor 34. Fusion layer 2 is used to stack the backbone features of other tasks besides the current task in the channel dimension, and obtain fused feature 1 after passing through fusion layer 2. Task attention module 33 is used to input the backbone features of the current task to obtain channel-dimensional attention with a spatial dimension of 1x1, which is the same as the channel dimension of fused feature 1. Primary predictor 34 is used to input fused feature 2 to obtain the primary prediction result of the current task. Specifically, the channel-dimensional attention output by task attention module 33 is multiplied by the fused feature 1 output by fusion layer 2, and then stacked with the backbone features of the current task in the channel dimension to obtain fused feature 2.

[0077] In some embodiments, Figure 2In the basic feature interaction submodule 321 shown, f1, f2, and f3 represent the backbone features corresponding to Task 1, Task 2, and Task 3, respectively, and these backbone features are output by the backbone module 321 based on a partial sharing structure. f2' represents the fused feature (i.e., fused feature 2 below) obtained by fusing the backbone features f3 corresponding to Task 3 and f1 corresponding to Task 1 with the backbone feature f2 corresponding to Task 2. Furthermore, in the multi-task collaborative enhancement submodule 322, p1, p2, and p3 represent the primary prediction results corresponding to Task 1, Task 2, and Task 3, respectively, and these prediction results are output by the basic feature interaction submodule 321. The p1 mask transformation layer is used to obtain the mask m1 of the primary prediction result p1, and the p3 mask transformation layer is used to obtain the mask m3 of the primary prediction result p3. w1 and w3 are the learnable weights corresponding to masks m1 and m3, respectively. O2 represents the output result corresponding to Task 2.

[0078] In some embodiments, the fusion layer 1, fusion layer 2, and attention module 33 may be a combination of one or more basic neural network layers such as convolutional layers, linear layers, activation layers, and pooling layers, but are not limited thereto.

[0079] It is understandable that the above Figure 1 or Figure 2 The network architecture 30 shown can be used to execute the multitasking method provided in this application. This network architecture 30 can be deployed on terminal devices such as mobile phones and smart glasses, or in cloud products. In the following embodiments, the various modules in the multitasking network architecture shown in Figure 8 are mainly used as the execution entities to describe the multitasking method provided in the embodiments of this application.

[0080] Next, combined Figure 2 The multi-task processing network architecture 30 shown is referenced. Figure 3 As shown, the flow of the multi-task processing method provided in this application embodiment is described in detail. The execution entities of this method flow are the various modules in the network architecture 30. The method flow includes two stages: the first stage is to input the input image into the backbone module 31 based on a partial shared structure to obtain multi-task backbone features; the second stage is to input the multi-task backbone features into the mask-based secondary interaction module 32 to obtain the output results of each task. Specifically, the method flow includes the following steps:

[0081] S501: The backbone module 31 based on the partial shared structure inputs the input image into the first partial shared layer, and outputs a task-independent feature and a task-related feature corresponding to each task after passing through the first partial shared layer.

[0082] S502: The backbone module 31 based on the partial shared structure inputs the task-independent features of level i-1 into the task-independent convolution of the i-th partial shared layer and outputs the task-independent features of level i; stacks the task-independent features of level i-1 and the task-related features of level i-1 in the channel dimension and inputs them into the task-related convolution of the i-th partial shared layer and outputs the task-related features of level i, until the N-th partial shared layer is reached.

[0083] It is understandable that the backbone module 31 based on the partial shared structure includes N layers of partial shared layers (or partial shared convolutions), and the Nth partial shared layer is the last partial shared layer. 2≤i≤N, and i is a positive integer. The (i-1)th level task-independent features and (i-1)th level task-related features are the output of the (i-1)th level partial shared layer.

[0084] In other words, the task-independent features of the previous level are output as the task-independent convolution of the current layer's partial shared convolution, while the task-independent features of the previous level and the task-related features of the previous level are stacked in the channel dimension and input into the task-related convolution of the current layer's partial shared convolution.

[0085] S503: For each task, the backbone module 31 based on the partial shared structure stacks the task-independent features output by one or more partial shared layers in the backbone module 31 based on the partial shared structure with the task-related features of the corresponding task in the channel dimension, and obtains the backbone features corresponding to each task through the fusion layer 1.

[0086] As an example, in this application, the task-independent features output by the last partially shared layer and the task-related features of the corresponding task can be stacked in the channel dimension and then passed through fusion layer 1 to obtain the backbone features corresponding to each task.

[0087] It is understandable that S501 to S503 are used to implement the first stage of multi-task processing.

[0088] Thus, the backbone module based on the partially shared structure in this application can achieve end-to-end decoupling of task-related and task-independent features of the multi-task backbone features, and improve the computing power allocation strategy of the backbone module by using a stripping coefficient, thereby effectively improving the multi-task prediction accuracy.

[0089] S504: The backbone module 31 based on the partially shared structure sends the backbone features corresponding to each task to the basic feature interaction submodule 321.

[0090] S505: The basic feature interaction submodule 321 takes the backbone features of multiple tasks as input, passes the backbone features of each task through the fusion layer 2, the task attention module 33 and the primary predictor 34 to obtain the primary prediction result of a task, and outputs the primary prediction results of each task.

[0091] As an example, refer to Figure 4The diagram illustrates the specific process by which the basic feature interaction submodule 321 outputs the initial prediction result for the current task. Similarly, this process can be followed to obtain the initial prediction result for each of multiple tasks. Specifically, as shown... Figure 4 As shown, the process of generating the primary prediction result of the current task based on the backbone features of the current task can be implemented through the following steps S5051 to S5054, that is, S505 includes S5051 to S5054:

[0092] S5051: The basic feature interaction submodule 321 stacks the backbone features of other tasks besides the current task in the channel dimension, and obtains the fused feature 1 through the fusion layer 2.

[0093] As an example, refer to Figure 2 The backbone features f1 of Task 1 (excluding Task 2) and f3 of Task 3 can be stacked in the channel dimension and then passed through the fusion layer 2 to obtain fused feature 1.

[0094] S5052: The basic feature interaction submodule 321 passes the backbone features of the current task through the task attention module 33 to obtain channel-dimensional attention with a spatial dimension of 1x1.

[0095] Among them, the channel dimension of the channel dimension attention mentioned above is the same as the channel dimension of fusion feature 1, that is, both are 1x1 channel dimensions.

[0096] As an example, refer to Figure 2 The main feature f2 of the current task 2 can be passed through the task attention module 33 to obtain the channel dimension attention of the current task 2.

[0097] S5053: The basic feature interaction submodule 321 multiplies the channel dimension attention of the current task with the fused feature 1, and stacks it with the backbone feature of the current task in the channel dimension to obtain the fused feature 2.

[0098] As an example, refer to Figure 2 The channel dimension attention of the current task 2 can be multiplied with the fusion feature 1, and then stacked with the backbone feature f2 of the current task 2 in the channel dimension to obtain the fusion feature 2 (i.e. f2').

[0099] S5054: The basic feature interaction submodule 321 inputs the fused feature 2 into the primary predictor 34 to obtain the primary prediction result for the current task. Similarly, the basic feature interaction submodule 321 can obtain the primary prediction results for multiple tasks according to the same steps.

[0100] As an example, refer to Figure 2 The fused feature 2 can be input into the primary predictor 34 to obtain the primary prediction result of the current task 2 (i.e., p2 in the following text).

[0101] Next, return Figure 3 Continuing with step S505:

[0102] S506: The basic feature interaction submodule 321 sends the initial prediction results of each task to the multi-task collaborative enhancement submodule 322.

[0103] S507: The multi-task collaborative enhancement submodule 322 takes the primary prediction results of each task as input, and passes the primary prediction results of each task through the p1 mask converter, p2 mask converter and secondary predictor 35 to obtain an output result, so as to output the output results of each task.

[0104] As an example, refer to Figure 5 As shown, the specific process of the multi-task collaboration enhancement submodule 322 detecting the output result of the current task is illustrated. Similarly, the final output result of each task in multiple tasks can be obtained by following this process, with the current task being task 2 as an example. Specifically, as... Figure 5 As shown, the process of generating the secondary prediction result of the current task based on the primary prediction result of the current task, and then obtaining the output result, can be implemented by the following S5071 to S5074, that is, the above S507 includes S5071 to S5074:

[0105] S5071: Multi-task collaborative enhancement submodule 322 obtains the masks for each other task by passing the primary prediction results of other tasks besides the current task through the mask transformation layer corresponding to each task.

[0106] For example, refer to Figure 2 As shown, the current task is Task 2. The multi-task collaboration enhancement submodule 322 includes a p1 mask converter corresponding to Task 1 and a p3 mask converter corresponding to Task 3. Furthermore, as... Figure 2 As shown, the initial prediction result p1 of Task 1 is transformed into the mask m1 of Task 1 through the corresponding p1 mask transformation layer, and the initial prediction result p3 of Task 3 is transformed into the mask m3 of Task 3 through the corresponding p3 mask transformation layer.

[0107] In some embodiments, different tasks are converted into masks in different ways, commonly including channel dimension weighted average, numerical range normalization, etc., but not limited to these.

[0108] Furthermore, the spatial dimension of the mask for each of the above tasks is the same as the spatial dimension of the initial prediction result of the current task, that is, the channel dimension of the mask is 1.

[0109] S5072: Multi-task collaborative enhancement submodule 322 multiplies the masks of each other task besides the current task with the primary prediction result of the current task to obtain the prediction result after masking.

[0110] For example, refer to Figure 2 As shown, the current task is task 2. The mask m1 of task 1 can be multiplied by the initial prediction result p2 of the current task 2 to obtain the prediction result after masking task 1. The mask m3 of task 3 can be multiplied by the initial prediction result p2 of the current task 2 to obtain the prediction result after masking task 3.

[0111] S5073: The multi-task collaborative enhancement submodule 322 multiplies and adds the prediction results of each other task after masking them according to the learnable weights of each other task to the current task, and then adds them to the initial prediction result of the current task to obtain the fused prediction result.

[0112] For example, refer to Figure 2 As shown, the current task is Task 2, the learnable weights from Task 1 to Task 2 are w1, and the learnable weights from Task 3 to Task 2 are w3. Therefore, the masked prediction result of Task 1 is multiplied and added together with the learnable weights w1 of Task 2, and then added to the initial prediction result p2 of Task 2 to obtain the fused prediction result of Task 1. Similarly, the masked prediction result of Task 3 is multiplied and added together with the learnable weights w3 of Task 2, and then added to the initial prediction result p2 of Task 2 to obtain the fused prediction result of Task 3.

[0113] S5074: The multi-task collaborative enhancement submodule 322 inputs the fusion prediction results of various other tasks into the secondary predictor 35 to obtain the secondary prediction result of the current task.

[0114] As an example, refer to Figure 2 The fused prediction results can be input into the secondary predictor 35 to obtain the secondary prediction results for the current task 2.

[0115] S5075: Multi-task collaborative enhancement submodule 32 adds the secondary prediction result of the current task to the primary prediction result of the current task to obtain the output result of the current task.

[0116] As an example, refer to Figure 2 The secondary prediction result of the current task 2 can be added to the primary prediction result p2 of the current task 2 to obtain the output result o2 of the current task 2.

[0117] Similarly, referring to the above description of the processing of the current task 2, tasks 1 and 2 can be processed separately to obtain the output results of each task, that is, the final prediction results.

[0118] Thus, the multi-task processing method provided in this application can decompose convolutional layers into task-independent convolutions and task-related convolutions based on a partially shared structure, achieving end-to-end task decoupling. Furthermore, by utilizing the partially shared structure, the abstract features of the previous layer's task-independent convolutions are integrated into the current layer's task-related convolutions. This partially shared structure, as a foundational structure, can be directly integrated into any visual task model, improving the adaptability of the multi-task processing in different scenarios and avoiding the need to compute task-independent features for each task. In addition, the mask-based two-level interaction structure in this application allows the multi-task information interaction process to use task attention to supplement the backbone features of other tasks into the backbone features of the current task. This first transforms the initial prediction results of other tasks into task masks, and then fuses the masks with the initial prediction results of the current task to obtain the multi-task prediction results, achieving efficient multi-task processing. Therefore, it improves the efficiency of multi-task interaction and the accuracy of multi-task prediction.

[0119] According to some embodiments of this application, the backbone module based on the partially shared structure provided in this application can be implemented based on HRNet18 to support the backbone module of the partially shared structure according to the layer-by-layer task-independent convolution and task-related layer convolution.

[0120] Reference Figure 6 The diagram shown is a typical structural diagram of HRNet18. Figure 6 As shown, the overall structure of the HRNet18 backbone is divided into four stages. Stages 1 through 4 each include branches, and each branch includes at least one convolutional layer (Conv) and one convolutional block (Conv block). Furthermore, the convolutional layer and convolutional block in each branch are used to extract different features from the input image, and the resolution of the features is reduced by a factor of 2.

[0121] Specifically, Stage 1 contains one branch, which includes a convolutional layer and a convolutional module. Stage 1 reduces the resolution of the input image by a factor of 4 to obtain the basic feature F-S1-1. Then, it uses a convolution with a stride of 2 to reduce the spatial resolution of feature F-S1-1 by half to obtain feature F-S1-2. Stage 2 contains two branches. Branch 1 (the first branch above) directly uses feature F-S1-1 as input to obtain feature F-S2-1; Branch 2 directly uses feature F-S1-2 as input to obtain feature F-S2-2. Then, Stage 2 uses a convolution with a stride of 2 to obtain feature F-S2-3. Stage 3 contains 3 branches. Before entering the branch structure, Stage 3 passes features F-S2-1,2,3 through a multi-scale fusion layer (HR blocks) to obtain features F-S3-R-1,2,3, which are then fed into the 3 branches to obtain F-S3-1,2,3 respectively. Then, Stage 3 passes F-S3-3 through a convolution with stride = 2 to obtain features F-S3-4. Stage 4 contains 4 branches. Before entering the branch structure, Stage 4 passes F-S3-1,2,3,4 through a multi-scale fusion layer (HR blocks) to obtain features F-S4-R-1,2,3,4, which are then fed into the 4 branches to obtain features F-S4-P-1,2,3,4 respectively. Then, Stage 4 inputs F-S4-P-1,2,3,4 into another multi-scale fusion layer for one round of multi-scale fusion to obtain features F-S4-1,2,3,4.

[0122] It is understandable that the above Figure 6 The branch numbers in each stage shown refer to the branches arranged from top to bottom.

[0123] Furthermore, each convolutional module in each branch of stages 2 to 4 above can be followed by a convolutional layer (Conv), although Figure 6 Not shown in the image.

[0124] In some embodiments, the backbone module based on a partially shared architecture provided in this application can be implemented by modifying the traditional HRNet18, but is not limited thereto.

[0125] Reference Figure 7 As shown, this is the partially shared HRNet18 backbone module provided in this application. HRNet18 is a... Figure 6 The HRNet18 shown is obtained through modification.

[0126] In some embodiments, with Figure 6 Compared to HRNet18, firstly, Figure 7The HRNet18 shown does not require modification to the overall structure; the basic convolutional layers (conv), batch normalization layers (BN), activation layers (ReLU), and pooling layers can be replaced, as shown in the example. Figure 7 Each branch contains "PS-Conv" and "PS-Conv Blocks". That is, in... Figure 7 Lieutenant General Figure 6 In each branch, the convolutional layer "Conv" is replaced with the convolutional layer "PS-Conv". The convolutional modules "Conv Blocks" are then modified by replacing the convolutional layers "Conv", batch normalization (BN) layers, ReLU activation layers, and pooling layers. This results in the convolutional modules "PS-Conv Blocks". Next, [the text abruptly ends here, likely due to an incomplete sentence or missing information]. Figure 6 Compared to HRNet18 shown, Figure 7 The HRNet18 shown also includes a Fusion Conv Group layer, used to fuse task-independent and task-related features from various tasks into the backbone features for that task. Correspondingly, Figure 7 The features F-S4-1,2,3,4 shown can be fused through various fusion convolutional group layers to obtain features F-S4-1',2',3',4'.

[0127] Understandable, Figure 7 Each of the features shown in F-S4-1', 2', 3', 4' includes some of the core features of each task in the multi-task, so features F-S4-1', 2', 3', 4' are used to represent the core features of each task in the multi-task, such as the core features of each task in tasks 1-3.

[0128] Furthermore, refer to Figure 8A The diagram shown is a schematic of convolutional layer replacement provided in an embodiment of this application.

[0129] in, Figure 8A The above is Figure 6 The original convolutional layer is shown to have input and output features. This convolutional layer cannot distinguish between task-related and task-independent features, and can only perform convolution operations on one type of input feature to obtain the corresponding output feature. Figure 8A Below is a schematic diagram of the replaced convolutional layer "PS-Conv" structure and its input and output in this application. Specifically, the input features of the convolutional layer "PS-Conv" can include task-related features and task-independent features. For example, from top to bottom, the input features are the task-related features corresponding to task 1, the task-related features corresponding to task 2, the task-related features corresponding to task 3, and task-independent features. It should be noted that... Figure 8AThe medium gray blocks represent task-independent features. For example, the last block in the input features refers to task-independent features, and the last block in the output features also refers to task-independent features.

[0130] Figure 8A The structure of the convolutional layer "PS-Conv" shown below includes multiple convolutional layers corresponding to different tasks, specifically the convolutional layer for task 1 (Task1Conv), the convolutional layer for task 2 (Task2Conv), the convolutional layer for task 3 (Task3Conv), and the control convolutional layer "MasterConv". "MasterConv" represents task-independent convolution.

[0131] like Figure 8A As shown below, the convolutional layers corresponding to each task can take in task-related features and task-independent features respectively. Specifically, the task-related features and task-independent features are stacked and then input into the convolutional layers corresponding to each task. The input of task-independent features is controlled by the "MasterConv" convolutional layer.

[0132] Furthermore, in the convolutional layer "PS-Conv", the convolutional layer corresponding to each task outputs the corresponding task-related features, and controls the convolutional layer "MasterConv" to output task-independent features, thereby obtaining the output features of the convolutional layer "PS-Conv".

[0133] Understandable, Figure 8A The method for replacing the convolutional module shown is applicable to Figure 6 The convolutional layers shown are located at any position in each branch of HRNet18, such as convolutional layers near the input or convolutional layers in convolutional modules, etc. This application does not make specific limitations on them.

[0134] Thus, the backbone module based on the partially shared structure in this embodiment can achieve end-to-end task association decoupling, and integrate the abstract features of the previous task-independent convolution into the next task-related convolution, which is beneficial to finally obtain richer task backbone features.

[0135] Reference Figure 8B The diagram shown is a replacement illustration for a convolutional module provided in this application. Specifically, Figure 8B The diagram illustrates the replacement of BN, ReLU, or Pool layers in the convolutional module. "MasterConv" represents task-independent convolution, and gray blocks represent task-independent features.

[0136] in, Figure 8B The above is Figure 6The input and output features of the BN layer, ReLU layer, or Pool layer in the original convolutional module are shown. This layer cannot distinguish between task-related features and task-independent features, and can only perform convolution operations on one type of input feature to obtain the corresponding output feature.

[0137] Figure 8B The diagram below illustrates the relationship between the BN layer, ReLU layer, or Pool layer in the replaced convolutional layer of this application and the input and output. Specifically, the input features of the BN layer, ReLU layer, or Pool layer may include task-related features and task-independent features. For example, from top to bottom, the input features are the task-related features corresponding to task 1, the task-related features corresponding to task 2, the task-related features corresponding to task 3, and task-independent features.

[0138] Figure 8B The BN, ReLU, or Pool layers shown below include convolutional layers corresponding to multiple tasks, specifically the network layer corresponding to Task 1 (Task1 BN / ReLU / Pool), the network layer corresponding to Task 2 (Task2 BN / ReLU / Pool), the network layer corresponding to Task 3 (Task3 BN / ReLU / Pool), and the control network layer (Master BN / ReLU / Pool).

[0139] like Figure 8B As shown below, the convolutional layers corresponding to each task can respectively input task-related features and task-independent features. Specifically, the task-related features are input into the network layers corresponding to each task, and the task-independent features are input into the control network layer.

[0140] Furthermore, the network layers corresponding to each task in the convolutional module "PS-Conv Blocks" output corresponding task-related features, and control the network layers to output task-independent features, thereby obtaining the output features of the convolutional module "PS-Conv Blocks".

[0141] Understandable, Figure 8B The method for replacing the convolutional module shown is applicable to Figure 6 The BN layer, ReLU layer, and Pool layer shown are located at any position in each branch of HRNet18, and this application does not make specific limitations on them.

[0142] According to some embodiments of this application, Figure 2 The partial shared layers shown are mainly through Figure 7 The convolutional layer implementations in the various branches shown are examples, but not limited to these.

[0143] In addition, it should be noted that, Figure 8A The convolutional layer shown or the convolutional layer in the convolutional module usually refers to Figure 7 The second convolutional layer and subsequent convolutional layers in the partially shared backbone module are shown.

[0144] According to some embodiments of this application, refer to Figure 9A The diagram shown illustrates the input and output of the initial convolutional layer or convolutional module. For example... Figure 9A The input to the initial convolutional layer is the input image. Therefore, when applying a partially shared architecture, no additional processing of the input is required; the output only needs to be divided into task-related features and task-independent features. For example, the convolutional layers corresponding to Task 1 (Task1Conv), Task 2 (Task2Conv), Task 3 (Task3Conv), and the control convolutional layer "MasterConv" all take the original input image as input and output the task-related features for Task 1, Task 2, Task 3, and task-independent features, respectively.

[0145] In addition, for Figure 7 The fused convolutional group layer shown can be implemented using convolutional layers. Specifically, Figure 7 After each sampling location of the backbone features shown, a set of convolutional layers needs to be added as fusion convolutional group layers to stack task-independent features and task-related features of each task, and then pass them through the convolutional layer of the corresponding task to obtain the fused features as the backbone features of that task.

[0146] Reference Figure 9B The diagram shown illustrates the structure and input / output of a fused convolutional layer according to an embodiment of this application. Figure 9B As shown, the fusion convolutional layer includes multiple convolutional layers, such as the convolutional layer corresponding to Task 1 (Task1Conv), the convolutional layer corresponding to Task 2 (Task2Conv), and the convolutional layer corresponding to Task 3 (Task3Conv). The input sampling point features also include task-related features and task-independent features. For example, the task-related and task-independent features corresponding to Tasks 1-3 are stacked and then input into the corresponding task's convolutional layer. Furthermore, the convolutional layer corresponding to each task outputs the backbone features corresponding to each task, i.e., the backbone features corresponding to Tasks 1-3 respectively. As an example, combined with... Figure 7 As shown, the output features F-S4-1, 2, 3, and 4 in stage 4 are the features of the sampling points. Each feature in F-S4-1, 2, 3, and 4 is input into a fusion convolutional layer to obtain output features F-S4-1', 2', 3', and 4' containing partial backbone features of each task in tasks 1-3. Furthermore, the backbone features corresponding to task 1 are composed of task-related features from the output features F-S4-1', 2', 3', and 4'.

[0147] In the following embodiments, image semantic segmentation, human body part segmentation, and saliency detection are taken as tasks 1 to 2 for multi-task processing, respectively, to illustrate the multi-task processing method provided in the embodiments of this application.

[0148] exist Figure 2 Based on the network architecture 30 shown, and referring to Figure 10 The diagram shown is a structural schematic of the basic feature interaction submodule 321 provided in this embodiment of the application. Figure 10 The example only shows the basic feature interaction submodule 321 corresponding to Task 2. Similarly, the basic feature interaction submodules 321 corresponding to Task 1 and Task 3 can be found in [reference needed]. Figure 10 Examples are provided, but will not be elaborated upon.

[0149] like Figure 10 As shown, the task attention module 33 includes an adaptive average pooling layer and two linear layers. In some embodiments, the linear layers can be implemented using convolutional layers, such as 1×1 convolutional layers, for linear processing of the input features.

[0150] The primary predictor 34 consists of a network layer composed of Conv-BN-ReLU and independent convolutional layers (Conv) to estimate the primary prediction results for Task 2.

[0151] also, Figure 10 The fusion layer 2 shown can be implemented using a convolutional layer (Conv).

[0152] exist Figure 2 Based on the network architecture 30 shown, and referring to Figure 11 The diagram shown is a structural schematic of the multi-task collaboration enhancement submodule 322 provided in this embodiment of the application. Figure 11 The example shown only includes the multi-task collaboration enhancement submodule 322 corresponding to Task 2. Similarly, the multi-task collaboration enhancement submodules 322 corresponding to Tasks 1 and 3 can be found in [reference needed]. Figure 11 Examples are provided, but will not be elaborated upon.

[0153] like Figure 11 As shown, the p1 mask converter for Task 1 can be implemented using a normalization operation, and the p3 mask converter for Task 3 can also be implemented using a normalization operation, but it is not limited to this. Of course, in practical applications, since Task 3 is saliency detection (such as edge detection), the initial prediction result p1 for Task 3 can be directly used as the mask m3 for Task 1. It can be understood that the normalization operation normalizes the pixels in the image represented by the initial prediction result to pixel values ​​of 0-1, thereby obtaining the output mask.

[0154] also, Figure 11The bipolar predictor 35 in the multi-task collaborative enhancement submodule 322 shown can be a network layer composed of Conv-BN-ReLU.

[0155] Next, combined Figure 7 and 12 The multi-task processing network architecture 30 shown takes image semantic segmentation (task 2) as an example. Figure 12 This paper provides a schematic diagram of the process for generating primary prediction results in a multi-task processing method. The main body executing this process is the basic feature interaction submodule 321 corresponding to the image semantic segmentation task, and the process includes the following steps:

[0156] S1401: The basic feature interaction submodule 321 stacks the backbone features of the other two tasks besides the current task 2 in the channel dimension, and then passes them through a fusion layer 2 constructed by a single convolutional layer to obtain fused feature 1.

[0157] Reference Figure 10 As shown, the basic feature interaction submodule 321 stacks the backbone features f1 of Task 1 and f3 of Task 3 in the channel dimension, and then passes them through the fusion layer 2 to obtain the fused feature 1.

[0158] S1402: The basic feature interaction submodule 321 obtains the channel-dimensional attention with a spatial dimension of 1x1 by passing the backbone feature f2 of the current task 2 through the pooling layer and convolutional layer to construct the task attention module 33.

[0159] It is understandable that the channel dimension attention of the basic feature interaction submodule 321 can indicate the backbone features corresponding to the current task 2, and can be used to characterize the correlation between the backbone features of other tasks and the current task 2.

[0160] S1403: The basic feature interaction submodule 321 multiplies the channel dimension attention with the fusion feature 1, and the result is stacked with the backbone feature f2 of the current task 2 to obtain the fusion feature 2.

[0161] Thus, the channel-dimensional attention is multiplied by fusion feature 1, and features irrelevant to the current task are filtered from the backbone features of other tasks based on their relevance to the current task. In other words, fusion feature 2 filters out features irrelevant to the current task 2 from the backbone features of tasks 1 and 3.

[0162] S1404: The basic feature interaction submodule 321 inputs the fused feature 2 into the primary predictor 34 of the current task 2 to obtain the primary prediction result of the current task 2.

[0163] Similarly, the same process S1401 to S1404 is used for human body segmentation (i.e., Task 1) and saliency detection (i.e., Task 3), where the input is replaced with the corresponding current task and other tasks.

[0164] Thus, the basic feature interaction submodule in this application can utilize the task attention mechanism to obtain the correlation between the backbone features of other tasks and the current task, filter out irrelevant features, and improve the efficiency of multi-task interaction and the accuracy of multi-task prediction.

[0165] Next, combined Figure 7 and 11 The multi-task processing network architecture 30 shown takes image semantic segmentation (task 2) as an example. Figure 13 This paper provides a schematic diagram of the process for generating secondary prediction results in a multi-task processing method. The main body executing this process is the multi-task collaborative enhancement submodule 322 corresponding to the image semantic segmentation task, and the process includes the following steps:

[0166] S1501: The multi-task collaborative enhancement submodule 322 obtains the task mask by passing the primary prediction results of the other two tasks besides the current task 2 through the mask transformation layer. For example, the results of human body part segmentation and saliency detection are normalized as the task mask.

[0167] S1502: Multi-task collaborative enhancement submodule 322 multiplies the masks of the other two tasks besides the current task 2 with the primary prediction result of the current task 2 to obtain the prediction result after masking.

[0168] S1503: The multi-task collaborative enhancement submodule 322 sums the masked prediction results by weighting them with the corresponding learnable weights W1-2 and W3-2, and then adds them to the primary prediction results of the current task 2 to obtain the fused prediction results.

[0169] S1504: The multi-task collaborative enhancement submodule 322 inputs the fused prediction results into the secondary predictor 35 of the current task to obtain the secondary prediction results of the current task 2.

[0170] S1505: The multi-task collaborative enhancement submodule 322 adds the secondary prediction result of the current task 2 to the primary prediction result of the current task 2 to obtain the output result of the current task.

[0171] Similarly, the same process S1501 to S1505 is used for human body segmentation (task 1) and saliency detection (task 3), wherein the input is replaced with the corresponding current task and other tasks, and the learnable weights are replaced with the corresponding weights. This application embodiment will not elaborate further on this.

[0172] In this way, the multi-task collaboration enhancement submodule can convert the initial prediction results of other tasks into task masks, and then fuse the masks with the initial prediction results of the current task, thereby improving the efficiency of multi-task interaction and the accuracy of multi-task prediction.

[0173] It should be noted that the network architecture used in the multi-task processing method provided in this application embodiment includes, but is not limited to, the above-described network architecture. Figure 2 The network architecture 30 shown is composed of the backbone module 31 based on the partially shared structure, the basic feature interaction submodule 321, and the multi-task collaboration enhancement submodule 322. In other words, some embodiments of this application provide multi-task processing methods based on the aforementioned network architecture 30.

[0174] In other embodiments, the multi-task processing method provided in this application can be partially improved based on the existing multi-task processing network architecture. For example, the multi-task processing network structure provided in this application can be obtained by deploying one or two of the following on the existing multi-task processing structure: a backbone module 31 based on a partially shared structure, a basic feature interaction submodule 321, and a multi-task collaboration enhancement submodule 322. That is to say, the multi-task processing method in some embodiments of this application can also be partially improved based on the existing multi-task processing method.

[0175] In some embodiments, the existing multitasking methods involved in the multitasking performance testing of this application include the following:

[0176] SingleTask: A method where each task uses an independent model and is trained independently.

[0177] BaseMultiTask: Without additional task processing, it directly uses a shared backbone module and connects multiple task header structures.

[0178] Cross-stitch: A method that retains a separate core module for each task, but designs an interaction structure between the core modules of each task.

[0179] Neural Discriminative Dimensionality Reduction-CNN (NDDR-CNN): A method that retains a separate backbone module for each task, but designs an interaction structure between the backbone modules of each task. The interaction structure design is different from CrossStitch.

[0180] Multi-Task Attention Network (MTAN): A method that uses a backbone module that is fully shared by all tasks, and each task extracts features from the backbone and then processes them.

[0181] Pan-Density Crowd Counting (PADNet): A method for secondary multi-task augmentation using a multi-task model distillation strategy;

[0182] Multi-Scale Task Interaction Networks (MTINet): Based on PADNet, it introduces a two-level multi-task enhancement method with a multi-scale pyramid structure.

[0183] In some embodiments, this application has conducted multi-task performance tests on various multi-task processing methods on the publicly available datasets NYUDv2 and PASCAL Context.

[0184] Table 1 shows the impact of the partially shared architecture on the multitasking performance on the NYUDv2 dataset.

[0185] Table 1:

[0186]

[0187] Table 2 shows the impact of a partially shared structure on the multitasking performance on the PASCAL Context dataset.

[0188] Table 2:

[0189]

[0190] It should be noted that Tables 1 and 2 show the impact of the partially shared structure-based backbone module on the visual multitasking performance in the embodiments of this application. In the tables, "SemSeg" represents the image semantic segmentation task, "Depth" represents the image depth estimation task, "HumanParts" represents the human body part segmentation task, and "Sal" represents the saliency detection task. MTLΔ is a multitasking metric that measures the average percentage improvement of each task. Those ending with "-PS" are methods applying the partially shared structure of this invention. It can be seen that the partially shared structure-based backbone module in the embodiments of this application can significantly improve multitasking performance.

[0191] Referring to Table 3, the validity verification of the various network structures provided in this application is shown on the NYUDv2 dataset.

[0192] Table 3:

[0193]

[0194] Referring to Table 4, the validity of each structure provided in the embodiments of this application on the PASCAL Context dataset is verified.

[0195] Table 4:

[0196]

[0197] It should be noted that Tables 3 and 4 above illustrate the effectiveness verification of various structures provided in some embodiments of this application. This experiment demonstrates the change in multi-task performance when the basic BaseMultiTask model is gradually modified to the structure provided in the embodiments of this application. PS represents the backbone module based on a partially shared structure, S1 represents the basic feature interaction submodule, and S2 is the multi-task collaboration enhancement submodule. "-" indicates that this structure is not used, and "√" indicates that this structure is applied. The results show that the modular structure for multi-task processing designed in this application can significantly improve multi-task performance.

[0198] Table 5 shows a comparison of the performance of various multi-task processing methods on the NYUDv2 dataset.

[0199] Table 5:

[0200]

[0201] Table 6 compares the performance of various multi-task processing methods on the PASCAL Context dataset.

[0202] Table 6:

[0203]

[0204] It should be noted that Tables 5 and 6 above compare the performance, parameter count, and computational cost of the multi-task processing method provided in this application with existing multi-task methods on various backbone structures and two datasets. Here, "Params" represents the number of model parameters, and "FLOPS" represents the model computational cost. It can be seen that this application achieves optimal performance while maintaining advantages in parameter count and computational cost.

[0205] Next, the hardware structure of the electronic device executing the multitasking processing method provided in the embodiments of this application will be described. As an example, a mobile phone will be used as an example for explanation. The network architecture for multitasking processing provided in the embodiments of this application is as follows: Figure 1 The network architecture 30 shown can be deployed in the electronic device to support the electronic device in performing the multitasking processing method provided in the above embodiments.

[0206] like Figure 14 As shown, the mobile phone 160 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0207] It is understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the mobile phone 160. In other embodiments of this application, the mobile phone 160 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0208] The processor 110 may include one or more processing units, such as processing modules or circuits of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro-programmed control unit (MCU), an AI (Artificial Intelligence) processor, or a field-programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. The processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in the processor 110 is a cache memory 180. For example, the data of the multitasking network architecture 30 described above may be stored in the cache memory 180, and the processor 110 may invoke the network architecture 30 to execute the multitasking method provided in this application embodiment.

[0209] The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display 102, the camera 170, and the wireless communication module 120, etc.

[0210] The mobile communication module 130 may include, but is not limited to, an antenna, a power amplifier, a filter, and an LNA (Low Noise Amplifier). The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the mobile phone 160. The mobile communication module 130 can receive electromagnetic waves via the antenna, filter and amplify the received electromagnetic waves, and then transmit them to a modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device. Wireless communication technologies can include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), Wireless Local Area Networks (WLAN), Near Field Communication (NFC), Frequency Modulation (FM) and / or Field Communication (NFC), Infrared (IR) technology, etc.The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0211] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide wireless communication solutions for use on the mobile phone 160, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The mobile phone 160 can communicate with networks and other devices through wireless communication technologies.

[0212] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the mobile phone 160 may also be located in the same module.

[0213] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. As an example, the display screen 102 can display the results of multi-task processing, such as image segmentation results from an image segmentation task, or bounding box detection results from an object detection task.

[0214] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0215] The audio module 150 is used to convert digital audio information into analog audio signals for output, or to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110. In some embodiments, the audio module 150 may include a speaker, a handset, a microphone, and a headphone jack.

[0216] Camera 170 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Image Signal Processing) to be converted into a digital image signal. Mobile phone 160 can implement its shooting function through the ISP, camera 170, video codec, GPU (Graphics Processing Unit), display 102, and application processor. As an example, camera 170 can acquire images in real time and use these images as input images for the multi-tasking processing method executed in this application.

[0217] Interface module 160 includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of mobile phone 160. The external memory card communicates with processor 110 through the external memory interface to perform data storage. The USB interface is used for communication between mobile phone 160 and other electronic devices. The SIM card interface is used to communicate with the SIM card installed in mobile phone 160, for example, to read or write phone numbers stored in the SIM card.

[0218] In some embodiments, the mobile phone 160 further includes a button 101, a motor, and an indicator. The button 101 may include volume buttons, a power button, etc. The motor is used to generate a vibration effect in the mobile phone 160, for example, vibrating when the user's mobile phone 160 is called to prompt the user to answer the call. The indicator may include a laser indicator, a radio frequency indicator, an LED indicator, etc.

[0219] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0220] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0221] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0222] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0223] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0224] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0225] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0226] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A multi-task processing method for electronic devices, characterized in that, include: Obtain the input image; For the input image, at least two task-independent features corresponding to M tasks are obtained through a neural network, and at least two task-related features corresponding to each of the M tasks are obtained. The M tasks are all image processing tasks and any two tasks are different from each other. The at least two task-independent features are shared features in the processing of the M tasks, and each task-related feature is a feature used independently in the processing of the corresponding task. For each of the M tasks, at least one task-independent feature from the at least two task-independent features is stacked and fused with at least one corresponding task-related feature in the channel dimension to obtain the backbone feature corresponding to each task. For the M tasks, the output results of each task are obtained by utilizing the main features corresponding to each task.

2. The method according to claim 1, characterized in that, The neural network includes a backbone module, which includes N layers of partially shared convolutions. Each partially shared convolutional layer includes a task-independent convolution for obtaining task-independent features and a task-related convolution for obtaining task-related features.

3. The method according to claim 2, characterized in that, The step of obtaining at least two task-independent features corresponding to M tasks and at least two task-related features corresponding to each of the M tasks through a neural network includes: The input image is input into the first layer of the partial shared convolution of the backbone module of the neural network, and the first-level task-independent features and the first-level task-related features are output. The task-independent features of level i-1 are input into the task-independent convolution of the i-th partially shared convolution layer, and the i-th level task-independent features are output. Stack the (i-1)th level task-independent features and the (i-1)th level task-related features in the channel dimension and input them into the task-related convolution of the i-th layer partial shared convolution to output the i-th level task-related features, until the N-th layer partial shared convolution is reached, 2≤i≤N; The at least two task-independent features include task-independent features of levels 1 to N, and the at least two task-related features corresponding to each task include the features corresponding to each task among the task-related features of levels 1 to N.

4. The method according to claim 3, characterized in that, At least one task-independent feature corresponding to each task includes one or more levels of task-independent features from the first to Nth levels of task-independent features; At least one task-related feature corresponding to each task includes one or more levels of task-related features from the first to Nth levels.

5. The method according to any one of claims 2 to 4, characterized in that, The N layers of partially shared convolutions correspond to N preset stripping coefficients. Each stripping coefficient is used to indicate the ratio of the computational power of task-independent convolutions to the computational power of task-related convolutions in a partially shared convolution. The stripping coefficient increases with the model depth of the neural network.

6. The method according to claim 5, characterized in that, The initial convolutional layer in the N-layer partial shared convolution includes M+1 first convolutional layers. Each of the M+1 first convolutional layers includes a first task-related convolution and a first task-independent convolution corresponding to the M tasks in sequence. The input image is input into each of the M+1 first convolutional layers. Each first task-related convolution in the M+1 first convolutional layers outputs a corresponding first-level task-related feature. The first task-independent convolution in the M+1 first convolutional layers outputs a first-level task-independent feature.

7. The method according to claim 5, characterized in that, The i-th partially shared convolutional layer includes M+1 second convolutional layers. The M+1 second convolutional layers include second task-related convolutions corresponding to the M tasks in sequence and one second task-independent convolution. The j-th second task-related convolution takes into input the task-related features corresponding to the j-th task in the (i-1)-th level task-related features and the features in the (i-1)-th level task-independent features, and outputs the features in the task-related features corresponding to the j-th task in the i-th level task-related features, where 1≤j≤M. The second task-independent convolution in the i-th layer partial shared convolution takes the features from the (i-1)-th level task-independent features and outputs the features from the i-th level task-independent features.

8. The method according to claim 6, characterized in that, The backbone module also includes N target network layers, which are batch normalization layers, activation layers, or pooling layers. The i-th target network layer includes M+1 target network layers. The M+1 target network layers include the first target network layers corresponding to the M tasks in sequence and one second target network layer that is unrelated to the task. The j-th first target network layer takes into input the task-related features corresponding to the j-th task in the (i-1)-th level task-related features and outputs the task-related features corresponding to the j-th task in the i-th level task-related features, where 1≤j≤M. The second target network layer in the i-th target network layer takes into account the features in the (i-1)-th level task-independent features and outputs the features in the i-th level task-independent features.

9. The method according to claim 7, characterized in that, The N-layer partially shared convolutions of the backbone module also include fused convolutional layer groups; The fusion convolutional layer group includes M fusion convolutional layers. The j-th fusion convolutional layer in the M fusion convolutional layers is used to input the task-related features corresponding to the j-th task in the N-th task-related features and the features obtained by stacking the N-th task-independent features, 1≤j≤M; The j-th fusion convolutional layer among the M fusion convolutional layers is used to output the backbone features corresponding to the j-th task.

10. The method according to any one of claims 1, 2, 3, 4, 6, 7, 8, and 9, characterized in that, For the M tasks, obtaining the output results corresponding to each task using the backbone features of each task includes: The primary prediction result of the j-th task is obtained by using the backbone features of the j-th task, where 1≤j≤M; Using the primary prediction result of the j-th task, obtain the secondary prediction result of the j-th task; The output result of the j-th task is obtained by adding the primary prediction result of the j-th task and the secondary prediction result of the j-th task.

11. The method according to claim 10, characterized in that, The process of obtaining the primary prediction result for the j-th task using the backbone features of the j-th task includes: For the M tasks, the backbone features corresponding to the M-1 tasks other than the j-th task are fused and stacked into the backbone features of the current task. The primary prediction result of the j-th task is then obtained by using the stacked backbone features.

12. The method according to claim 11, characterized in that, The step of obtaining the secondary prediction result of the j-th task using the primary prediction result of the j-th task includes: For the M tasks, the primary prediction results of the M-1 tasks other than the j-th task are converted into masks respectively; Multiply the masks corresponding to the M-1 tasks other than the j-th task by the primary prediction result of the j-th task to obtain the corresponding masked prediction result. The prediction results of the M-1 tasks other than the j-th task are weighted and summed according to their respective predefined weights, and then added to the primary prediction result of the j-th task to obtain the fusion prediction result corresponding to the j-th task. The secondary prediction result of the j-th task is predicted using the fusion prediction result corresponding to the j-th task.

13. The method according to claim 11, characterized in that, The neural network includes a first fusion layer, a task attention module, and a primary prediction module. The first fusion layer is a convolutional layer. The task attention module includes a pooling layer and a convolutional layer. The primary prediction module includes a convolutional layer, a batch normalization layer, an activation layer, and a convolutional layer. The initial prediction result for the j-th task is generated in the following manner: The backbone features corresponding to the M-1 tasks other than the j-th task are stacked and input into the first fusion layer to obtain the first fusion feature; Input the backbone features of the j-th task into the task attention module to obtain channel-dimensional attention with a spatial dimension of 1×1; The channel-dimensional attention is multiplied by the first fusion feature and stacked with the backbone feature of the j-th task to obtain the second fusion feature; The second fusion feature is input into the primary prediction module, which outputs the primary prediction result for the j-th task.

14. The method according to claim 12, characterized in that, The neural network includes a two-level prediction module, which includes convolutional layers, batch normalization layers, activation layers, and convolutional layers. The secondary prediction result of the j-th task is obtained by inputting the fusion prediction result corresponding to the j-th task into the secondary prediction module.

15. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the multitasking method of any one of claims 1 to 14.

16. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of an electronic device, and a processor, one of the processors of the electronic device, for executing the multitasking method of any one of claims 1 to 14.

Images