Multitask intensive prediction method based on bidirectional interaction and related device

Abstract

The invention belongs to the technical field of computer vision, and discloses a multitask intensive prediction method based on bidirectional interaction and a related device. The multi-task intensive prediction method comprises the steps of performing multi-task prediction by utilizing a trained multi-task intensive prediction model based on an obtained RGB image, and obtaining a prediction result of each task; the multi-task intensive prediction model comprises an encoder used for inputting RGB images and outputting task general features; the decoder is used for inputting the general characteristics of the tasks and outputting the finally enhanced specific characteristics of the tasks; and task heads corresponding to the tasks in the multi-task intensive prediction are respectively used for inputting the finally enhanced specific features of the tasks and performing prediction, and correspondingly outputting prediction results of the tasks. According to the method, the contradiction between cross-task interaction sufficiency and interaction efficiency in multi-task intensive prediction is solved, and a finer and more accurate multi-task intensive prediction result can be obtained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a multi-task dense prediction method based on bidirectional interaction and a related device. BACKGROUND

[0002] Multi-task dense prediction is a key computer vision task aiming to predict the outputs of multiple pixel-level tasks simultaneously, such as semantic segmentation, depth prediction, etc. It is the core technology of many other fields, such as autonomous driving, embodied intelligence, medical image analysis, etc. The purpose is to utilize the relationship between vision tasks to improve overall performance.

[0003] Sufficient cross-task interaction is crucial for multi-task dense prediction, but sufficient interaction usually leads to high computational complexity. The above situation forces existing solutions to make a trade-off between cross-task interaction sufficiency and interaction efficiency. Specifically, some current methods that pursue interaction efficiency usually extract a shared interaction representation from all task features to interact with each task. This can achieve linear growth of interaction complexity with the number of tasks, but only part of the features of each task can be used for interaction, limiting the sufficiency of interaction. In addition, some current methods that pursue interaction sufficiency usually adopt a pairwise interaction strategy, which brings quadratic interaction complexity. In summary, there is an urgent need for a multi-task dense prediction solution that can balance cross-task interaction sufficiency and interaction complexity. SUMMARY

[0004] The present application aims to provide a multi-task dense prediction method based on bidirectional interaction and a related device to solve the technical problem that the existing technical solutions cannot balance cross-task interaction sufficiency and complexity. The technical solution disclosed in the present application solves the contradiction between cross-task interaction sufficiency and interaction efficiency in multi-task dense prediction, and can obtain more accurate multi-task dense prediction results.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect of the present application, a multi-task dense prediction method based on bidirectional interaction is provided, comprising the following steps:

[0007] Obtaining an RGB image to be subjected to multi-task dense prediction;

[0008] Based on the obtained RGB image, a trained multi-task dense prediction model is used for multi-task prediction to obtain task prediction results;

[0009] The multi-task dense prediction model comprises:

[0010] The encoder is used to input RGB images, gradually extract image features through a series of downsampling operations, and output task-general features;

[0011] The decoder is used to input the common features of the task, gradually restore the size of the feature map through a series of upsampling operations, and output the final enhanced specific features of each task; wherein, at each stage of the decoder, feature refinement is performed by a feature refinement module, and the feature refinement module includes T multi-scale single-task modules and a bidirectional cross-task interaction module; each multi-scale single-task module is used to input the fusion features for a specific task and use the multi-scale scanning mechanism to process them, and output the task-specific features of the specific task; the bidirectional cross-task interaction module is used to input the task-specific features output by all multi-scale single-task modules, and use the bidirectional interactive scanning mechanism to perform cross-task The multi-scale single-task module processes the task information interactively and outputs upsampled features. In the first stage of the decoder, the fused features input by the multi-scale single-task module are the upsampled features of the task-general features of the last stage of the encoder and the fused features of the task-general feature channels of the penultimate stage. In the subsequent stages of the decoder, the fused features input by the multi-scale single-task module are the upsampled features output by the bidirectional cross-task interaction module in the previous stage of the decoder and the fused features of the task-general feature channels of the corresponding stage of the encoder. In the last stage of the decoder, the upsampled features output by the bidirectional cross-task interaction module are the final enhanced task-specific features.

[0012] The task heads corresponding to each task in multi-task intensive prediction are used to input the final enhanced task-specific features and make predictions, and output the corresponding prediction results of each task.

[0013] A further improvement of the technical solution of the present invention is that the multi-scale single-task module includes: a normalization layer, a scanning branch and a gating branch; wherein,

[0014] The normalization layer is used to input the fusion features and perform normalization processing, and output the normalized fusion features; the normalized fusion features are sequentially processed by the linear layer, the convolution layer, the activation layer and the multi-scale scanning mechanism to obtain multi-scale task information; in the gating branch, the normalized fusion features are sequentially processed by the linear layer and the activation layer to obtain the gating information;

[0015] Finally, the gate information is multiplied with the multi-scale task information and then connected with the input fusion feature residual to output task-specific features.

[0016] A further improvement of the technical solution of the present invention is that the multi-scale scanning mechanism is to input the features processed by the activation layer, split them into multiple parallel feature subspaces along the channel dimension, perform scanning modeling of different preset scales in each feature subspace branch, and finally splice and fuse the features of all feature subspace branches along the channel dimension to obtain the final multi-scale task information.

[0017] A further improvement of the technical solution of the present invention is that the step of performing scanning modeling at different preset scales in each feature subspace branch includes:

[0018] Window feature tokenization, including: setting the input feature x of the i-th feature subspace branch i The size is m×H×W, and the scanning scale is s o , m is the channel dimension, H and W are the height and width respectively; the feature x i Divide into spatially adjacent but non-overlapping i ×s i The feature size of each window is m×s i ×s i ; Stack the features in each window along the channel dimension to form a size of (m×s i ×s i )×1×1 feature tokens; all feature tokens formed by each window are rearranged according to the original spatial position to form a feature token of size Tokenized feature graph X i ;

[0019] Feature serialization and modeling, including: tokenizing feature graph X i Multiple feature sequences are constructed along the preset scanning pattern and input into the state space model for sequence modeling to obtain the features of the i-th feature subspace branch.

[0020] A further improvement of the technical solution of the present invention is that, in the bidirectional cross-task interaction module, the steps of inputting the task-specific features output by all multi-scale single-task modules, performing cross-task information interaction processing using a bidirectional interactive scanning mechanism, and outputting the final enhanced task-specific features include:

[0021] First, all the input task-specific features are spliced ​​along the channel to obtain the spliced ​​task-specific features; then, the spliced ​​task-specific features and the input task-specific features are normalized respectively to obtain the normalized spliced ​​task-specific features and the normalized task-specific features; finally, the normalized spliced ​​task-specific features are processed using a bidirectional interactive scanning mechanism to obtain the scan-enhanced task-specific features; the scan-enhanced task-specific features are fused with the normalized task-specific features respectively, and then connected with the residual of each task-specific feature of the input respectively to obtain the final enhanced task-specific features.

[0022] A further improvement of the technical solution of the present invention is that the normalized spliced ​​task-specific features are processed using a bidirectional interactive scanning mechanism to obtain scan-enhanced task-specific features; the scan-enhanced task-specific features are respectively fused with the normalized task-specific features, and then respectively connected with the residuals of the input task-specific features to obtain the final enhanced task-specific features, which specifically include the following steps:

[0023] Feature preparation: prepare the features to be interacted It is divided into two equal parts in the channel dimension and sent to the forward scanning module and the backward scanning module respectively. For the backward scanning module, its input features are first processed in reverse order in the task dimension.

[0024] Bidirectional scanning and modeling: Both the forward scanning module and the backward scanning module use hierarchical cross-task interactive scanning, including two modes:

[0025] Task-priority scanning mode: for each task t’s features Spatial serialization is performed according to a preset scanning pattern to form T subsequences of length H × W. The subsequences are then concatenated along the length dimension according to the task order to form a global task sequence of length T × (H × W). The global sequence is then modeled using a state-space model to capture the overall dependencies between tasks.

[0026] Position-first scanning mode: The output features of the task-first scanning mode are restored to the original spatial dimensions of each task. For each spatial position (H, W) on the feature map, its corresponding feature tokens across T tasks are concatenated to form H×W subsequences of length T. These subsequences are then concatenated along the length dimension according to the preset scanning mode to form a global position sequence of length (H×W)×T. The global position sequence is also modeled using the state-space model to capture local correlations across tasks in spatial positions.

[0027] Interaction result integration: the output of the forward scanning module The output of the backward scanning module after the task dimension is reversed The splicing is performed to obtain the shared feature F after bidirectional interaction sh ;

[0028] Feature adaptive fusion: each task branch t will obtain the part F corresponding to the task in the obtained shared feature sh [t] is selected by a learnable gate and the original input feature is adaptively fused to obtain the final enhanced task-specific feature.

[0029] Further improvement of the technical scheme of the present application is that in the training process of the multi-task dense prediction model, an end-to-end joint training is adopted, and the overall loss function L is represented as:

[0030]

[0031] In the formula, is the prediction result of task t, Yt is the true label of task t; Lt is the loss function selected by task t, λ t is the loss weight of task t, used to balance the contribution of different tasks to the overall optimization.

[0032] In the second aspect of the present application, a multi-task dense prediction system based on bidirectional interaction is provided, comprising:

[0033] A data acquisition module is configured to acquire an RGB image to be subjected to multi-task dense prediction.

[0034] A task prediction module is configured to perform multi-task prediction based on the acquired RGB image by using the trained multi-task dense prediction model to obtain task prediction results.

[0035] The multi-task dense prediction model comprises:

[0036] An encoder is configured to input the RGB image and gradually extract image features by a series of downsampling operations to output task general features.

[0037] The decoder is used to input the common features of the task, gradually restore the size of the feature map through a series of upsampling operations, and output the final enhanced specific features of each task; wherein, at each stage of the decoder, feature refinement is performed by a feature refinement module, and the feature refinement module includes T multi-scale single-task modules and a bidirectional cross-task interaction module; each multi-scale single-task module is used to input the fusion features for a specific task and use the multi-scale scanning mechanism to process them, and output the task-specific features of the specific task; the bidirectional cross-task interaction module is used to input the task-specific features output by all multi-scale single-task modules, and use the bidirectional interactive scanning mechanism to perform cross-task The multi-scale single-task module processes the task information interactively and outputs upsampled features. In the first stage of the decoder, the fused features input by the multi-scale single-task module are the upsampled features of the task-general features of the last stage of the encoder and the fused features of the task-general feature channels of the penultimate stage. In the subsequent stages of the decoder, the fused features input by the multi-scale single-task module are the upsampled features output by the bidirectional cross-task interaction module in the previous stage of the decoder and the fused features of the task-general feature channels of the corresponding stage of the encoder. In the last stage of the decoder, the upsampled features output by the bidirectional cross-task interaction module are the final enhanced task-specific features.

[0038] The task heads corresponding to each task in multi-task intensive prediction are used to input the final enhanced task-specific features and make predictions, and output the corresponding prediction results of each task.

[0039] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-task intensive prediction method based on bidirectional interaction as described in any one of the first aspects of the present invention is implemented.

[0040] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-task intensive prediction method based on bidirectional interaction as described in any one of the first aspects of the present invention.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] In order to solve the contradiction between the sufficiency and efficiency of cross-task interaction in the existing multi-task dense prediction, the application discloses a multi-task dense prediction method based on bidirectional interaction, which can obtain more accurate multi-task dense prediction results based on the trained multi-task dense prediction model. Specifically, in the multi-task dense prediction model of the application, the general encoder is used to extract the general features, the decoder is used to obtain the enhanced specific features of each task, and then the task head is used to predict the results. Based on the state space model, the application designs two simple and effective scanning mechanisms to assist the state space model to solve the multi-task dense prediction problem, which can realize efficient and sufficient cross-task relationship modeling in the cross-task interaction, and then obtain more accurate prediction results, solving the contradiction between the sufficiency and efficiency of cross-task interaction in the multi-task dense prediction.

[0043] In the technical scheme of the application, the process of obtaining enhanced specific features of each task based on the decoder integrates a bidirectional interaction scanning mechanism and a multi-scale scanning mechanism; the bidirectional interaction scanning mechanism constructs a task-specific bidirectional sequence, and fuses a task-priority and a position-priority scanning mode to efficiently retain key cross-task information with linear complexity; the multi-scale scanning mechanism realizes multi-granularity scene modeling, meets the needs of different task granularities, and enhances detailed cross-task feature interaction. Specifically, the application provides a simple and effective task feature enhancement method for multi-task dense prediction, which includes two novel scanning mechanisms for enhancing the state space model, thereby solving the conflict problem between the task interaction sufficiency and complexity. The first is a bidirectional cross-task scanning mechanism, which constructs each task representation into a sequence and uses a bidirectional scanning strategy for cross-task interaction modeling, effectively reducing information loss while maintaining linear interaction complexity; in addition, the mechanism fuses the task-priority and position-priority scanning modes to realize multi-level cross-task interaction, thereby helping to model the task relationship more finely. The second is a multi-scale scanning mechanism, the input features are divided into multiple spaces, so that the scene structure information of the image is modeled at different scales; then, the specific task integrates the multi-scale scene structure information to enhance the specific task representation and meet the different granularity needs of each task for scene structure information. In summary, the preferred technical scheme of the application maintains linear interaction complexity while fully modeling the cross-task relationship, significantly improves the interaction efficiency and sufficiency, and can obtain more accurate multi-task dense prediction results with less computational complexity. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 1 is a flowchart of a multi-task intensive prediction method based on two-way interaction in an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of the architecture of a multi-task intensive prediction model in an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the architecture of the multi-scale single-task module and the bidirectional cross-task interaction module in an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a multi-scale scanning mechanism in an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of a bidirectional interactive scanning mechanism in an embodiment of the present invention;

[0050] Figure 6 Schematic diagram of the effects of the present invention compared with the existing most advanced methods in various complex scenarios in an embodiment of the present invention;

[0051] Figure 7 4 is a schematic diagram of a multi-task intensive prediction system based on two-way interaction in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0053] All other embodiments obtained by persons of ordinary skill in the art based on the technical solutions disclosed in the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0054] See also Figure 1 and Figure 2 The embodiment of the present invention provides a multi-task intensive prediction method based on two-way interaction, comprising the following steps:

[0055] Step 1: Obtain the RGB image to be used for multi-task dense prediction;

[0056] Step 2: Based on the RGB image obtained in step 1, the trained multi-task intensive prediction model is used to perform task prediction and obtain the prediction results of each task;

[0057] The multi-task intensive prediction model includes:

[0058] The encoder is used to input RGB images, gradually extract image features through a series of downsampling operations, and finally output task-general features; for example, Figure 2 As shown in Figure 2, the specific steps may include: (1) Image partitioning: First, the input RGB image (size is 3×H×W, 3 represents the three color channels of the image (red, green, blue), H and W represent the height and width of the image respectively) is divided into multiple small blocks (Patch Partition); (2) Feature extraction: Use multiple Swin modules to extract features; after each Swin module, the size of the feature map is halved and the number of channels is doubled. For example, the size of the initial feature map is After the first Swin module, the feature map size becomes After passing through the next Swin module, it becomes And so on, until the last Swin module outputs the common characteristics of the task C is the number of channels;

[0059] The decoder is used to input task-general features, gradually restore the feature map size through a series of upsampling operations, and output enhanced task-specific features; wherein, at each stage of the decoder, feature refinement is performed by a feature refinement module, and the feature refinement module includes multiple multi-scale single-task modules and a two-way cross-task interaction module; each multi-scale single-task module is used for a specific task, inputs the fusion features of the task-general features in the first stage, and inputs the enhanced task-specific features of each task in the previous stage and the fusion features of the task-general features in the subsequent stage, and outputs the task-specific features of the specific task; the two-way cross-task interaction module is used to input the task-specific features output by all multi-scale single-task modules, perform cross-task information interaction, and output enhanced task-specific features; illustratively, as Figure 2 As shown in Figure 2, in both the first stage (G1) and subsequent stages (G2, G3, and G4), the role of this bidirectional cross-task interaction module is to promote feature fusion and enhancement between different tasks.

[0060] The task heads corresponding to each task in multi-task intensive prediction are used to input the enhanced task-specific features of each task and make predictions, and output the prediction results of each task accordingly. Explanatorily, the type and number of task heads can be determined according to the type and number of tasks in multi-task intensive prediction, and the number of multi-scale single-task modules can be determined according to the number of tasks in multi-task intensive prediction.

[0061] See also Figure 3 It should be emphasized that the multi-scale single-task module includes: a normalization layer, a scanning branch and a gating branch; wherein the normalization layer is used to input the fusion features and perform normalization processing, and output the normalized fusion features; in the scanning branch, the normalized fusion features are processed in sequence by the linear layer, the convolution layer, the activation layer and the multi-scale scanning mechanism to obtain the multi-scale task information; in the gating branch, the normalized fusion features are processed in sequence by the linear layer and the activation layer to obtain the gating information; finally, the gating information is multiplied by the multi-scale task information, and then connected with the input fusion feature residual to output the task-specific features.

[0062] See also Figure 3 It should be emphasized that in the bidirectional cross-task interaction module, first, all the input task-specific features are spliced ​​along the channel to obtain the spliced ​​task-specific features; then, the spliced ​​task-specific features and the input task-specific features are normalized respectively to obtain the normalized task-specific features and the normalized spliced ​​task-specific features; the normalized spliced ​​task-specific features are processed using the bidirectional interactive scanning mechanism to obtain the scanned enhanced task-specific features; the scanned enhanced task-specific features are fused with the normalized task-specific features respectively, and then connected with the residual of the input task-specific features respectively to obtain the final enhanced task-specific features.

[0063] See also Figure 4 and Figure 5 The embodiment of the present invention provides a multi-task intensive prediction method based on two-way interaction. Figure 6 Taking the four dense prediction tasks (semantic segmentation, depth prediction, surface normal vector detection and edge detection) in as an example, the steps are as follows:

[0064] Step 1: Obtain common task features, including:

[0065] To handle multi-task learning scenarios involving T (T=4) dense prediction tasks, the task-general features G are generally composed of the image or video frame to be processed. Obtained by a pre-trained or non-pre-trained encoder φ; wherein the encoder includes but is not limited to the network structure of CNN and Transformer architecture;

[0066] The common features of the task are expressed as:

[0067] G={G 1 ,G 2 ,G 3 ,G 4}=φ(I);

[0068] Among them, G i represents the general features obtained by the encoder in the i-th stage, where i is 1, 2, 3, or 4.

[0069] Step 2: Obtain task-specific features from task-general features, including:

[0070] The task-specific module based on the state-space model Mamba, namely the MSST module in the feature refinement module (MFR block), refines the general features into task-specific features; in each MFR block, the features input to the MSST module (for example, the output of the previous MFR block or the fusion of the encoder features after upsampling and the encoder's higher resolution features) are passed through the multi-scale single-task module (MSST) for multi-granularity scene modeling to enhance the representation of information within the task. In the MSST module, the core operation of the scanning branch is the multi-scale scanning mechanism (MS-Scan), as shown in the attached figure. Figure 4 As shown, it includes: the features processed by the input activation layer are first divided into N parallel feature subspaces along the channel dimension, and different preset scales S are performed in each feature subspace branch. i Finally, the features of all branches are spliced ​​and fused along the channel dimension to obtain the final multi-scale processing result.

[0071] like Figure 4 As shown, specifically and exemplarily, in order to achieve multi-scale without significantly increasing the amount of calculation, an embodiment of the present invention proposes an effective multi-scale processing strategy, specifically:

[0072] 1) Window feature tokenization: Assume that the input feature x of the i-th scanning branch is i The size is m×H×W, and the scanning scale is S i . First, the feature map x i Divide into spatially adjacent but non-overlapping blocks of size S i ×s i The feature size of each window is m×s i ×s i Subsequently, the features within each window are stacked along the channel dimension (i.e., the spatial dimension is flattened and connected to the channel dimension) to form a feature vector of size (m×s i ×s i )×1×1 feature tokens. All feature tokens formed by each window are rearranged according to their original spatial positions to form a feature token of size Tokenized feature graph Xi .

[0073] 2) Feature serialization and Mamba modeling: Tokenized feature maps X i are constructed into multiple feature sequences along different scanning paths. These sequences are then input into the state space model Mamba for sequence modeling, obtaining enhanced task-specific features y i .

[0074] 3) Multi-scale fusion: The outputs y i from N branches are fused into the final MSST module output features through the inverse process of window tokenization and channel concatenation operations.

[0075] Step 3, cross-task interaction is performed using a bidirectional interaction module, which specifically includes:

[0076] To fully utilize the collaborative relationship between tasks to further enhance the feature representation of each task, the present application adopts a bidirectional cross-task interaction module (BCFR) based on a state space model. The core of this module is a bidirectional interaction scanning mechanism (BI-Scan), as shown in Figure 5 , and the specific steps are as follows:

[0077] 1) Feature preparation: The interaction features to be interacted from T different task branches, which have been preliminarily refined by the MSST module in step 2 (or directly from the encoder, depending on the level at which the MFR module is located) are divided into two equal parts in the channel dimension and sent into the forward scanning module and the backward scanning module, respectively. For the backward scanning module, the input features are first processed in reverse order in the task dimension.

[0078] 2) Bidirectional scanning and modeling: Within the forward and backward scanning modules, hierarchical cross-task interaction scanning is adopted, including two modes:

[0079] a) Task-priority scanning mode: First, the features of each task t are spatially serialized according to the preset scanning mode (notably, the preset scanning mode, as shown in the mode section of the attached Figure 5 , contains four types, namely horizontal "Z" type order and reverse order, and vertical "Z" type order and reverse order). Then, these sub-sequences are concatenated along the length dimension according to the task order, forming a global task sequence with a length of T x (H x W). This global sequence is then modeled by the state space model to capture the overall dependency between tasks.

[0080] b) Position-first scanning mode: The output features of the task-first mode are first restored to the original spatial dimensions of each task. Then, for each spatial position (H, W) on the feature map, its corresponding feature tokens across T tasks are concatenated to form (H×W) subsequences of length T. These subsequences are then concatenated along the length dimension according to the preset scanning mode to form a global position sequence of length (H×W)×T. This global sequence is also modeled using a state-space model to capture local cross-task correlations in spatial position.

[0081] 3) Integration of interactive results: The output of the forward scanning module The output of the backward scanning module after the task dimension is reversed Splice and get the shared feature F after bidirectional interaction sh .

[0082] 4) Feature Adaptive Fusion: Each task branch t will obtain the part F of the shared features corresponding to the task sh [t], selected by a learnable gate With the original input features Perform adaptive fusion to obtain the final enhanced task-specific features

[0083] Step 4: The task-specific task header obtains the prediction results, which includes:

[0084] The final task-specific feature F is gradually refined and enhanced by the multi-layer MFR module t Feed it into the task-specific prediction head H t To generate the final prediction results It mainly includes scale transformation (Ψ) and channel mapping (P t ).

[0085] Step 5: Model training, including:

[0086] The entire multi-task intensive prediction model is trained jointly in an end-to-end manner, including the encoder in step 1, the decoder composed of steps 2 and 3, and the task head H in step 4. t .

[0087] 1) Loss function construction: Define an overall loss function L, which is the individual loss L of all T tasks t The weighted sum of: in, is the prediction result of task t, Yt is the true label of task t, Lt is the loss function selected by task t, and λ t is the loss weight of task t, which is used to balance the contribution of different tasks to the overall optimization and can be preset or dynamically adjusted.

[0088] 2) Implementation Details: The model uses the Swin-Large Transformer pre-trained on ImageNet-22K as the encoder. During training, the Adam optimizer was used for all datasets, and the initial learning rate was set to 5×10 -5 , the weight decay rate is 1×10 -5 , combined with a polynomial learning rate scheduling strategy. For the NYUD-v2 dataset, the model was trained for 50,000 iterations with a batch size of 4, and the input images were resized to 448×576 pixels. For the PASCAL-Context dataset, the model was trained for 75,000 iterations with a batch size of 6, and the input images were resized to 512×512 pixels. Data augmentation techniques including random color jittering, random cropping, random scaling, and random horizontal flipping were applied during training. Model parameters were optimized by minimizing the overall loss function consisting of the weighted sum of the losses for each task.

[0089] Figure 6 The visualization results of the method of the embodiment of the present invention and the best existing method (MTMamba) in two challenging multi-task prediction scenarios are shown. Figure 6 The upper part shows the comparison in the scenario of 4 tasks (semantic segmentation, depth estimation, surface normal estimation and edge detection). It can be clearly observed that the method proposed in this invention can more accurately segment the edge of the pillow in the semantic segmentation task (as shown in the orange circle), showing a more powerful scene understanding and multi-task processing capability. Figure 6 The lower half shows a comparison of five tasks (semantic segmentation, portrait segmentation, saliency detection, surface normal estimation, and edge detection). In the semantic segmentation task, the method of the present invention can detect objects that previous methods have not noticed (as shown in the orange circle); in the portrait segmentation task, the method of the present invention can more finely distinguish the upper and lower arms of the portrait. In summary, the prediction map generated by the method proposed in this invention is more accurate in details, with clearer edges and more consistent semantic understanding, significantly outperforming the comparison method, intuitively demonstrating the superiority of the present invention in multi-task intensive prediction quality.

[0090] In view of the fact that the existing technology is difficult to effectively balance the sufficiency of cross-task information interaction and computational efficiency when dealing with multi-task intensive prediction problems, the model has limited prediction accuracy in complex scenarios. The present invention proposes a novel and efficient cross-task interaction paradigm based on the state-space model, the core of which lies in the innovative bidirectional interactive scanning (BI-Scan) mechanism and multi-scale scanning (MS-Scan) mechanism. The BI-Scan mechanism constructs a task-specific bidirectional sequence representation and cleverly integrates the task priority and position priority scanning modes. It can efficiently and fully capture and utilize key cross-task collaborative information under linear computational complexity, effectively avoiding information bottlenecks and loss of details. The MS-Scan mechanism accurately meets the diverse scale requirements of scene understanding for different intensive prediction tasks through multi-granularity scene modeling, and enhances the expressive power of each task feature. This framework enables multi-task intensive prediction technology to significantly improve the prediction accuracy and robustness of each task when processing complex visual scenes, providing more reliable scene understanding capabilities for related application fields. The multi-task intensive prediction method provided by the embodiment of the present invention demonstrates powerful comprehensive prediction performance when faced with complex scenarios that require simultaneous processing of multiple visual perception tasks (semantic segmentation, depth estimation, surface normal estimation, and edge detection).

[0091] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0092] See also Figure 7 In an embodiment of the present invention, a multi-task intensive prediction system based on two-way interaction is provided, comprising:

[0093] Data acquisition module, used to obtain RGB images to be used for multi-task intensive prediction;

[0094] The task prediction module is used to perform multi-task prediction based on the acquired RGB image using the trained multi-task intensive prediction model to obtain the prediction results of each task;

[0095] The multi-task intensive prediction model includes:

[0096] The encoder is used to input RGB images, gradually extract image features through a series of downsampling operations, and output task-general features;

[0097] The decoder is used to input the common features of the task, gradually restore the size of the feature map through a series of upsampling operations, and output the final enhanced specific features of each task; wherein, at each stage of the decoder, feature refinement is performed by a feature refinement module, and the feature refinement module includes T multi-scale single-task modules and a bidirectional cross-task interaction module; each multi-scale single-task module is used to input the fusion features for a specific task and use the multi-scale scanning mechanism to process them, and output the task-specific features of the specific task; the bidirectional cross-task interaction module is used to input the task-specific features output by all multi-scale single-task modules, and use the bidirectional interactive scanning mechanism to perform cross-task The multi-scale single-task module processes the task information interactively and outputs upsampled features. In the first stage of the decoder, the fused features input by the multi-scale single-task module are the upsampled features of the task-general features of the last stage of the encoder and the fused features of the task-general feature channels of the penultimate stage. In the subsequent stages of the decoder, the fused features input by the multi-scale single-task module are the upsampled features output by the bidirectional cross-task interaction module in the previous stage of the decoder and the fused features of the task-general feature channels of the corresponding stage of the encoder. In the last stage of the decoder, the upsampled features output by the bidirectional cross-task interaction module are the final enhanced task-specific features.

[0098] The task heads corresponding to each task in multi-task intensive prediction are used to input the final enhanced task-specific features and make predictions, and output the corresponding prediction results of each task.

[0099] In one embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the multi-task intensive prediction method based on bidirectional interaction.

[0100] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-task intensive prediction method based on bidirectional interaction in the above embodiment.

[0101] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) that contain computer-usable program code.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multi-task intensive prediction method based on two-way interaction, characterized in that: The following steps are involved: Get the RGB image to be used for multi-task dense prediction; Based on the acquired RGB image, multi-task prediction is performed using the trained multi-task intensive prediction model to obtain the prediction results of each task; The multi-task intensive prediction model includes: The encoder is used to input RGB images, gradually extract image features through a series of downsampling operations, and output task-general features; The decoder is used to input the common features of the task, gradually restore the size of the feature map through a series of upsampling operations, and output the final enhanced specific features of each task; wherein, at each stage of the decoder, feature refinement is performed by a feature refinement module, and the feature refinement module includes T multi-scale single-task modules and a bidirectional cross-task interaction module; each multi-scale single-task module is used to input the fusion features for a specific task and use the multi-scale scanning mechanism to process them, and output the task-specific features of the specific task; the bidirectional cross-task interaction module is used to input the task-specific features output by all multi-scale single-task modules, and use the bidirectional interactive scanning mechanism to perform cross-task The multi-scale single-task module processes the task information interactively and outputs upsampled features. In the first stage of the decoder, the fused features input by the multi-scale single-task module are the upsampled features of the task-general features of the last stage of the encoder and the fused features of the task-general feature channels of the penultimate stage. In the subsequent stages of the decoder, the fused features input by the multi-scale single-task module are the upsampled features output by the bidirectional cross-task interaction module in the previous stage of the decoder and the fused features of the task-general feature channels of the corresponding stage of the encoder. In the last stage of the decoder, the upsampled features output by the bidirectional cross-task interaction module are the final enhanced task-specific features. The task heads corresponding to each task in multi-task intensive prediction are used to input the final enhanced task-specific features and make predictions, and output the corresponding prediction results of each task.

2. The multi-task intensive prediction method based on two-way interaction according to claim 1, characterized in that: The multi-scale single-task module includes: a normalization layer, a scanning branch and a gating branch; wherein, The normalization layer is used to input the fusion features and perform normalization processing, and output the normalized fusion features; the normalized fusion features are sequentially processed by the linear layer, the convolution layer, the activation layer and the multi-scale scanning mechanism to obtain multi-scale task information; in the gating branch, the normalized fusion features are sequentially processed by the linear layer and the activation layer to obtain the gating information; Finally, the gate information is multiplied with the multi-scale task information and then connected with the input fusion feature residual to output task-specific features.

3. The multi-task intensive prediction method based on two-way interaction according to claim 2, characterized in that: The multi-scale scanning mechanism is as follows: the features processed by the input activation layer are divided into multiple parallel feature subspaces along the channel dimension, scanning modeling of different preset scales is performed in each feature subspace branch, and finally the features of all feature subspace branches are spliced ​​and fused along the channel dimension to obtain the final multi-scale task information.

4. The multi-task intensive prediction method based on two-way interaction according to claim 3, characterized in that: The step of performing scanning modeling at different preset scales in each feature subspace branch includes: Window feature tokenization, including: setting the input feature x of the i-th feature subspace branch i The size is m×H×W, and the scanning scale is s i , m is the channel dimension, H and W are the height and width respectively; the feature x i Divide into spatially adjacent but non-overlapping i ×s i The feature size of each window is m×s i ×s i ; Stack the features in each window along the channel dimension to form a size of (m×s i ×s i )×1×1 feature tokens; all feature tokens formed by each window are rearranged according to the original spatial position to form a feature token of size Tokenized feature graph X i ; Feature serialization and modeling, including: tokenizing feature graph X i Multiple feature sequences are constructed along the preset scanning pattern and input into the state space model for sequence modeling to obtain the features of the i-th feature subspace branch.

5. The multi-task intensive prediction method based on two-way interaction according to claim 1, characterized in that: In the bidirectional cross-task interaction module, the steps of inputting the task-specific features output by all multi-scale single-task modules, performing cross-task information interaction processing using a bidirectional interactive scanning mechanism, and outputting the final enhanced task-specific features include: First, all the input task-specific features are spliced ​​along the channel to obtain the spliced ​​task-specific features; then, the spliced ​​task-specific features and the input task-specific features are normalized respectively to obtain the normalized spliced ​​task-specific features and the normalized task-specific features; finally, the normalized spliced ​​task-specific features are processed using a bidirectional interactive scanning mechanism to obtain the scan-enhanced task-specific features; the scan-enhanced task-specific features are fused with the normalized task-specific features respectively, and then connected with the residual of each task-specific feature of the input respectively to obtain the final enhanced task-specific features.

6. The multi-task intensive prediction method based on two-way interaction according to claim 5, characterized in that: The normalized spliced ​​task-specific features are processed using a bidirectional interactive scanning mechanism to obtain scan-enhanced task-specific features; The steps of fusing the enhanced task-specific features with the normalized task-specific features, and then connecting them with the residuals of the input task-specific features to obtain the final enhanced task-specific features include: Feature preparation: prepare the features to be interacted It is divided into two equal parts in the channel dimension and sent to the forward scanning module and the backward scanning module respectively. For the backward scanning module, its input features are first processed in reverse order in the task dimension. Bidirectional scanning and modeling: Both the forward scanning module and the backward scanning module use hierarchical cross-task interactive scanning, including two modes: Task-priority scanning mode: for each task t’s features Spatial serialization is performed according to a preset scanning pattern to form T subsequences of length H × W. The subsequences are then concatenated along the length dimension according to the task order to form a global task sequence of length T × (H × W). The global sequence is then modeled using a state-space model to capture the overall dependencies between tasks. Position-first scanning mode: The output features of the task-first scanning mode are restored to the original spatial dimensions of each task. For each spatial position (H, W) on the feature map, its corresponding feature tokens across T tasks are concatenated to form H×W subsequences of length T. These subsequences are then concatenated along the length dimension according to the preset scanning mode to form a global position sequence of length (H×W)×T. The global position sequence is also modeled using the state-space model to capture local correlations across tasks in spatial positions. Interaction result integration: the output of the forward scanning module The output of the backward scanning module after the task dimension is reversed Splice and get the shared feature F after bidirectional interaction sh ; Feature adaptive fusion: Each task branch t will obtain the part F of the shared features corresponding to this task sh [t], selected by a learnable gate With the original input features Adaptive fusion is performed to obtain the final enhanced task-specific features.

7. The multi-task intensive prediction method based on two-way interaction according to claim 1, characterized in that: During the training process of the multi-task intensive prediction model, an end-to-end joint training method is adopted, and the overall loss function L is expressed as: Where, is the prediction result of task t, Yt is the true label of task t; Lt is the loss function selected by task t, λ t is the loss weight of task t, which is used to balance the contribution of different tasks to the overall optimization.

8. A multi-task intensive prediction system based on two-way interaction, characterized in that: include: Data acquisition module, used to obtain RGB images to be used for multi-task intensive prediction; The task prediction module is used to perform multi-task prediction based on the acquired RGB image using the trained multi-task intensive prediction model to obtain the prediction results of each task; The multi-task intensive prediction model includes: The encoder is used to input RGB images, gradually extract image features through a series of downsampling operations, and output task-general features; The decoder is used to input the common features of the task, gradually restore the size of the feature map through a series of upsampling operations, and output the final enhanced specific features of each task; wherein, at each stage of the decoder, feature refinement is performed by a feature refinement module, and the feature refinement module includes T multi-scale single-task modules and a bidirectional cross-task interaction module; each multi-scale single-task module is used to input the fusion features for a specific task and use the multi-scale scanning mechanism to process them, and output the task-specific features of the specific task; the bidirectional cross-task interaction module is used to input the task-specific features output by all multi-scale single-task modules, and use the bidirectional interactive scanning mechanism to perform cross-task The multi-scale single-task module processes the task information interactively and outputs upsampled features. In the first stage of the decoder, the fused features input by the multi-scale single-task module are the upsampled features of the task-general features of the last stage of the encoder and the fused features of the task-general feature channels of the penultimate stage. In the subsequent stages of the decoder, the fused features input by the multi-scale single-task module are the upsampled features output by the bidirectional cross-task interaction module in the previous stage of the decoder and the fused features of the task-general feature channels of the corresponding stage of the encoder. In the last stage of the decoder, the upsampled features output by the bidirectional cross-task interaction module are the final enhanced task-specific features. The task heads corresponding to each task in multi-task intensive prediction are used to input the final enhanced task-specific features and make predictions, and output the corresponding prediction results of each task.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-task intensive prediction method based on bidirectional interaction according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-task intensive prediction method based on bidirectional interaction according to any one of claims 1 to 7 is implemented.

Images