Multi-task visual processing and method for training model thereof, electronic device, storage medium, program product

Abstract

Embodiments of the present application provide a multi-task visual processing method and a model training method thereof, an electronic device, a storage medium and a program product, relating to the technical field of visual processing. The method comprises: generating a multi-scale feature map of a target image, and performing self-attention encoding processing on the multi-scale feature map to obtain a multi-scale feature sequence; performing multiple decoding processes based on the multi-scale feature sequence and a splicing task query vector of a multi-task visual processing model; the decoding process comprises: performing cross-attention calculation on the multi-scale feature sequence and the splicing task query vector, and performing self-attention calculation on each task query vector to update each task query vector; for each task, performing task prediction according to the task query vector obtained after multiple decoding processes to obtain a task prediction result. Embodiments of the present application introduce a multi-task semantic decoupling mechanism based on multiple task query vectors, thereby improving the collaborative processing accuracy of multiple tasks.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a multi-task vision processing method and model training method, electronic device, storage medium, and program product. Background Technology

[0002] Currently, when visual models handle multiple visual processing tasks such as detection and segmentation in a unified manner, some solutions configure independent convolutional or fully connected layers for each task head and introduce weighted loss functions for training to avoid direct competition between tasks. Other solutions maintain global feature sharing during the decoding stage, simply copying features and feeding them into different task heads to complete parallel inference. While these solutions can achieve a certain level of accuracy on closed datasets, when detection and segmentation tasks have significantly different requirements for spatial granularity, contextual scope, or semantic focus, the cross-information in the shared features can easily lead to representational conflicts between tasks, reducing the overall performance of multi-task visual processing methods for each task. Summary of the Invention

[0003] This application provides a method, electronic device, storage medium, and program product for training multi-task vision processing and its model, in order to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a multi-task visual processing method, including:

[0005] A multi-scale feature map of the target image is generated, and self-attention encoding is performed on the multi-scale feature map to obtain a multi-scale feature sequence;

[0006] Multiple decoding processes are performed on the concatenated task query vector based on the multi-scale feature sequence and the multi-task visual processing model; the concatenated task query vector is obtained by concatenating multiple task query vectors that represent multiple tasks respectively; the decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculation on each of the task query vectors to update each of the task query vectors.

[0007] For each of the tasks, task prediction is performed based on the task query vector obtained after multiple decoding processes to obtain the task prediction result corresponding to the task.

[0008] In some implementations, the plurality of tasks includes an object detection task. For the object detection task, performing task prediction based on the task query vector obtained after multiple decoding processes to obtain the task prediction result corresponding to the task includes:

[0009] The first task query vector corresponding to the target detection task is input into a parallel first feedforward neural network and a second feedforward neural network; the first task query vector is obtained by the target detection task's task query vector after multiple decoding processes.

[0010] The task prediction results corresponding to the target detection task are output, including: the target bounding boxes output by the first feedforward neural network and the target category labels corresponding to the target bounding boxes output by the second feedforward neural network.

[0011] In some implementations, the target category label is predicted from a plurality of preset target category labels, which include at least one known target category label and at least one unknown category label.

[0012] In some implementations, the plurality of tasks includes a background segmentation task, and for the background segmentation task, performing task prediction based on the task query vector corresponding to the task includes:

[0013] Perform feature adaptation processing on the multi-scale feature sequence to obtain the adapted feature tensor;

[0014] A background segmentation mask for the target image is generated based on the second task query vector corresponding to the background segmentation task; the second task query vector is obtained by the task query vector of the background segmentation task after multiple decoding processes.

[0015] The adaptation feature tensor and the background segmentation mask are concatenated according to the channel dimension to form a fused feature tensor;

[0016] Spatial sampling is performed on the fused feature tensor to obtain a background segmentation image with the same size as the target image. Each pixel in the background segmentation image represents the foreground and background using different pixel values.

[0017] In some implementations, generating a multi-scale feature map of the target image and performing self-attention computation on the multi-scale feature map to obtain a multi-scale feature sequence includes:

[0018] Multiple feature maps of the target image are extracted based on a convolutional neural network model and a feature pyramid network model. The multiple feature maps have different spatial scales and feature dimensions of d, where d is a positive integer.

[0019] Each feature map is flattened according to its spatial scale to obtain a sequence of feature vectors at each spatial scale; the feature dimension of each feature vector in each feature vector sequence is d.

[0020] The position encoding is injected into each of the aforementioned feature vectors, and then the sequences of the aforementioned feature vectors are concatenated to form the multi-scale feature map;

[0021] Self-attention encoding is performed on the multi-scale feature map to obtain the multi-scale feature sequence.

[0022] Secondly, embodiments of this application provide a training method for a multi-task visual processing model, including:

[0023] Initial task query vectors are generated for each of the multiple tasks; each task query vector is used to represent the corresponding task.

[0024] The multi-task vision processing model undergoes multiple training processes, the training processes including:

[0025] A multi-scale feature map of the sample image is generated, and self-attention encoding is performed on the multi-scale feature map to obtain a multi-scale feature sequence;

[0026] Multiple decoding processes are performed based on the multi-scale feature sequence and the concatenated task query vector; the concatenated task query vector is obtained by concatenating multiple task query vectors corresponding to multiple tasks; the decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculation on each task query vector to update each task query vector;

[0027] For each of the tasks, task prediction is performed based on the task query vector obtained after multiple decoding processes to obtain the task prediction result corresponding to the task.

[0028] The loss function value of the multi-task visual processing model is calculated based on the task prediction results of each task and the task training labels of the sample images.

[0029] Adjust at least one model parameter in the multi-task vision processing model based on the loss function value;

[0030] The next training process is performed using the updated multi-task vision processing model and the last updated task query vectors, until the training convergence condition is met.

[0031] Thirdly, embodiments of this application provide a multi-task vision processing system, including:

[0032] The feature extraction and encoding module is used to generate a multi-scale feature map of the target image and perform self-attention encoding on the multi-scale feature map to obtain a multi-scale feature sequence.

[0033] The decoding module is used to perform multiple decoding processes based on the multi-scale feature sequence and the preset concatenated task query vector; the concatenated task query vector is obtained by concatenating multiple task query vectors that represent multiple tasks respectively; the decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculation on each of the task query vectors to update each of the task query vectors;

[0034] Multiple task prediction modules are provided, each corresponding to one of the multiple tasks. Each task prediction module is used to perform task prediction based on the task query vector obtained after multiple decoding processes for the task, and to obtain the task prediction result corresponding to the task.

[0035] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0036] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0037] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0038] Based on the above-mentioned multi-task visual processing methods and systems, training methods for multi-task visual processing models, electronic devices, storage media, and program products, this application has at least the following beneficial effects or advantages:

[0039] This embodiment generates a multi-scale feature map of the target image and performs self-attention encoding on the multi-scale feature map to obtain a multi-scale feature sequence. Then, based on the multi-scale feature sequence and the concatenated task query vector of a multi-task visual processing model, multiple decoding processes are performed. The concatenated task query vector is obtained by concatenating multiple task query vectors representing multiple tasks. The decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector; performing self-attention calculation on each task query vector to update each task query vector; and for each task, performing task prediction based on the task query vector obtained after multiple decoding processes to obtain the corresponding task prediction result. This embodiment introduces a multi-task semantic decoupling mechanism based on multiple task query vectors representing different tasks, and performs self-attention calculation on different task query vectors separately, thereby improving the accuracy of collaborative processing of multiple tasks.

[0040] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0041] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0042] Figure 1 A schematic diagram of a multi-task vision processing system provided in an embodiment of this application is shown;

[0043] Figure 2 A flowchart of a multi-task vision processing method provided in an embodiment of this application is shown;

[0044] Figure 3 A flowchart illustrating a training method for a multi-task visual processing model provided in an embodiment of this application is shown.

[0045] Figure 4 A schematic diagram of a multi-task vision processing system provided in an embodiment of this application is shown;

[0046] Figure 5 This illustration shows a schematic diagram of the decoding layer in a decoding module provided in an embodiment of this application;

[0047] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0048] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0049] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0050] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0051] Object detection is one of the fundamental tasks in computer vision, with its core objective being to identify target objects in an image and locate their bounding boxes. Traditional object detection methods can be broadly categorized into two types: two-stage methods and one-stage methods. Two-stage methods, such as Faster R-CNN (Fast Region Convolutional Neural Network), generate candidate regions through a Region Proposal Network (RPN) and perform classification and regression in the second stage, achieving high detection accuracy but relatively low computational efficiency. One-stage methods, such as the YOLO series, transform object detection into a dense prediction task, exhibiting significant advantages in speed and deployment, but their accuracy is relatively limited in complex scenes.

[0052] With the breakthroughs achieved by the Transformer architecture in natural language processing, its global attention mechanism has been introduced into object detection tasks. A representative work is DETR (Detection Transformer, an object detection model based on the Transformer architecture). DETR proposes an end-to-end object detection framework that uses learnable query vectors to interact with the Transformer decoder, significantly simplifying the detection process and demonstrating excellent representation capabilities and generalization. It removes the traditional NMS (Non-Maximum Suppression) module, a crucial post-processing step in object detection tasks, which filters out redundant overlapping bounding boxes, retaining only the bounding boxes most likely to be the target.

[0053] In real-world scenarios, models often need to perform multiple sub-tasks simultaneously, such as background and foreground segmentation (semantic segmentation), detection of known class objects (multi-class classification and localization), and discovery of unknown class objects (open-world object detection).

[0054] Some studies have attempted to use multi-task learning frameworks for joint modeling of object detection and segmentation. For example, Mask R-CNN (Mask Region Convolutional Neural Network) adds branching structures to Faster R-CNN for semantic segmentation. However, such structures typically require complex module design and task-to-task loss balancing, making it difficult to achieve unified and efficient inference. A related technical approach aims to unify different dense prediction tasks (such as semantic segmentation, instance segmentation, and panoptic segmentation). It uses the interaction of a Query (query term or query vector) and a Transformer to perform mask prediction (when processing sequential or image data, by masking (obscuring) part of the input information, the model learns to predict these masked parts. By predicting the masked pixels, the model can better learn local and global features of the image, thereby improving image segmentation performance. By masking part of the input, the model is forced to learn more general features, thus improving its generalization ability on unseen data). Its innovation lies in introducing a Mask Attention mechanism to strengthen the explicit modeling of spatial masks by the query, thereby achieving leading performance in multiple semantic tasks. This technical solution also enables integrated modeling of multiple segmented tasks through a cross-task sharing architecture, significantly improving task consistency and model reusability.

[0055] Open-world detection is particularly challenging because the model needs to learn using only a limited number of categories during training, but must be able to detect and identify unseen categories during inference, which places higher demands on the model's generalization ability. Existing solutions for open-world detection have proposed strategies such as category space expansion and energy distribution modeling to detect unknown targets, but these rely on complex post-processing procedures and lack a unified Transformer decoding and inference architecture.

[0056] To address the aforementioned issues, this application proposes a unified task modeling framework that simultaneously supports multiple task types, including background segmentation, known-category object detection, and unknown-category detection. This application enables multi-task inference to be performed within a single model, balancing accuracy and efficiency, improving the model's adaptability and robustness in complex environments, and making it applicable to open scenarios.

[0057] Current visual processing methods still have many shortcomings in terms of unified multi-task processing, structural generalization ability, and open-world adaptability, mainly reflected in the following aspects:

[0058] (1) It is difficult to unify the modeling of detection and segmentation tasks.

[0059] Current mainstream detection methods focus on bounding box localization and category recognition, lacking the ability to model pixel-level semantic information; while semantic / instance segmentation methods such as Mask R-CNN and Mask2Former focus on mask generation and lack the ability to model spatial information at the bounding box level. As a result, detection and segmentation tasks often require the design of different models or independent branches, making it difficult to achieve structural unity and inference collaboration, increasing model complexity and deployment burden.

[0060] (2) Interference between information between multiple tasks leads to representation conflicts.

[0061] In multi-task settings, the lack of explicit task modeling mechanisms often leads to gradient conflicts and information confusion when different tasks share feature representations. For example, detection tasks focus on object boundaries and locations, while segmentation tasks emphasize semantic regions of objects; the two have inherent differences in spatial scale and semantic granularity. Existing methods such as Mask2Former attempt to unify segmentation tasks, but lack clear task decoupling strategies when introducing object detection tasks, making it difficult to maintain performance balance among subtasks.

[0062] (3) Insufficient ability to identify unknown categories and limited generalization.

[0063] Most detectors are trained in closed-world settings and can only identify predefined categories, lacking adaptation mechanisms for unknown classes. Although some open-world detection methods attempt to introduce energy modeling or query expansion strategies to identify unknown targets, they typically rely on complex auxiliary modules (such as open class miners and energy discriminators) and cumbersome post-processing procedures, increasing the difficulty of model training and the complexity of actual deployment. In addition, existing methods often neglect the mask representation and pixel-level modeling of unknown targets, resulting in imperfect perception capabilities in open scenes.

[0064] (4) Lack of ability to integrate multi-tasking and open-world operations under a unified Transformer architecture

[0065] While DETR and its variants demonstrate the potential of Transformers in object detection, their structural designs are largely focused on detection tasks of known objects, lacking the ability to model across tasks and class spaces. Existing work has not fully explored how to achieve collaborative reasoning for segmentation, detection, and unknown class modeling within a unified Transformer decoder, thus limiting its breadth of application and robustness in real-world, dynamic environments.

[0066] In summary, current visual processing methods still have significant shortcomings in terms of structural uniformity, task collaboration, and open-world adaptability. There is an urgent need for a new detection architecture that can integrate multiple task types under a unified framework, explicitly guide task semantic differentiation, and have sensitivity and generalization capabilities for unknown targets, so as to meet the needs of multi-task visual understanding in complex real-world scenarios.

[0067] To support multiple task requirements simultaneously, this application provides a multi-task vision processing system, referencing... Figure 1 It includes the following modules:

[0068] The feature extraction and encoding module 101 is used to generate multi-scale feature maps of the target image and perform self-attention encoding on the multi-scale feature maps to obtain a multi-scale feature sequence.

[0069] The decoding module 102 is used to perform multiple decoding processes based on the multi-scale feature sequence and the preset concatenated task query vector. The concatenated task query vector is obtained by concatenating multiple task query vectors that represent multiple tasks respectively. The decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculation on each task query vector to update each task query vector.

[0070] Multiple task prediction modules 103 correspond one-to-one with multiple tasks. Each task prediction module is used to perform task prediction based on the task query vector obtained after multiple decoding processes, and obtain the task prediction result corresponding to the task.

[0071] It is understood that the multi-task vision processing system provided in the embodiments of this application can be implemented as a model, application, service, etc.

[0072] Multiple tasks can include:

[0073] Background segmentation task: to distinguish between background and foreground at the pixel level;

[0074] Specific category detection task: Detect and classify targets of known categories;

[0075] Unknown category detection task: Detect objects without labeled categories in an open world setting;

[0076] Depth evaluation task: Evaluate each pixel to predict its distance to the camera, i.e., the depth value.

[0077] The target image is input to the feature extraction and encoding module 101. First, features are extracted from the input data through the backbone network. The backbone network is the foundation of the entire model, providing feature representations for subsequent tasks (such as classification, object detection, and segmentation), generating multi-scale feature maps. The backbone network can be a pre-trained neural network model, encoded into a unified-dimensional multi-scale feature sequence based on Transformer. Then, based on the concatenated task query vector obtained by concatenating task query vectors representing multiple tasks, the Transformer decoder is guided to infer in the global context, completing the joint prediction of multiple tasks and obtaining task prediction results such as segmentation masks, detection boxes, and class labels. This unified modeling approach significantly simplifies the multi-task modeling process in complex scenes and improves detection and segmentation performance in open-world environments.

[0078] The embodiments of this application represent different tasks based on multiple task query vectors, introduce a multi-task semantic decoupling mechanism, and perform self-attention calculation on different task query vectors respectively, thereby improving the accuracy of collaborative processing of multiple tasks.

[0079] This application provides a multi-task vision processing method, which can be executed by the multi-task vision processing system provided in this application. Specific implementation methods and technical effects can be referenced interchangeably. (References) Figure 2 The method specifically includes steps 201-203.

[0080] Step 201: Generate a multi-scale feature map of the target image and perform self-attention encoding on the multi-scale feature map to obtain a multi-scale feature sequence.

[0081] Step 202: Perform multiple decoding processes on the spliced ​​task query vector based on the multi-scale feature sequence and the multi-task visual processing model.

[0082] Step 203: For each task, perform task prediction based on the task query vector obtained after multiple decoding processes to obtain the task prediction result corresponding to the task.

[0083] Multi-scale feature maps are extracted based on the backbone network. Specifically, a Feature Pyramid Network (FPN) structure based on the backbone of a neural network model can be used to extract feature maps of different scales, which can represent features of different granularities in the target image. Then, self-attention encoding is used to calculate a multi-scale feature sequence concatenated with a unified dimension.

[0084] The concatenated task query vector is pre-trained and obtained by concatenating multiple task query vectors, each representing a different task. The decoding process includes: performing cross-attention calculations on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculations on each task query vector to update them. In practice, the decoding process can be implemented using the same decoding layer, with multiple decoding layers connected end-to-end to achieve multiple decoding processes. After multiple decoding processes, each task query vector can extract features from the multi-scale feature sequence of the target image.

[0085] After multiple decoding iterations, task prediction is performed for each task query vector to obtain the corresponding task prediction result. Different task prediction heads are designed based on different tasks and learn task prediction capabilities through pre-training, forming the parameters in the task prediction model.

[0086] This embodiment generates a multi-scale feature map of the target image and performs self-attention encoding on the multi-scale feature map to obtain a multi-scale feature sequence. Then, based on the multi-scale feature sequence and the concatenated task query vector of a multi-task visual processing model, multiple decoding processes are performed. The concatenated task query vector is obtained by concatenating multiple task query vectors representing multiple tasks. The decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector; performing self-attention calculation on each task query vector to update each task query vector; and for each task, performing task prediction based on the task query vector obtained after multiple decoding processes to obtain the corresponding task prediction result. This embodiment introduces a multi-task semantic decoupling mechanism based on multiple task query vectors representing different tasks, and performs self-attention calculation on different task query vectors separately, thereby improving the accuracy of collaborative processing of multiple tasks.

[0087] In some implementations, multiple tasks include an object detection task. Accordingly, for the object detection task, step 203 performs task prediction based on the task query vector obtained after multiple decoding processes to obtain the task prediction result corresponding to the task. Specifically, this includes the following steps: inputting the first task query vector corresponding to the object detection task into a parallel first feedforward neural network and a second feedforward neural network. The first task query vector is obtained after multiple decoding processes of the task query vector of the object detection task. Outputting the task prediction result corresponding to the object detection task includes: the bounding boxes of each target output by the first feedforward neural network and the target category labels corresponding to the bounding boxes of each target output by the second feedforward neural network.

[0088] In this embodiment, object detection is one of several tasks. The task result is predicted by performing task prediction on the task query vector corresponding to the object detection task. The task detection result required by the object detection task includes the target bounding box and the target category label corresponding to the bounding box. This implementation can be a task detection module, which includes two parallel feedforward neural networks. The first feedforward neural network is responsible for predicting the bounding box of the target, i.e., determining the position and size of the target in the image. The second feedforward neural network predicts the corresponding target category label for these bounding boxes, i.e., identifying which known category the target in the bounding box belongs to. This parallel processing method improves the efficiency and accuracy of object detection. By processing spatial information (bounding box) and semantic information (category label) simultaneously through the same task query vector, it can accurately identify targets in the image and determine their categories while maintaining high efficiency in the processing. This is particularly important for applications requiring real-time feedback. Through this design, the bounding box prediction and category recognition are decoupled in the task prediction processing of the object detection task, allowing each feedforward neural network to focus on its specific information type, thereby improving the overall detection performance and accuracy.

[0089] In some implementations, the target category label is predicted from a plurality of preset target category labels, which include at least one known target category label and at least one unknown category label.

[0090] Multiple predefined target category labels can be represented by a category label vector. For example, in the vector {x1, x2, ..., xn}, each element represents a category, and the specific value of each element represents the probability of belonging to the corresponding category. Among the multiple categories, at least one unknown category may be included. It can be understood that all categories other than the known categories can be uniformly classified as unknown categories. Accordingly, an element (such as xn) is designed in the category label vector to represent the unknown category label.

[0091] This implementation enhances the model's adaptability to new categories and improves its generalization performance in open-world scenarios by introducing unknown category labels. The design of the category label vectors enables the model to identify and process unknown targets, thus making it more robust and flexible in practical applications.

[0092] In some implementations, multiple tasks include a background segmentation task. Accordingly, for the background segmentation task, step 203 performs task prediction based on the task query vector corresponding to the task. Specifically, this may include the following steps: performing feature adaptation processing on the multi-scale feature sequence to obtain an adapted feature tensor; generating a background segmentation mask for the target image based on the second task query vector corresponding to the background segmentation task, where the second task query vector is obtained after multiple decoding processes. Next, the adapted feature tensor and the background segmentation mask are concatenated according to the channel dimension to form a fused feature tensor, and spatial sampling is performed on the fused feature tensor to obtain a background segmentation image with the same size as the target image. In the background segmentation image, each pixel uses a different pixel value to represent the foreground and background.

[0093] In one implementation, background segmentation is one of several tasks, which involves separating the foreground and background in a target image. First, a multi-scale feature sequence is optimized through feature adaptation to obtain a more suitable adaptive feature tensor for the background segmentation task. This process involves adjusting the features to highlight information useful for the segmentation task. Then, a background segmentation mask for the target image is generated using a second task query vector obtained through multiple decoding processes. This mask identifies whether each pixel in the image belongs to the foreground or background. Next, the adapted feature tensor and the background segmentation mask are concatenated along the channel dimension to form a fused feature tensor. This step helps integrate information from different sources to improve segmentation accuracy.

[0094] Suppose we have an adaptation feature tensor and a background segmentation mask, and we need to concatenate them along the channel dimension. Here's a concrete example to illustrate this process:

[0095] Adaptive Feature Tensor: Suppose we obtain an adaptive feature tensor from the convolutional neural network, with dimensions H×W×D, where H is the height, W is the width, and D is the feature dimension (number of channels). For example, the adaptive feature tensor might be 64×64×128, meaning it has 64 heights and 64 widths, with 128 feature channels at each location.

[0096] Background segmentation mask: The background segmentation task generates a mask of size H×W×C, where C is the number of classes. In a binary classification background segmentation task (foreground and background), the mask might be 64×64×2, meaning that each position has two channels, representing the probabilities or class labels of the foreground and background, respectively.

[0097] Channel-dimensional concatenation: To concatenate these two tensors along the channel dimension, we need to ensure that their height and width are the same, and then add their channel numbers together. In the example above, we concatenate the 128-channel adaptation feature tensor and the 2-channel background segmentation mask to obtain a new tensor with a size of 64×64×(128+2)=64×64×130.

[0098] Fusion Feature Tensor: The concatenated tensor is called the fusion feature tensor, which contains feature information from the original image and category information for background segmentation. This fusion feature tensor can be used in subsequent processing steps, such as spatial sampling, to generate the final background segmentation image.

[0099] Through this concatenation, the model can simultaneously consider the content features of the image and the category information of the segmentation task in a unified representation, thereby improving the accuracy and efficiency of background segmentation. Finally, spatial sampling is performed on the fused feature tensor to generate a background segmentation image with the same size as the original target image, where the value of each pixel indicates whether it belongs to the foreground or the background. It can be understood that if the scale of the generated fused feature tensor is larger than the size of the original target image, spatial downsampling is performed; otherwise, spatial upsampling is performed. This implementation can accurately extract foreground objects from complex backgrounds, providing high-quality segmentation results for applications such as image editing and object tracking, while maintaining the efficiency of the processing.

[0100] In some implementations, step 201 generates multi-scale feature maps of the target image and performs self-attention computation on the multi-scale feature maps to obtain a multi-scale feature sequence. Specifically, this includes the following steps: extracting multiple feature maps of the target image based on a convolutional neural network model and a feature pyramid network model. These feature maps have different spatial scales and a feature dimension of d, where d is a positive integer. Each feature map is flattened according to its spatial scale to obtain a feature vector sequence at each spatial scale. The feature dimension of each feature vector in each feature vector sequence is d. Position encoding is injected into each feature vector, and the feature vector sequences are then concatenated to form a multi-scale feature map. Self-attention encoding is then performed on the multi-scale feature map to obtain a multi-scale feature sequence.

[0101] In this embodiment, multiple feature maps with different spatial scales can be extracted from the target image using convolutional neural networks (CNNs) and feature pyramid networks (FPNs). These feature maps have different resolutions but the same feature dimension d. Then, each feature map is flattened according to its spatial scale, converted into a sequence of feature vectors, and a positional encoding is injected into each feature vector to preserve spatial information. Next, all feature vector sequences are concatenated into a multi-scale feature map, which is then subjected to self-attention encoding to obtain a multi-scale feature sequence capable of capturing the dependencies between different regions in the image.

[0102] For example, suppose a target image, after being processed by CNN and FPN, yields three feature maps of different scales: 64x64, 32x32, and 16x16, with a feature dimension d of 256. These feature maps are flattened to obtain three feature vector sequences, each with a vector dimension of 256. After injecting positional encoding into each vector, they are concatenated to form a multi-scale feature map containing information from all scales. Finally, self-attention is performed on this map to obtain a multi-scale feature sequence that can be used for prediction in subsequent tasks.

[0103] This implementation method, through multi-scale feature extraction and self-attention mechanism, can capture image features more comprehensively, improving the accuracy and robustness of visual tasks, especially when processing images with complex backgrounds and multi-scale targets.

[0104] This application provides a training method for a multi-task visual processing model. The training method for the multi-task visual processing model provided in this application can train a multi-task visual processing model. It is understood that the multi-task visual processing model can be implemented as the multi-task visual processing system or multi-task visual processing method provided in this application. Specific implementation methods and technical effects can be referred to each other and will not be repeated.

[0105] refer to Figure 3 The training method for the multi-task visual processing model provided in this application embodiment includes steps 301-305:

[0106] Step 301: Generate initial task query vectors for each of the multiple tasks; each task query vector is used to represent the corresponding task.

[0107] Step 302: Perform multiple training processes on the multi-task visual processing model. The training process includes the following steps:

[0108] Step 3021: Generate multi-scale feature maps of the sample images and perform self-attention encoding on the multi-scale feature maps to obtain multi-scale feature sequences.

[0109] Step 3022 involves performing multiple decoding processes based on the multi-scale feature sequence and the concatenated task query vector. The concatenated task query vector is obtained by concatenating multiple task query vectors corresponding to multiple tasks. The decoding process includes: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculation on each task query vector to update each task query vector.

[0110] Step 3023: For each task, perform task prediction based on the task query vector obtained after multiple decoding processes to obtain the task prediction result corresponding to the task.

[0111] Step 303: Calculate the loss function value of the multi-task visual processing model based on the task prediction results of each task and the task training labels of the sample images.

[0112] Step 304: Adjust at least one model parameter in the multi-task vision processing model based on the loss function value.

[0113] Step 305: Execute the next training process using the updated multi-task visual processing model and the last updated query vectors for each task until the training convergence condition is met.

[0114] In traditional DETR, the decoder relies solely on a set of randomly initialized learnable queries, gradually focusing on potential targets in the image through self-attention and cross-attention mechanisms. While this structure offers end-to-end prediction capabilities and high representational flexibility, in multi-task scenarios (such as object detection and background semantic segmentation), all task signals are mixed within the same query space, easily leading to gradient conflicts and feature confusion between tasks, and lacking explicit modeling and decoupled representation of task characteristics. To address this issue, the multi-task vision processing system provided in this application adopts hybrid queries (Hybrid Queries) in the decoding module, i.e., concatenating task query vectors as input, to balance task guidance and object perception capabilities.

[0115] The multi-task vision processing system provided in this application embodiment can be implemented as a vision processing model, referencing... Figure 4 The input to the multi-task vision processing system is the target image. The algorithm employs ResNet50 as the backbone network to extract image features. To address the need for fine-grained features in small object detection and background region segmentation, a feature pyramid (FPN) structure based on the backbone network is used for multi-scale feature extraction, generating multi-scale feature maps. Specifically, the backbone network (e.g., ResNet-50) extracts feature maps at three scales {C3, C4, C5}, and then performs upsampling and element-wise addition fusion through a top-down path to construct a multi-scale feature pyramid {P3, P4, P5}. 1x1 convolutions are used to map the number of channels of each multi-scale feature map to the same dimension d, and positional encoding is used to unfold the position-encoded multi-scale feature maps into sequences, which are then concatenated. Finally, the sequences from multiple scales are concatenated into a long sequence and fed into a Transformer encoder for self-attention encoding. The Transformer encoder utilizes a multi-head self-attention mechanism to model the global interactive context of features across space and scale, enhancing semantic expressiveness and spatial information integration. After processing through multiple layers of self-attention and feedforward networks, the encoder outputs a context-enhanced feature sequence of the same shape, i.e., a multi-scale feature sequence, denoted as: This is a global semantic fusion of input multi-scale features, combining spatial and cross-scale information for subsequent decoders to use in tasks such as object detection and segmentation.

[0116] Each task query vector in the concatenated task query vector is a learnable vector, continuously optimized during training to provide task-aware guidance and enhance the consistency and accuracy of downstream predictions. In one instance, a set of token vectors specifically designed for background semantic segmentation can be used. ), to play the role of task awareness guide.

[0117] In multi-task scenarios involving object detection and background semantic segmentation, the concatenated task query vector includes:

[0118] First task query vector As a display task guidance signal for the decoding module, it is used to highlight the semantic space required for specific tasks such as background semantic segmentation.

[0119] Second task: query vector Random initialization, independent of the first task query vector, aims to model salient region and structural information at the target level, exhibiting strong generalization ability. This part is used to capture the appearance and spatial features of potential targets, serving the target detection task.

[0120] The query vector for the concatenation task is:

[0121]

[0122] This structure maintains DETR's original end-to-end prediction advantages while enabling explicit differentiation between tasks and enhancing its ability to detect open categories.

[0123] To further enhance the decoupling of multi-task decoding, this embodiment reconstructs the traditional Transformer decoder structure, employing a dual-channel attention mechanism of "task-aware + query-aware" to explicitly isolate the information flow between different task query vectors, avoiding task confusion and semantic pollution. The semantic segmentation task involved in this embodiment mainly targets background regions (such as sky and ground), while the object detection task focuses on foreground objects (such as pedestrians and vehicles). The two differ significantly in semantic focus and representation space, making the introduction of an inter-group decoupling mechanism particularly suitable. Specifically, the first task query vector focuses on pixel-level background region modeling, while the second task query vector emphasizes target structure and appearance information; they have almost no overlap in encoding space and supervision signals. This separation of semantic focus means that isolation processing does not lead to information fragmentation but rather helps avoid feature pollution and gradient interference, thereby improving the modeling purity and stability of each task.

[0124] The decoding module comprises N decoding layers. To ensure the continuity of input-output logic and the consistency of information transmission across the multiple decoding layers, each decoding layer is processed and iteratively updated by concatenating task query vectors. (Reference) Figure 5 The specific process of each decoding layer is as follows:

[0125] Let the first The query vector input for the layer decoding layer is the concatenation task query vector:

[0126]

[0127] For the The layer decoder takes the multi-scale feature sequence of the input encoder as its input. , concatenate the task query vector and multi-scale feature sequences The input mask is used for cross-attention calculation. The calculation is performed, and the result is:

[0128]

[0129]

[0130] After addition and normalization, each task query vector is then independently updated using masked multi-head self-attention, resulting in:

[0131]

[0132]

[0133] After further addition and normalization, a feedforward network, and another round of addition and normalization, the update results of the query vectors from each task are concatenated as the output of the current decoding layer.

[0134]

[0135] This design achieves explicit decoupling between the task guidance path and the target modeling path within the decoder, which not only improves the aggregation capability of semantic segmentation tasks but also avoids interference from semantic tasks on the target detection representation, and strengthens the exclusive representation space of each type of query in a specific task.

[0136] After multiple decoding layers, the task prediction module branches for each task extract the corresponding task query vector from the corresponding position of the concatenated task query vector.

[0137] For different types of tasks, corresponding task prediction modules were designed.

[0138] For the first task query vector output by the decoding module Two parallel feedforward neural networks, such as MLP (Multilayer Perceptron) heads, are used for processing. The two MLP heads are used for bounding box regression and target category prediction, respectively. For the bounding box regression head to predict the spatial location information of the target, a multilayer perceptron structure is also used. Each bounding box is represented by its center coordinates and dimensions (x, y, w, h), resulting in an output vector. To improve training stability, a Sigmoid activation is added before bounding box regression, normalizing the coordinate values ​​to the [0,1] interval. For the category classification head predicting the category of each query, the output dimension is... The number of categories The number of known categories The number of unknown categories in open-world identification is 1. To support multi-label classification, allowing a single query to point to multiple categories simultaneously (including unknown open categories), the classification head uses a Sigmoid activation function to generate independent probability values ​​for each category.

[0139] Although a task separation mechanism was introduced in the decoding module stage to concatenate the query vector, the multi-scale feature sequence output by the encoding module... It remains completely shared. If different tasks have different optimal regions of interest in the spatial or channel dimensions, gradient conflicts and competition between tasks may still occur. To further mitigate this issue, a lightweight adaptive module—task-specific adapters—can be introduced after the encoding module as a low-cost task decoupling module, improving compatibility and stability between tasks. Each task corresponds to a lightweight bottleneck-type Adapter module, designed for background segmentation tasks, and may include a fully connected dimensionality reduction layer. Activation function (ReLU) Fully Connected Upgrade Layer And residual connections, to obtain the adaptive feature tensor :

[0140]

[0141] Task query vector based on background segmentation task The process guides the generation of a segmentation mask. Specifically, the task query vector for the background segmentation task... By extending to the spatial dimension and using the adaptive feature tensor The components are concatenated along the channel dimension to form a fused feature tensor. The fused feature tensor is processed by a lightweight mask decoder to progressively restore it to the input image size, outputting a single-channel mask. The mask output is normalized using a sigmoid function to represent the foreground probability of each pixel, yielding the task prediction result for the background segmentation task. This is then used in conjunction with the task query vector for the background segmentation task. Guided segmentation not only improves the consistency of the mask, but also effectively reduces the computational overhead in traditional pixel classification methods.

[0142] To effectively supervise the joint learning process of the multi-task visual processing model on object detection and pixel-level background segmentation tasks during training, a task-aware multi-task loss function system was designed. The overall loss function consists of detection loss... and segmentation loss Composition, represented as:

[0143]

[0144] in, , These are the weighting coefficients for the losses of each task, used to balance the importance of different tasks. By jointly optimizing the losses of each task, the embodiments of this application can effectively fuse potential collaborative information between tasks while maintaining representation decoupling, achieving high-precision end-to-end object detection and background semantic segmentation.

[0145] For detecting branch loss This implementation uses a Hungarian matching mechanism to match ground truth bounding boxes with predicted bounding boxes one-to-one, ensuring the uniqueness and robustness of the matching relationships. After matching, the bounding box regression loss and multi-label classification loss are calculated for each pair of matched targets. The total loss is as follows:

[0146]

[0147] in, For hyperparameters, It is the bounding box regression loss. It is a multi-category label classification loss.

[0148] Bounding box regression loss measures the geometric difference between the predicted and ground truth bounding boxes. A combination of L1 loss and generalized IoU (GIoU) ​​loss is used to simultaneously optimize localization accuracy and matching robustness.

[0149]

[0150] in It is a prediction box. It is the normalized true bounding box. and It's a hyperparameter.

[0151] The multi-label classification loss uses the multi-label binary cross entropy (BCE) loss to supervise the class distribution of each target, supporting single-target multi-label prediction and reserving space for future open-world detection. The multi-label classification loss can be expressed as follows:

[0152]

[0153] in, It is the predicted probability of category c on query j. It is the corresponding category label indicator. The number of targets is represented by , and the number of categories is represented by M. This design effectively supports multi-label prediction of known and unknown categories, adapting to the needs of open-world target recognition.

[0154] Branching loss The class distribution of each ObjectQuery can be supervised using multi-label binary cross-entropy loss and Dice loss, supporting single-object multi-label prediction and reserving space for future open-world detection. (Segmentation branch loss) This can be expressed as follows:

[0155]

[0156] In the above formula, For the predicted background mask, For a normalized true background mask, Indicates the number of sample images. This represents the weighting coefficient of the Dice loss, used to balance the importance of the two losses.

[0157] Pixel-level binary cross-entropy loss For the binary segmentation task of each pixel, directly supervise whether each position in the mask output is the target region:

[0158]

[0159] Where H and W are the mask resolutions.

[0160] Dice loss More sensitive to small targets and foreground-background imbalances, it can be used to encourage overall regional overlap, improve boundary quality, and enhance regional consistency.

[0161]

[0162] in It is a smoothing term to prevent the denominator from being zero.

[0163] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0164] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0165] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0166] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0167] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0168] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0169] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0170] It should be understood that the aforementioned processor can be a CPU (Central Processing Unit), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0171] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0172] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0173] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0174] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0175] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0176] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0177] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0178] Furthermore, the functional units in the various embodiments of this application can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0179] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of multi-task visual processing, the method comprising: The method comprises the following steps: generating a multi-scale feature map of a target image, and performing self-attention encoding processing on the multi-scale feature map to obtain a multi-scale feature sequence; performing multiple decoding processes based on the multi-scale feature sequence and a splicing task query vector of a multi-task visual processing model; the splicing task query vector is obtained by splicing multiple task query vectors respectively representing multiple tasks, and the decoding process comprises: performing cross-attention calculation on the multi-scale feature sequence and the splicing task query vector, and performing self-attention calculation on each task query vector to update each task query vector; for each task, performing task prediction according to the task query vector obtained after multiple decoding processes to obtain a task prediction result corresponding to the task; the multiple tasks include a target detection task, and for the target detection task, the task prediction according to the task query vector obtained after multiple decoding processes to obtain a task prediction result corresponding to the task comprises: inputting the first task query vector corresponding to the target detection task into a first feedforward neural network and a second feedforward neural network in parallel; the first task query vector is obtained by performing multiple decoding processes on the task query vector of the target detection task; outputting the task prediction result corresponding to the target detection task, which comprises each target bounding box output by the first feedforward neural network and the target class label corresponding to each target bounding box output by the second feedforward neural network; the multiple tasks include a background segmentation task, and for the background segmentation task, the task prediction according to the task query vector corresponding to the task comprises: performing feature adaptation processing on the multi-scale feature sequence to obtain an adapted feature tensor; generating a background segmentation mask of the target image according to a second task query vector corresponding to the background segmentation task; the second task query vector is obtained by performing multiple decoding processes on the task query vector of the background segmentation task; splicing the adapted feature tensor and the background segmentation mask in the channel dimension to form a fusion feature tensor; performing spatial sampling on the fusion feature tensor to obtain a background segmentation image with the same size as the target image, and each pixel in the background segmentation image represents foreground and background with different pixel values.

2. The method of claim 1, wherein, The target class label is predicted from multiple preset target class labels, and the multiple preset target class labels include at least one known target class label and at least one unknown class label.

3. The method of claim 1, wherein, The method for generating a multi-scale feature map of a target image and performing self-attention calculation on the multi-scale feature map to obtain a multi-scale feature sequence comprises the following steps: extracting multiple feature maps of the target image based on a convolutional neural network model and a feature pyramid network model, wherein the multiple feature maps have different spatial scales and feature dimensions d, and d is a positive integer; flattening each feature map according to the spatial scale to obtain a feature vector sequence under each spatial scale; each feature vector in each feature vector sequence has a feature dimension d. Injecting position coding to each feature vector, and concatenating each feature vector sequence into the multi-scale feature map; Performing self-attention coding processing on the multi-scale feature map to obtain the multi-scale feature sequence.

4. A method for training a multitask visual processing model, the method comprising: The method comprises: Generating initial task query vectors for multiple tasks respectively; Each task query vector is used to represent a corresponding task; Performing multiple training processes on the multi-task visual processing model, the training processes comprising: Generating a multi-scale feature map of a sample image, and performing self-attention coding processing on the multi-scale feature map to obtain a multi-scale feature sequence; Performing multiple decoding processes based on the multi-scale feature sequence and a concatenated task query vector; the concatenated task query vector is obtained by concatenating multiple task query vectors corresponding to the multiple tasks, and the decoding processes comprise: performing cross-attention calculation on the multi-scale feature sequence and the concatenated task query vector, and performing self-attention calculation on each task query vector to update each task query vector; For each task, performing task prediction based on the task query vector obtained after multiple decoding processes to obtain a task prediction result corresponding to the task; According to the task prediction result of each task and the task training label of the sample image, calculating a loss function value of the multi-task visual processing model; Adjusting at least one model parameter in the multi-task visual processing model according to the loss function value; Performing the next training process using the updated multi-task visual processing model and the last updated task query vector until the training convergence condition is reached; The multiple tasks comprise a target detection task, and for the target detection task, the task prediction based on the task query vector obtained after multiple decoding processes to obtain a task prediction result corresponding to the task comprises: Inputting the first task query vector corresponding to the target detection task into a first feedforward neural network and a second feedforward neural network in parallel; the first task query vector is obtained by performing multiple decoding processes on the task query vector of the target detection task; Outputting the task prediction result corresponding to the target detection task comprises: outputting each target bounding box by the first feedforward neural network and outputting a target class label corresponding to each target bounding box by the second feedforward neural network; The multiple tasks comprise a background segmentation task, and for the background segmentation task, the task prediction based on the task query vector corresponding to the task comprises: Performing feature adaptation processing on the multi-scale feature sequence to obtain an adapted feature tensor; Generating a background segmentation mask of the target image based on a second task query vector corresponding to the background segmentation task; the second task query vector is obtained by performing multiple decoding processes on the task query vector of the background segmentation task; Concatenating the adapted feature tensor and the background segmentation mask in the channel dimension to form a fusion feature tensor; performing spatial sampling on the fusion feature tensor to obtain a background segmentation image with the same size as the target image, each pixel in the background segmentation image representing foreground and background with different pixel values.

5. A multi-task vision processing system, characterized by, Comprise: a feature extraction and encoding module configured to generate a multi-scale feature map of a target image and perform self-attention encoding processing on the multi-scale feature map to obtain a multi-scale feature sequence; a decoding module configured to perform multiple decoding processes based on the multi-scale feature sequence and a preset splicing task query vector; the splicing task query vector is obtained by splicing multiple task query vectors respectively representing multiple tasks, and the decoding process comprises: performing cross-attention calculation on the multi-scale feature sequence and the splicing task query vector, and performing self-attention calculation on each task query vector to update each task query vector; a plurality of task prediction modules corresponding to the plurality of tasks, each task prediction module being configured to perform task prediction on the task query vector obtained after multiple decoding processes to obtain a task prediction result corresponding to the task; the multiple tasks include a target detection task, and for the target detection task, the task prediction based on the task query vector obtained after multiple decoding processes to obtain a task prediction result corresponding to the task includes: inputting the first task query vector corresponding to the target detection task into a first feedforward neural network and a second feedforward neural network in parallel; the first task query vector is obtained by performing multiple decoding processes on the task query vector of the target detection task; outputting the task prediction result corresponding to the target detection task includes: each target bounding box output by the first feedforward neural network and the target class label corresponding to each target bounding box output by the second feedforward neural network; the multiple tasks include a background segmentation task, and for the background segmentation task, the task prediction based on the task query vector corresponding to the task includes: performing feature adaptation processing on the multi-scale feature sequence to obtain an adapted feature tensor; generating a background segmentation mask of the target image according to a second task query vector corresponding to the background segmentation task; the second task query vector is obtained by performing multiple decoding processes on the task query vector of the background segmentation task; splicing the adapted feature tensor and the background segmentation mask in the channel dimension to form a fusion feature tensor; performing spatial sampling on the fusion feature tensor to obtain a background segmentation image with the same size as the target image, each pixel in the background segmentation image representing foreground and background with different pixel values.

6. An electronic device comprising a storage, a processor and a computer program stored on the storage, characterized in that The processor implements the method of any one of claims 1-3 when executing the computer program.

7. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program implements the method of any one of claims 1-3 when executed by a processor.

8. A computer program product comprising a computer program, characterized in that, The computer program implements the method of any one of claims 1-3 when executed by a processor.

Images